Something so very aggravating about clicking through to that Steam post and seeing that the top reply basically just says "you're lying." And it even uses the word "verifiably"--naturally, it doesn't actually link or even reference any evidence that Lindgren is wrong. Maybe this is just all Discourse online now, but it's depressing how predictably and quickly gamers get whipped up into a righteous fervor. It's been like 15 years of gaming discourse feeling like a never-ending loop of this. And when you get through all the froth, there doesn't seem to be any substance. Like, I thought the complaints about Steam Deck compatibility were legit, and then I realized the post was weeks old and you can play the game just fine on Steam Deck (a tad bit finicky but that's very normal with the Steam Deck)… the complaints were pre-release and purely hypothetical. And, as has often been in the case in the past, I'm guessing there won't really be any evidence for the performance issues either.
Anyways, I really appreciated your piece and have subscribed. Good to hear from people in the industry and in particular I'm glad Riot has been willing to lift the veil a tiny bit. I will say that as someone who is especially paranoid about my security and privacy, I do dislike the idea of kernel anti-cheats, but I recognize I'm talking about a really marginal risk here and one that even paranoid people like me are already accepting on a daily basis. I don't play any games that use 'em, but that's not a principled boycott or anything. I just haven't fallen in love with one of those games yet. Though Riot's about to expand Vanguard to LoL, so that might change soon.
I will say that even though I know it's a great slam dunk for anti-cheat, I hate that Riot's Vanguard implementation requires it to be continuously on from device startup. That's a lot of trust I'm handing over in the sense that you really gotta believe they've done their best to avoid performance impacts, compatibility issues, and security risks. But in the age of SSDs and 10 second restarts, even my paranoid self can't really complain too much.
I wonder when Koskinas made Maplestory cheats. @Ryan if you get a chance, could you ask him if he ever worked on Gamekillers Terminal or Gamersouls Blight, or if it was way earlier?
I got to really question the knowledge of those interviewed people. Someone who is saying that a software running in the application layer can do the same things as a kernel driver has either not knowledge and is ignorant about the topic OR is just LYING. Why do we even need all of those layers if they all can do the same thing anyway?
As a game developer, I would say that most peoples knows that game developers don't want to steal your data. The issue with kernel-level anti-cheat is that you end up having a lot of them installed on your computer. If only one is compromised with a supply chain attack, the attacker can access all the players computers.
A significant example of a supply chain attack was the Solarwinds cyber attack, in which their build server got compromised. Luckily for their client, Solarwinds products don't run in ring 0, so the attack impact was limited. Still, if they did, the attacker would have completely controlled all their customer's servers, including Microsoft, and all the data of Microsoft clients. If SolarWinds and Microsoft can get compromised, then a game studio or an anti-cheat developer can also have their build server compromised.
Nobody should have full access to your computer besides you because you can't audit their security and be sure they have good security practices or won't be breached.
Hi, I am a software engineer who used to work on video game (engine and graphics) so I think I have a decent understanding of how game dev works. I always roll my eyes at articles like this because when people get harmed by others, there are usually two reasons: 1) maliciousness, 2) incompetence. Game developers love to latch on to (1) and claim they have no incentive to harm their players etc, which may be true, but keep in mind we don't know what deployment or code review policies each team uses and from having worked in games before a lot of teams have a pretty lax code policy compared to other areas I have worked in (aerospace). It's not crazy to assume that backdoors could still be injected by malicious employees even if the company didn't want to.
The more important issue is (2) (incompetence). Programmers like to think of themselves as rockstars, but *everyone* makes mistakes. Having a kernel level anti-cheat significantly increases the attack vector compared to a user-level app, and if you have an unfortunate bug or supply chain attack (e.g. via a third-party package) the end result is more catastrophic. In security, we believe in defense in depth, and using a kernel level driver directly goes against that. When you talked about trust, note that you are asking for people to not just trust that you are not malicious, but that trusting you are not incompetent.
This also leads to the comment that a user-level program can access the webcam. I don't think this is necessarily true. It depends on the OS / hardware, but there are permissions that you have to seek before you can access the webcam. A kernel level driver can potentially do other things such as being able to record silently without turning on the light, etc. You essentially get to bypass all of the safeguards modern OSes have put on to prevent such malicious usages. Riot's Vanguard system also runs at boot, meaning that you are vulnerable all the time, rather than only when playing the game. Even if Vanguard doesn't talk to the internet, it talks to other programs on your PC, and those programs (including the software updater that updates Vanguard) talk to the internet. It's a layer of indirection, but if there are flaws in the way the IPC (interprocess communication) then Vanguard could be exploited all the same. The fact that it's an always-on kernel driver gives a lot of incentives for attackers to target it.
Ultimately, I get it. It's hard to do anti-cheat on PCs because of the power users have (although other OSes like macOS actually have stuff like app attestation that help in providing user-level apps some safeguards against cheats), but for me personally I value my security more than being able to headshot someone.
In order for this article to really be fair, you really should have interviewed third-party security professionals as well, aka people who work in security, but *not* in video game anti-cheat software. Obviously people who work on anti-cheat themselves are going to say their stuff works, no shit.
Great article, but the one thing you don't seem to touch on is the fact that anti-cheat just plain doesn't work. From Roblox to Tarkov, Call of Duty to Helldivers 2, GTA V to Diablo 4, there are always going to be cheaters. Most anti-cheat solutions don't even reduce the frequency of the cheaters that much, because it only takes one smart hacker to break the anti-cheat and then distribute their knowledge or software to thousands of users, who can all exploit the hole.
Folks like Koskinas might compare anti-cheat to whack-a-mole, but hey, isn't that exactly what they'd be doing if they *didn't* have an invasive kernel module loaded onto my system? And by not shipping said kernel module, they would be allowing their game to run on Mac and Linux through Wine, and eliminating any possibility that their anti-cheat kernel module might be crashing peoples' systems, which is a widely reported issue; or that the anti-cheat kernel module exposes vulnerabilities of its own and gives programs (or even servers online) low-level access to your system through an unintended backdoor.
If kernel anti-cheat actually worked, I might even accept it as a useful tool in the toolbox. But other than Valorant, which just appears to be significantly ahead of the cheaters' technology right now (but I guarantee that will change in a few months) -- most kernel anti-cheats are useless.
They're even more useless when they don't evolve and constantly improve, stepping up their anti-cheat game in the never-ending cat and mouse. nProtect is especially poor in that area, as we're about 2 weeks into Helldivers 2, and nProtect is already losing the battle against the cheaters. Cheating by memory editing is becoming very pervasive already.
Kernel anti-cheat is a solution looking for a problem, but in the end, it just ends up creating more problems. User trust problems, technical glitches, unintended vulnerabilities, Mac and Linux compatibility showstoppers, and in the end, for what? Most people with 40+ hours in Helldivers 2 at this point have seen at least one cheater; if they haven't, they're not paying attention. No, you and your team didn't manage to loot 99 rare samples on that mission.
Meanwhile.. the chinese classified information dump was almost 100% all related to online hacking and stealing of US information from various platforms... including gaming..
but yeah.. no big deal.. Lets have Tencent sponsor every US release, no harm no foul right?
/eyeroll give me a break. Anyone who ISNT concerned about PC vulnerability is a complete toolbag. Full of Dollar General products no less.
Jazz hands and well wishing isnt going to distract people from the simple fact that between Russia and China, US based systems are under constant attack 24/7 365.
How long till they find the way in? .. more importantly, why are people giving them a head start by installing unreliable and dangerous rootkit software that has proven time and time again to be dangerous to the end user?
Post all the articles you want, but the fact you completely "left out" everything we just learned about China's info leak... is a HUGE red flag. Not to be trusted.
Good article in terms of giving insight of the developers view. But the ending is a bit condescending towards end-users, while there certainly is a lot of misinformation, there is also a lot valid complaints from them. Many which developers want to sweep under the rug, but in the end, many potential users dont consider kernel-level anti-cheats to be worth the cost in privacy, PC security, closing off the system, risks of it messing with OS and hardware etc., for the moderate increase in anti-cheat capabilities.
And it makes sense for devs in kernel anti-cheat to see Vanguard as the industry leader, but the future most likely lies in server-side anti-cheats and ML.
disappointing missed interview opportunity most the article didn't talk about nGuard and when it did it didn't address much i.e. comments I seen such as: "Most ACs if they think is a cheat, close the game.
This AC, if detecting something it thinks is a cheat, shuts the 'cheat' down. If it erroneously thinks a windows process is a cheat, whoops, crash. If it thinks your CPU cooling fan is a cheat, PC may burn. It's known to cause issues where it interferes with programs it has no business interfering with, the damage it causes to hardware, the lies about how the program will uninstall with the game and that it only runs with the game, even after you manage to get rid of it, it will still leave backdoors into your system completely bypassing firewall and AV, its also the fact that this malware blocks several firewalls and anti-virus softwares"
I get that if an AC was such a widespread hardware killer it may be more known though it don't help posts have been getting deleted by Helldivers2 mods about AC complaints
Paranoia aside, the funny part of using an outdated, often-proved ineffective tool such as nProtect GameGuard is that it took cheaters less than a week to discover flaws in the game protection enough to ruin some players' progression. This is a really weird call considering there are better players in the same industry, such as Easy Anti-Cheat.
Something so very aggravating about clicking through to that Steam post and seeing that the top reply basically just says "you're lying." And it even uses the word "verifiably"--naturally, it doesn't actually link or even reference any evidence that Lindgren is wrong. Maybe this is just all Discourse online now, but it's depressing how predictably and quickly gamers get whipped up into a righteous fervor. It's been like 15 years of gaming discourse feeling like a never-ending loop of this. And when you get through all the froth, there doesn't seem to be any substance. Like, I thought the complaints about Steam Deck compatibility were legit, and then I realized the post was weeks old and you can play the game just fine on Steam Deck (a tad bit finicky but that's very normal with the Steam Deck)… the complaints were pre-release and purely hypothetical. And, as has often been in the case in the past, I'm guessing there won't really be any evidence for the performance issues either.
Anyways, I really appreciated your piece and have subscribed. Good to hear from people in the industry and in particular I'm glad Riot has been willing to lift the veil a tiny bit. I will say that as someone who is especially paranoid about my security and privacy, I do dislike the idea of kernel anti-cheats, but I recognize I'm talking about a really marginal risk here and one that even paranoid people like me are already accepting on a daily basis. I don't play any games that use 'em, but that's not a principled boycott or anything. I just haven't fallen in love with one of those games yet. Though Riot's about to expand Vanguard to LoL, so that might change soon.
I will say that even though I know it's a great slam dunk for anti-cheat, I hate that Riot's Vanguard implementation requires it to be continuously on from device startup. That's a lot of trust I'm handing over in the sense that you really gotta believe they've done their best to avoid performance impacts, compatibility issues, and security risks. But in the age of SSDs and 10 second restarts, even my paranoid self can't really complain too much.
I love your writing style Ryan. I miss it on apex with every update.
I wonder when Koskinas made Maplestory cheats. @Ryan if you get a chance, could you ask him if he ever worked on Gamekillers Terminal or Gamersouls Blight, or if it was way earlier?
I got to really question the knowledge of those interviewed people. Someone who is saying that a software running in the application layer can do the same things as a kernel driver has either not knowledge and is ignorant about the topic OR is just LYING. Why do we even need all of those layers if they all can do the same thing anyway?
As a game developer, I would say that most peoples knows that game developers don't want to steal your data. The issue with kernel-level anti-cheat is that you end up having a lot of them installed on your computer. If only one is compromised with a supply chain attack, the attacker can access all the players computers.
A significant example of a supply chain attack was the Solarwinds cyber attack, in which their build server got compromised. Luckily for their client, Solarwinds products don't run in ring 0, so the attack impact was limited. Still, if they did, the attacker would have completely controlled all their customer's servers, including Microsoft, and all the data of Microsoft clients. If SolarWinds and Microsoft can get compromised, then a game studio or an anti-cheat developer can also have their build server compromised.
Nobody should have full access to your computer besides you because you can't audit their security and be sure they have good security practices or won't be breached.
Hi, I am a software engineer who used to work on video game (engine and graphics) so I think I have a decent understanding of how game dev works. I always roll my eyes at articles like this because when people get harmed by others, there are usually two reasons: 1) maliciousness, 2) incompetence. Game developers love to latch on to (1) and claim they have no incentive to harm their players etc, which may be true, but keep in mind we don't know what deployment or code review policies each team uses and from having worked in games before a lot of teams have a pretty lax code policy compared to other areas I have worked in (aerospace). It's not crazy to assume that backdoors could still be injected by malicious employees even if the company didn't want to.
The more important issue is (2) (incompetence). Programmers like to think of themselves as rockstars, but *everyone* makes mistakes. Having a kernel level anti-cheat significantly increases the attack vector compared to a user-level app, and if you have an unfortunate bug or supply chain attack (e.g. via a third-party package) the end result is more catastrophic. In security, we believe in defense in depth, and using a kernel level driver directly goes against that. When you talked about trust, note that you are asking for people to not just trust that you are not malicious, but that trusting you are not incompetent.
This also leads to the comment that a user-level program can access the webcam. I don't think this is necessarily true. It depends on the OS / hardware, but there are permissions that you have to seek before you can access the webcam. A kernel level driver can potentially do other things such as being able to record silently without turning on the light, etc. You essentially get to bypass all of the safeguards modern OSes have put on to prevent such malicious usages. Riot's Vanguard system also runs at boot, meaning that you are vulnerable all the time, rather than only when playing the game. Even if Vanguard doesn't talk to the internet, it talks to other programs on your PC, and those programs (including the software updater that updates Vanguard) talk to the internet. It's a layer of indirection, but if there are flaws in the way the IPC (interprocess communication) then Vanguard could be exploited all the same. The fact that it's an always-on kernel driver gives a lot of incentives for attackers to target it.
Ultimately, I get it. It's hard to do anti-cheat on PCs because of the power users have (although other OSes like macOS actually have stuff like app attestation that help in providing user-level apps some safeguards against cheats), but for me personally I value my security more than being able to headshot someone.
In order for this article to really be fair, you really should have interviewed third-party security professionals as well, aka people who work in security, but *not* in video game anti-cheat software. Obviously people who work on anti-cheat themselves are going to say their stuff works, no shit.
Great article, but the one thing you don't seem to touch on is the fact that anti-cheat just plain doesn't work. From Roblox to Tarkov, Call of Duty to Helldivers 2, GTA V to Diablo 4, there are always going to be cheaters. Most anti-cheat solutions don't even reduce the frequency of the cheaters that much, because it only takes one smart hacker to break the anti-cheat and then distribute their knowledge or software to thousands of users, who can all exploit the hole.
Folks like Koskinas might compare anti-cheat to whack-a-mole, but hey, isn't that exactly what they'd be doing if they *didn't* have an invasive kernel module loaded onto my system? And by not shipping said kernel module, they would be allowing their game to run on Mac and Linux through Wine, and eliminating any possibility that their anti-cheat kernel module might be crashing peoples' systems, which is a widely reported issue; or that the anti-cheat kernel module exposes vulnerabilities of its own and gives programs (or even servers online) low-level access to your system through an unintended backdoor.
If kernel anti-cheat actually worked, I might even accept it as a useful tool in the toolbox. But other than Valorant, which just appears to be significantly ahead of the cheaters' technology right now (but I guarantee that will change in a few months) -- most kernel anti-cheats are useless.
They're even more useless when they don't evolve and constantly improve, stepping up their anti-cheat game in the never-ending cat and mouse. nProtect is especially poor in that area, as we're about 2 weeks into Helldivers 2, and nProtect is already losing the battle against the cheaters. Cheating by memory editing is becoming very pervasive already.
Kernel anti-cheat is a solution looking for a problem, but in the end, it just ends up creating more problems. User trust problems, technical glitches, unintended vulnerabilities, Mac and Linux compatibility showstoppers, and in the end, for what? Most people with 40+ hours in Helldivers 2 at this point have seen at least one cheater; if they haven't, they're not paying attention. No, you and your team didn't manage to loot 99 rare samples on that mission.
Meanwhile.. the chinese classified information dump was almost 100% all related to online hacking and stealing of US information from various platforms... including gaming..
but yeah.. no big deal.. Lets have Tencent sponsor every US release, no harm no foul right?
/eyeroll give me a break. Anyone who ISNT concerned about PC vulnerability is a complete toolbag. Full of Dollar General products no less.
Jazz hands and well wishing isnt going to distract people from the simple fact that between Russia and China, US based systems are under constant attack 24/7 365.
How long till they find the way in? .. more importantly, why are people giving them a head start by installing unreliable and dangerous rootkit software that has proven time and time again to be dangerous to the end user?
Post all the articles you want, but the fact you completely "left out" everything we just learned about China's info leak... is a HUGE red flag. Not to be trusted.
Good article in terms of giving insight of the developers view. But the ending is a bit condescending towards end-users, while there certainly is a lot of misinformation, there is also a lot valid complaints from them. Many which developers want to sweep under the rug, but in the end, many potential users dont consider kernel-level anti-cheats to be worth the cost in privacy, PC security, closing off the system, risks of it messing with OS and hardware etc., for the moderate increase in anti-cheat capabilities.
And it makes sense for devs in kernel anti-cheat to see Vanguard as the industry leader, but the future most likely lies in server-side anti-cheats and ML.
disappointing missed interview opportunity most the article didn't talk about nGuard and when it did it didn't address much i.e. comments I seen such as: "Most ACs if they think is a cheat, close the game.
This AC, if detecting something it thinks is a cheat, shuts the 'cheat' down. If it erroneously thinks a windows process is a cheat, whoops, crash. If it thinks your CPU cooling fan is a cheat, PC may burn. It's known to cause issues where it interferes with programs it has no business interfering with, the damage it causes to hardware, the lies about how the program will uninstall with the game and that it only runs with the game, even after you manage to get rid of it, it will still leave backdoors into your system completely bypassing firewall and AV, its also the fact that this malware blocks several firewalls and anti-virus softwares"
I get that if an AC was such a widespread hardware killer it may be more known though it don't help posts have been getting deleted by Helldivers2 mods about AC complaints
Paranoia aside, the funny part of using an outdated, often-proved ineffective tool such as nProtect GameGuard is that it took cheaters less than a week to discover flaws in the game protection enough to ruin some players' progression. This is a really weird call considering there are better players in the same industry, such as Easy Anti-Cheat.