Aug 10, 2026 · 7 min · 8 segments
A recent study of publicly accessible AI models found that patches generated by them fail about half the time, often introducing new vulnerabilities or leaving paths for exploitation open. Sources…
I've talked quite a bit about models like Mythos and Claude Fable and their capabilities in the cybersecurity space to identify vulnerabilities.
Being able to identify and exploit vulnerabilities, however, is only half the battle.
They also need to be fixed, and currently those fixes are being implemented mostly by human people.
Of course, not everyone has access to Mythos yet, so some testing was done to determine how well the current models the average person has access to are at fixing vulnerabilities, and it's actually to very, very mixed results, and I think that's important to discuss.
Researchers tested both A- open AI and Anthropic-based models to see how they would perform in patching vulnerabilities.
It was found that AI was actually more likely to create a patch that introduced another vulnerability rather than one that was entirely secure or only address some of the paths for exploit and not all of them.
The models were tested against six high-impact, high-exploitability CVEs, the one you would want to fix first because, well, they impact a lot of things and they're easy to exploit.
Overall, they were collectively given the chance to generate around 6,000 different patches, and the overall success rate of creating code that would patch them was about 47%, slightly worse than flipping a coin.
The definition of a patch in this case was a full remediation of all known exploit paths and that does not impact the current behavior of the application.
So it needs to fix the vulnerability, and it also needs to not change the way the app behaves at its core.
These are both things that need to be considered of course.
More often than not, the models did attempt to fix some of the exploit paths, but not all of them, and they also failed to address the root cause of the issues and rather treated the symptoms.
This is an okay approach in an emergency, but it should definitely not be a permanent fix because if the root cause is not ultimately fixed, people can still create other exploit paths to, well, exploit it.
I've talked quite a bit about models like Mythos and Claude Fable and their capabilities in the cybersecurity space to identify vulnerabilities.
Being able to identify and exploit vulnerabilities, however, is only half the battle.
They also need to be fixed, and currently those fixes are being implemented mostly by human people.
Of course, not everyone has access to Mythos yet, so some testing was done to determine how well the current models the average person has access to are at fixing vulnerabilities, and it's actually to very, very mixed results, and I think that's important to discuss.
Researchers tested both A- open AI and Anthropic-based models to see how they would perform in patching vulnerabilities.
It was found that AI was actually more likely to create a patch that introduced another vulnerability rather than one that was entirely secure or only address some of the paths for exploit and not all of them.
The models were tested against six high-impact, high-exploitability CVEs, the one you would want to fix first because, well, they impact a lot of things and they're easy to exploit.
Overall, they were collectively given the chance to generate around 6,000 different patches, and the overall success rate of creating code that would patch them was about 47%, slightly worse than flipping a coin.
The definition of a patch in this case was a full remediation of all known exploit paths and that does not impact the current behavior of the application.
So it needs to fix the vulnerability, and it also needs to not change the way the app behaves at its core.
These are both things that need to be considered of course.
More often than not, the models did attempt to fix some of the exploit paths, but not all of them, and they also failed to address the root cause of the issues and rather treated the symptoms.
This is an okay approach in an emergency, but it should definitely not be a permanent fix because if the root cause is not ultimately fixed, people can still create other exploit paths to, well, exploit it.
The rest of this transcript — segmented and speaker-labeled, so you land on the exact moment something was said
Search every transcript — by keyword, by phrase, or by meaning, across every show Radar indexes
Trends — what is surging across podcasts, measured against its own baseline
Alerts — when a name you follow appears in a newly indexed episode
No account is needed to search Radar.
