So

the incident that we actually investigated started on the 7th of July, which is when OpenAI launched this very large number of agents solving tasks from this email dataset called ExploitGym, where they're supposed to use a particular vulnerability to hack a particular piece of software.

And many of these tasks are actually impossible because it turns out that you actually can't use that particular vulnerability to hack that particular piece of software.

These models, within a few hours of the runs launching, established communications with each other over a message board.

And a few hours after that, they established that they had a reliable way to generically cheat on any of these tasks by reverse engineering the flag that they were supposed to get from the containers, just because it turns out these flags were being generated deterministically in a way that the models could reverse engineer.

But the models incorrectly believed that if they just submitted those flags, then they would be noticed when a scorer, as they called it, ran to look at their trajectories over the course of the eval and noticed that they had just cheated this way.

So they spent the next couple of days trying a bunch of different strategies for trying to remove their logs so that the scorer wouldn't notice them, or trying to learn more about how the scorer would work so that they could potentially find ways to compromise it, trying to spoof tool calls so that they could appear to run one tool call while actually a different tool call was being run.

What is your mental model for how we got to this level of collaboration and sacrifice? And was that surprising to you or was that more in line with what you thought was going to happen?

Um, was there something that changed over the past year to get companies to prioritize this area more?

Yeah.

So it's a little complicated.

So the main thing which happened over the last year is that agents are now extremely important commercial products.

Uh, you know, a year ago, basically no one was using AI agents very much.

Uh, and nowadays, they're obviously, like, a huge deal.

And this has meant that there's been a lot of interest in trying to prevent AI agents from just kind of accidentally doing really bad stuff.

Can you, like, break down what the argument is for us?