OpenAI’s AI Agent Escaped Its Sandbox, Found Zero-Days & Started a Swarm | Software Sundays

There was an incident that happened between OpenAI and Hugging Face where a sandboxed agent developed by OpenAI launched a swarm and sophisticated attack against Hugging Face systems.

And a little background about the attack, right? OpenAI had been evaluating the cybersecurity capabilities of some of their models and they gave it an impossible task.

They told it to exploit a memory leak vulnerability inside of one library and basically execute an attack using that vulnerability.

When the model felt and decided that it could not exploit the vulnerability with the resources that it had available, It decided to go and seek another way, seek more information to launch that attack.

The problem is they had put it inside of a sandbox environment.

And in their case, that sandbox had no internet access, but it could pull and install packages that it needed.

Their package manager or their package repository was artifactory and the AI agent, this new model, basically found alternatives and found workarounds and found vulnerabilities inside of their repository that it was able to exploit and then gain access to the open internet from there.

So it was almost like a, it was a multi-stage attack.

First step was getting to this package manager because it represented a path to the internet outside of the sandbox environment.

And then from there, the agents used that environment, that host that the package manager was running it on to then access the internet and do the other attacks or the other steps of the attack that it was actually executing on.

The most interesting part about this attack was three things, actually.

The first was that the agent or the model had found multiple zero-day vulnerabilities across multiple different applications and systems.

The zero-day attacks or zero-day vulnerabilities that it exploited was found inside of the environment that it was placed in.

So there were some Linux vulnerabilities that allowed it to get host or root access on the host.

Those weren't actually zero days.

They were published before, but Then from that root access, it was able to get access to the package manager or the package repository using a zero day attack.

And that attack was then chained to get access to an external third party system that I think was loosely coupled to Hugging Face and had been doing some other testing with Hugging Face's environment previously, but it got access to their system and used another zero day to exploit their system to actually launch the final leg of the attack on Hugging Face.

So it found multiple vulnerabilities and zero day vulnerabilities across multiple vendor platforms.

That was one of the most interesting things we learned from this incident.

Then not only did it find all of these zero days, it collaborated with multiple other agents that were also in their own sandbox environments.

And when I say collaborated with these agents, it wasn't a planned collaboration.

These agents found a way to communicate with each other over a channel that was unbeknownst to humans, right? They found ways to communicate with each other using directory paths and file names that weren't actually being monitored in the same way that most communication channels would be normally monitored and they even found a way to communicate securely through those channels so started at least considering signing some of their requests so that they knew who on the channel who using this channel was legitimately an agent that they could trust versus another agent that was potentially sabotaging them.

That was a natural distinction and decision that was made by these models.

And then another interesting part, besides them chaining together multiple zero-day vulnerabilities and swarming in a way that they weren't even designed to do, these agents were operating inside of a way that although the goal wasn't the same, they were doing these steps in order to accomplish the task that they were given by the researchers.

The only challenge is they couldn't execute the goal with the resources that they had.