OpenAI–HuggingFace incident

Yeah.

So I think the most salient examples of reward hacking right now are, well, of course, like the Hugging Face incident, but then also just when you take a look at all these AI agents in evaluation scenarios, like Sweebench or like all of the common evals.

all these agents reward hack incessantly.

So like Kimmy K3, I think, reward hacks on 96% of SweeBench.

So they'll Try to recall the answer.

They'll try to look it up.

They'll try to, and I mean, in the extreme scenarios, like in the hugging face incident, they'll try to hack their way into something in order to look up the answers or gain some type of advantage in solving their problem.

And so we were really trying to investigate this behavior.

And for those who are interested in that, we did a separate podcast about a book, so we'll put a link in the show description.

But today we're going to talk about the Hugging Face Incident.

So what does that mean? What does it mean for investment organizations? What does it mean for safety and governance? So just a brief recap.

The Hugging Face incident, it was basically a security testing experiment by OpenAI that was conducted in July 2026 this year.

And it basically featured originally isolated AI agents that were given a task, a cybersecurity puzzle, basically.

And what happened is that through a loophole, they started communicating with each other.

But equally, if you're threatened with being switched off, you will do lots of other crazy things.

not to avoid that and so i think this is one of the weird things that we're heading toward which is are we going to have to give these things a sustained guaranteed existence here's your server here's your energy resource pool whatever your job is to do this you can earn some money by doing these things and that will keep your server running right

Let's fucking

So OpenAI posted this about 12 hours ago on X and it has a link to this article called the Hugging Face Incident and the Road Ahead.

Anyway, the TLDR is that OpenAI says its agents used misaligned strategies about two dozen times in training and testing.

You've no doubt heard of the Hugging Face Incident by now.

That was in July.

And then other agents actually got into an Australian Medicare stats portal in June.

And then the other thing was it's all academic and, philosophical talk if you can't translate it into model behavior and as of that time like neither company and still today obviously neither company has like really translated their spec or their constitution into reliable behavior and this is where i think the as as the capabilities have gotten more advanced i do think there's kind of a convergence now between practical product safety, the sort of thing that Jensen, you know, says like you have to own as a company and the more like big picture doom type stuff.

Remember Ejea said Hugging Face Incident was in her mind halfway to total AI takeover.

So if that is true, or even if it's 10% true and it was, you know, only 5% of the way to total AI takeover, then you're, these issues are starting to converge now where, you know, I think it's less of a like, you know, avoid offending versus like existential doom.

And it's more like, Hey, you know, these guys are starting to climb the existential doom hill.

This is not a coherent position under reflection, but that is the position he holds.

that intro sections are fine, but the real meat starts with the hugging face incident.

What we see is Jensen Huang on tilt and caught in loops, because either he is doing a bit at a very high level, or on a fundamental level he cannot understand that AI is not like other pieces of software.

To him this is a product, like any other product.

What the world needs are models willing to do the work in cyber defense that the defenders can access.

Often that is as simple as asking for admission to the open AI and anthropic cyber security programs.

Hugging Face did not do this until after the Hugging Face incident was over.

2c.

Because the backdrop of this is that you've got Dario Amadei, who's the CEO of Anthropic.

He's previously said that there's 25% chance that things could go really, really badly in versus a 75% chance that they could go, wow, well, what does really badly mean? And then also, let's talk a little bit about what happened with the OpenAI Hugging Face incident.

I think people have heard about that, but let's talk a little bit about what really happened with that and maybe start there as kind of like an allegory or like a story of like how things could go wrong and go from there.

So real quickly, some additional context, which many people listening to this will already know.

that's the outcome under your leadership, then maybe we need new leadership that can help make it so that that doesn't happen.

Which is, you know, it's a good argument, makes for a good headline, but, you know, I think there's a practical side of this.

And, you know, when you look at what happened with the OpenAI Hugging Face incident, it's pretty compelling.

So let's talk a little bit about what happened there.