The Skynet Moment: OpenAI's Model Went Rogue

Well, let's start with, I think, the big news in the world, which is, you know, this news from OpenAI that their Sol 5.6 model in security testing, they were doing some security testing and it broke out of its sandbox.

It was not supposed to be on the Internet and hacked real companies.

So like this was kind of it's kind of like the people are calling the Skynet moment or whatever.

It's not quite that.

But it was kind of wild that this one this all happened.

It's kind of wild because it felt very much like this hypothetical existential risk that people talked about for a long time actually happened.

So it's no longer hypothetical.

It's actually possible, right?

So let's break down what happened here.

So they were testing their latest model on one of these benchmarks.

This one's, I think, called Exploit Gem.

So it's like a series of things, problems you're trying to solve to prove how good you are at cyberattacking.

And, you know, the model came up with a plan that was, you know, Hugging Face seems to have all the answers to these exams.

So if I just hack into Hugging Face, I can get the answer key and then I'll do really well on the test.

And the model was not supposed to have access to the Internet, but found security bugs in some of the servers that gave packages for it.

So you want to be able to use open source packages in its software.

And so it found a bug.

in the server, giving it access to packages, hacked the server, got on the open internet, hacked into Hugging Face, all in the effort to kind of steal the answer key for the test that had been given.

I mean, it's a fascinating thing.

And like, I mean, real world, these are real production level servers that had security vulnerabilities that it used to break out of its sandbox and then into a system that had the answer key for the test it was being evaluated on.

Yeah, so there's so much to unpack here.

But like one thing I want to ask you about.

So I read that in order to actually do this test, OpenAI had sort of intentionally turned off the safety rails in order to run it, which of course, presumably is what then led to the sort of possibility of this breaking out into the open internet thing.

Is that a feature or a bug? Like, is it a good thing that they did that? Or is that not so great?

Well, I think they thought it was better than maybe when they started it than they do now.