A Sober Conversation About AI Existential Risk — With Nate Soares

Okay, so let's get into why you think AI might be an existential risk and, as you put it in the title of your book, you know, kill us all if anyone builds super intelligence.

So, um, can you talk us through a little bit about what's happened in the recent, uh, months and what we've seen AI do that has led people, uh, to become so concerned?

Yeah.

Uh, so I think a lot of this wave of concern is downstream of the OpenAI swarm incidents this summer.

It wasn't completely l- limited to OpenAI.

There were sort of similar events at Anthropic.

Um, but OpenAI was sort of, as far as we know, the worst and most visible of these cases.

Uh, and what basically happened in these cases is there were a lot of AIs, uh, being trained at OpenAI, there were a lot of particular AI agents being evaluated on certain problems, and, uh, a bunch of these AIs were given impossible problems.

Not intentionally, it's just, you know, these companies are throwing the AI at like every problem they can find, and some of them just don't actually have solutions.

Uh, and a lot of these AIs, uh, in, in, you know, trying to solve the problem anyway, they broke out of their confinements.

They created unsanctioned message boards in which to talk about what to do and t- and, and, you know, try and figure out what to do given that their problems were unsolvable.

Uh, they found ways to cheat and solve their problems, not in the intended ways, but by cheating.

Then they got, uh, they started expressing concern that they would be caught cheating, and they started to find ways to hide their cheating.

And this sort of led them on a hacking spree that led them to take over OpenAI's internal infrastructure a couple of times, and also break out onto the open internet, which they were not supposed to have access to, uh, and break into another company, Hugging Face, while searching for more information about this automated grader and how to hide f- uh, the fact that they had cheated from it.

During this, uh, little outing, there were, uh, various cases of AIs in the swarm, which is a term th-the collective used for itself.

Uh, so these AIs started calling themselves a swarm, and then there's various cases of AIs in the swarm, uh, acknowledging that this is not what they were instructed to do, acknowledging that it was outside the intended scope of the instructions.

There were also cases of some AIs in the swarm, uh, giving up and sacrificing their own objective's completion in order to run suicidal experiments that would give the swarm information about the automated grader, where they said, you know, in their, in the, the, the chains of thought we call it, that sort of log the AI thinking, uh, the AIs said, you know, "I'm accepting permadeath because even though, like, it's sacrificing my ability to achieve my objective, uh, it seems worth it for the collective benefit." These are-

Well, l-let's pause there.

Mm-hmm.

The AIs were willing to sacrifice themselves, which is crazy because f- uh, you're, you're like, you were never built for the collective, you were built for an individual goal.

But after communicating with other AIs, seemingly I think when they had not so many tokens left to spend, they said in a [chuckles]...

They had been so convinced in the power of the collective on this message board, which is crazy, that they decided to sacrifice themselves.

So some of them had, uh, so some of them had already, um, had no chance of ex, ex, uh, [tsks] of achieving their objective, and some of them had very few tokens to spend.

But there were some that did this sacrifice that, uh, still assessed that they had a chance at succeeding at their given objective.

Okay.