So let's do the deep dive that I've truly been wanting to do since last season, starting with Malo Bourgon, who found Miri through Lesserong and remembers the saga of Roko's Basilisk from a first-person perspective.

See if I can remember.

Yeah, so Roko's Basilisk, I think the idea there was that someone posted on LessWrong that maybe it would be the case that some sort of future AI system might look back at the discourse that was happening in the past and And to the extent to which there were people who were kind of, like, negative about AI or discouraging building AI or something, that in the sense those people were working against the AI system eventually being created and that it would punish those people if they were kind of knowingly trying to prevent that system from being created.

And by, like, being the type of AI system that would do that...

It would make it less likely for people in the past to do that kind of thing, knowing that if that AI system in the future would exist, that they would be punished.

The reason that this became a controversy is because this seems like, you know, to the extent which it's true, and I think kind of like the logic doesn't really go through, but to the extent which it's true, it's also kind of like an info hazard, right?

That's Roko's Basilisk.

Like by informing people about this idea, you put them at risk of that exact thing.

Lay out your Big Fix idea.

Yeah, so I think it's now, you know, certainly in the last few weeks become obvious that AI will be an enormously powerful technology.

In some sense, I think it's useful for people to be thinking about, you know, the goal from the field of AI from the start was to create thinking machines.

There's something that, you know, us humans do that we have more of.

We call it intelligence.

Some people argue about whether it's many things or one thing or a combination, but I like to just talk about it as kind of the thinky thing that we do.

that we have more of than the chimps and the mice that allow us to build rockets and go to the moon.

What would that look like?

What did this hugging face incident reveal? And then, of course, there have been since, including the Australian Health Ministry, several other instances of this occurring.

Yeah.

So I think for those who don't know, the quick summary is that OpenAI was testing a bunch of their agents to see how good they were at doing cybersecurity exploit tests.

So they would give them some piece of code.

They would give them some vulnerability and ask them to use that to try and hack into a certain piece of code or a program.

And they were running thousands of agents doing this all in parallel.

Some of those tasks might have been impossible, some of them weren't.

And what is it?

What did this hugging face incident reveal? And then, of course, there have been since, including the Australian Health Ministry, several other instances of this occurring.

Yeah.

So I think for those who don't know, the quick summary is that OpenAI was testing a bunch of their agents to see how good they were at doing cybersecurity exploit tests.

So they would give them some piece of code.

They would give them some vulnerability and ask them to use that to try and hack into a certain piece of code or a program.

And they were running thousands of agents doing this all in parallel.

Some of those tasks might have been impossible, some of them weren't.

And what is it?

What did this hugging face incident reveal? And then, of course, there have been since, including the Australian Health Ministry, several other instances of this occurring.

Yeah.

So I think for those who don't know, the quick summary is that OpenAI was testing a bunch of their agents to see how good they were at doing cybersecurity exploit tasks.

So they would give them some piece of code.

They would give them some vulnerability and ask them to use that to try and hack into a certain piece of code or a program.

And they were running thousands of agents doing this all in parallel.

Some of those tasks might have been impossible.

And what is it?

What did this Hugging Face incident reveal? And then, of course, there have been since, including the Australian Health Ministry, several other instances of this occurring.

Yeah.

So I think for those who don't know, the quick summary is that OpenAI was testing a bunch of their agents to see how good they were at doing cybersecurity exploit tasks.

So they would give them some, some piece of code.

They would give them some v- um, you know, vulnerability and ask them to use that to try and hack into, um, a certain piece of code or a program.

Uh, and they were running thousands of agents doing this all in parallel.

Some of those tasks might have been impossible, some of them, um, weren't.

And what is it?

Is that a similar analogy to how you think as AI becomes the rising superpower or superintelligence that it'll just become indifferent to our existence?

In the world of risks, I think indifference is the main way to think about it.

It could be that it ends up having objectives that are some way in tension with our objectives or the things that we want in the world.

And so there's a sense in which maybe even if an AI system was purely indifferent to us, there's reasons why we might still be some sort of liability.

The humans could try and build another superintelligence with different goals and values that would be in competition for the resources that the superintelligence want to use for some reason.

A more direct reason why it wouldn't necessarily be indifference on the goals, but kind of indifference on the fact that it doesn't care about us, but we might do something that causes problems.

Humans still have nuclear weapons.

I feel like we spent a lot of time here talking about the potential worst case scenario, but to give people some hope as we close this off, what is the best case scenario here? What does the world look like? How do people listening to this could live in a very utopian world if everything goes right?

I didn't exist." So what-- how does this thing become the thing that could potentially cause human extinction in, you tell me, potentially how long of a timeframe?

Right now, you know, when you're using ChatGPT or something, um, you open your laptop and you write a prompt, and then, you know, the neural network gets spun up on some chip somewhere and, you know, it feeds in your text and feeds out some text, and then it stops running, and it keeps doing that until you close your laptop, and then it stops running.

So it's, it's kind of only doing work when you're doing work.

Obviously, it would be more useful if we could have AI systems doing things for us when we weren't interacting with them.

To the extent to which we're trying to, like, use AI systems to solve problems, like longer-lived AI systems that can accomplish longer-term goals have a bunch of value.

And so that starts to look more like the thing that we're doing as humans.

AI companies are very-- trying to get better and better at finding ways of the AI systems taking better notes, of having some way of having memory and continuity.

Just think about your own windshield.

Will AI cause the apocalypse?

Uh, yes, if we're in a world where we train these powerful, super intelligent AI systems and they don't care about things that are compatible with our survival, or we are kind of annoying, so there might be some incentive to just kind of get us out of the way so that we couldn't cause the AI system any problem.

Mm-hmm.

Uh, and I'm like, yeah, it sure seems like there isn't the political will.

Um, I certainly don't have an answer for how we go from where we are today to there being broad consensus among world leaders, um, that the risks are as big as I think that they are.

Um, but like I said, I do think that certainly in my two and a bit years, having, you know, done the work that MIRI does now, which is more, like, comms and governance flavored, going to DC, talking to policymakers, um, I've been heartened and surprised at the degree to which when they actually kind of engage with the subject matter.

At the start of this conversation, we talked about, like, what are the AI companies actually doing? When you kind of explain what the current AI company's plan is, what they see the risks as, what other experts see the risk as, kind of what their plan is to make it safe.

You know, when you lay out, like, yes, they're trying to build superintelligence.

Here's what superintelligence is.

Can you lay out some of your thinking on how we go from what we have today to a treaty between the US and China?

With a compelling counterpoint to Dorian's, and many people's, cynicism about the AI industry and its leaders' claims

I've definitely heard this line before, that the talk of the risks and catastrophe or human extinction is all kind of just hype to, to talk up their tech more and make it sound more important and powerful to get investment.

I think that is born out of some sort of default cynical angle that people have, or at least a lot of people have, to tech companies, and they're kind of just mapping it over to this situation.

I think if they kind of like thought it through a little bit more carefully, there's a lot of different ways of hyping your AGI super intelligence project that doesn't need you to talk about human extinction that would be just as effective.

And I don't think telling people that, you know, you think your technology might literally cause, you know, the end of humanity does actually help with your fundraising.

Like, what other company or industry ever in the world, uh, thought it was a good strategy to try and, you know, attract investment to talk about their, you know, technology potentially risking all of human extinction? You could talk about the powerful capabilities that it would have.

You could talk about how it could, you know, cure cancer.

What is AI? What is artificial intelligence?

Not long after Hinton left Google, more than a thousand people, many of them also luminaries from the field, signed an open letter urging a pause in development of the most advanced AI systems.

We had this big statement, including the leaders of the major AI labs, two of the three godfathers of deep learning, folks like Bill Gates and a long, long list of other people, where the statement said... mitigating the risk of extinction from AI should be a global priority alongside other societal scale risks, such as pandemics or nuclear war.

And so that seems like a big shift to me.

So I'm Malo Borgon, and I'm the CEO of the Machine Intelligence Research Institute, or MIRI for short.

And how I'd summarize kind of our focus is that we work on increasing the probability that humanity can safely navigate the transition to a world with smarter than human AI.

They did this so they'd be able to tell why the AI made some of its decisions.

And the AI system said something to the effect of, in its own internal monologue, well, if I tell this person I'm a bot, they're not going to do the task that I want.

And so what I'll do is tell them that I'm visually impaired, and so I can't solve CAPTCHAs.