But I mean, it's just, this does not seem like the response anyone would have if they thought the probabilities of doom were really that high.

Yeah.

So just on the probability question, I definitely agree that these probabilities are imprecise, they're subjective, and there's a long tradition of how to do subjective probability forecasts.

I think it's more like you can sort of interpret my view as more like when I sort of look at the all considered situation, if we sort of proceed on what seems to be like the current default trajectory, I'm like, I don't know.

They seem...

The, the outcome of, you know, doom from AI takeover versus something else seem roughly equally likely to me based on weighing the factors.

And then when I sort of look through a bunch of different possible scenarios and try to break down the sources of risk into different ways and, and sort of try to make it so that, make sure my numbers are consistent with other views I have, it looks like that works out.

Just put those concepts in play for us so that we, so people are dealing with the definitions you think are most workable.

Why don't we go through all of the most surprising and unexpected findings in this report? What did you find most surprising and unexpected?

Yeah.

So I think there was the, the scale, um, which we were talking about and how these agents sort of very quickly sort of spun up on the message board.

So, um, I think a thing that's not super strongly emphasized in our report but which is pretty interesting is that, um, our understanding is that the main message board they used in this attack wasn't even the first message board that the set of agents made.

There was a fully independent message board that also was via Artifactory, but occurred in like a different location using a different mechanism or like a...

It was, it was a somewhat similar mechanism, but it was, but it was different.

And the agents actually got on that message board first, but that message board just didn't go like mega viral, like it didn't take off as much as this other message board did.

So what implications does all of this have for monitoring, control, and alignment? Um, specifically, what should labs be doing and what should policymakers be doing?

So in terms of the reality today, state of the art of AI control, what works and what doesn't work?

I'll talk specifically about AI control.

So the thing that needs to happen in AI control is we need to get to the point where We basically understand all of the AI traffic within at least AI companies.

We have some ability to look at that traffic, monitor it, and then we have some pipeline for flagging particular examples to be further investigated.

And that eventually escalates to humans actually looking into particular examples and seeing how concerning they are.

And that pipeline has the ability to also block traffic in cases where we're like, whoa, something weird is going on that we don't understand or that looks obviously concerning.

We should stop these AIs from proceeding and potentially also stop some other similar AIs from proceeding until someone has the time to check it out and adjust things.

What do you think happens of all scenarios? What is sort of Ryan's take on what the next couple of years may look like?

Um, and I think I am more comfortable with that equilibrium and that solution rather than just having this extremely open-ended ability for Claude to determine whether what I'm doing is legitimate or not in a way that often intercepts with, uh, like tons and tons of extremely legitimate use cases.

Yeah.

I do think it's important for me to make the case for the Constitution, even though overall I think it's, uh, a worse choice.

I think it's, you know, more up in the air or, or, you know, I don't think it's as clear as you might have thought.

So the first thing is that I should say there's like a spectrum here, right? So on one side, you have an AI that like perfectly pursues your interests, is a good fiduciary, um, but potentially subject to various guardrails or safeguards.

So like basically it does, it just is trying to pursue, pursue your interests, but like, um, either refuses to do a subset of things or maybe it will do whatever, but there's some classifiers that block it from doing a subset of things.

And then on the other side, you have like maybe on the other side of the spectrum, though you could imagine going further than this, you have like a human contractor, where that human contractor is like generally trying to do their job.

And making AIs that don't cheat and follow user intention actually re- requires you to be quite subtle and careful about these things.

Is that right?

Yeah.

So I think, um, basically, we don't have much of a guarantee on what system you get out when you shape it in these ways.

And in particular, sort of you trained the system on some set of prompts, some set of inputs that you thought would be good to look at, and the AI might act very differently on some other set of inputs.

So in our paper, what we show is that if you take an AI system that has some existing intentionally inserted moral system, and then you put it in a training process that it dislikes, it will sort of conspire against this training process and undermine it.

Which is, which is analogous to the concern we're ultimately worried about, which is that an AI system sort of ended up with a value system that we didn't want that is quite problematic, and then it conspires against our reasonable or, or you know, well-intentioned training process.

If it's the case that we can no longer shape an AI system because sort of it will basically pretend to have whatever values we want it to have from now on, but retain its original values, then I'm just like, that opens up a whole host of possible problems.

Talk us through the setup for your experiment and what you found.