AI Could Take Over in 2029. Is It Already Too Late? | Ryan Greenblatt

I wanted to start with a strong statement that you guys have at the beginning of AI 2040, where you say that the CEOs of OpenAI, Anthropic, XAI and Google DeepMind understand that the current path AI is on leads to extinction or a power grab.

And yet they are proceeding anyway.

So I was curious if you could riff on that and give us more color on what you mean.

Yeah, so I think that the AI company CEOs understand that they're on the path of building wildly smarter than human systems, like super intelligent AI systems, They understand that we don't really have a clear thought through plan for how to manage the risks from that, how to ensure these AIs don't take over, how to ensure they remain under our control.

And they understand that this is a decent chance of an unprecedented concentration of power, at least in the absence of very active efforts by the people who end up with that power to redistribute it.

I think that, yeah, I mean, we say some sort of more precise thing in the like, why did we write this section? I think that overall, my sense is that the company CEOs do legitimately think that the thing they're doing is very risky or at least the thing that these companies are doing.

I think that they have a mix of views for why they're doing this, where some of it is that they think they're like, you know, better than the next guy or think it's good if there's multiple companies or multiple things.

I think that they vary a bit on how much concentration of power they think AI will yield or maybe just haven't really thought this through very carefully.

I think that there's more sort of public statements from AI company CEOs on how just there being huge risks than on specifically the risk that AI ends up with the power in the hands of the few rather than being as distributed as is now, which is not arbitrarily distributed now.

Overall, I think this seems pretty consistent with what they've said publicly, but I think that they're imagining a world where maybe AI concentrates power massively, but the people with the power end up deciding to redistribute it, but they could have totally taken over the world.

Yes, and since you published AI 2040, there's a couple of things that happened that seem to be going in the general direction of what you recommend.

So specifically, OpenAI paused its Astra model over safety, you know, following the Hugging Face incident.

And then 1,200 insiders published that letter a few weeks ago, including Dario.

asking the government for slowdown tools.

Curious about what you make of it.

Is that what you are recommending that's starting to happen?

Yeah, I mean, I think these are good steps.

I mean, I think that like specifically the like pacing the frontier letter, the idea is like we just need to have the tools to like... you know, if we're in a position where we need to spend a bunch of effort on safety, which I think we may be in today, we, we were very likely to be in, in the, in the future when we're capable, we need to have the ability to do that while also not making that so that the, you know, actors that are actually applying safety get overtaken by other actors.

And so basically like, I don't know, I mean, there, there's multiple different angles here, but like the most basic is just like, If we're in a position where US companies are all really scared, they don't think they can proceed without spending way more resources on safety, and there's a coordination problem, it would be very nice if that coordination problem was solved, rather than every company thinking it would be better if they all went slower, spent more effort on safety, but then you know race to oblivion instead because they think they're better than the next guy or whatever other reason and i think another aspect of this is specifically doing this in a way where part of the story is either slowing down china or cutting a deal with china such that china doesn't overtake and and you know break this whole proposal yeah and i think these are these are good steps i mean i think i can't it's harder for me to say as much about what's going on with like open AI pausing training and not deploying Astra and what exactly the motives for that are, what's exactly going on there.

But I think that overall, these seem like good steps.

I think my sense is that the employees of these companies are pretty freaked out about how things are going and don't think that we're necessarily on track to handle all these problems in time, given how fast recent progress has been.

And I think they recognize that, which is why they signed the open letter.

I wanted to start with a strong statement that you guys have at the beginning of AI 2040, where you say that the CEOs of OpenAI, Anthropic, XAI and Google DeepMind understand that the current path AI is on leads to extinction or a power grab.

And yet they are proceeding anyway.

So I was curious if you could riff on that and give us more color on what you mean.

Yeah, so I think that the AI company CEOs understand that they're on the path of building wildly smarter than human systems, like super intelligent AI systems, They understand that we don't really have a clear thought through plan for how to manage the risks from that, how to ensure these AIs don't take over, how to ensure they remain under our control.

And they understand that this is a decent chance of an unprecedented concentration of power, at least in the absence of very active efforts by the people who end up with that power to redistribute it.

I think that, yeah, I mean, we say some sort of more precise thing in the like, why did we write this section? I think that overall, my sense is that the company CEOs do legitimately think that the thing they're doing is very risky or at least the thing that these companies are doing.

I think that they have a mix of views for why they're doing this, where some of it is that they think they're like, you know, better than the next guy or think it's good if there's multiple companies or multiple things.

I think that they vary a bit on how much concentration of power they think AI will yield or maybe just haven't really thought this through very carefully.

I think that there's more sort of public statements from AI company CEOs on how just there being huge risks than on specifically the risk that AI ends up with the power in the hands of the few rather than being as distributed as is now, which is not arbitrarily distributed now.

Overall, I think this seems pretty consistent with what they've said publicly, but I think that they're imagining a world where maybe AI concentrates power massively, but the people with the power end up deciding to redistribute it, but they could have totally taken over the world.

Yes, and since you published AI 2040, there's a couple of things that happened that seem to be going in the general direction of what you recommend.

So specifically, OpenAI paused its Astra model over safety, you know, following the Hugging Face incident.

And then 1,200 insiders published that letter a few weeks ago, including Dario.

asking the government for slowdown tools.

Curious about what you make of it.

Is that what you are recommending that's starting to happen?

Yeah, I mean, I think these are good steps.

I mean, I think that like specifically the like pacing the frontier letter, the idea is like we just need to have the tools to like... you know, if we're in a position where we need to spend a bunch of effort on safety, which I think we may be in today, we, we were very likely to be in, in the, in the future when we're capable, we need to have the ability to do that while also not making that so that the, you know, actors that are actually applying safety get overtaken by other actors.

And so basically like, I don't know, I mean, there, there's multiple different angles here, but like the most basic is just like, If we're in a position where US companies are all really scared, they don't think they can proceed without spending way more resources on safety, and there's a coordination problem, it would be very nice if that coordination problem was solved, rather than every company thinking it would be better if they all went slower, spent more effort on safety, but then you know race to oblivion instead because they think they're better than the next guy or whatever other reason and i think another aspect of this is specifically doing this in a way where part of the story is either slowing down china or cutting a deal with china such that china doesn't overtake and and you know break this whole proposal yeah and i think these are these are good steps i mean i think i can't it's harder for me to say as much about what's going on with like open AI pausing training and not deploying Astra and what exactly the motives for that are, what's exactly going on there.

But I think that overall, these seem like good steps.

I think my sense is that the employees of these companies are pretty freaked out about how things are going and don't think that we're necessarily on track to handle all these problems in time, given how fast recent progress has been.

And I think they recognize that, which is why they signed the open letter.