Yeah.

I mean, to cut to a little of that, Chase, I do think AI and I'm heartened to see a number of people articulating the same sentiment that AI is going to demand some degree of international coordination.

I mean, only this week as we taped this, there was a letter signed by more than a thousand people in the industry, including very prominent people.

saying, calling on the U.S. government to at least set the stage for, start thinking about the challenge of international coordination.

But I do think this is kind of related to the question you did raise about the direction of evolution in a way.

And when I say direction, again, I'm not talking about anything spooky.

I think you can describe this, you can explain this direction in Darwinian terms by reference to material factors and so on.

Have you thought concretely about what international governance of this technology would look like?

What it, what is it about language and coding, which are really the two breakthrough technologies in AI of the past five years, language and coding, what i- what is it about those specific skill sets that is going to make this, these AI breakthroughs an earthquake as opposed to just a really cool innovation?

Okay, let's start with the chess thing.

Um, so as it happens, when the IBM computer beat the world chess champion Garry Kasparov in, I think 1999, um, Time Magazine asked me to write their cover story about it.

And I, I, I did, but I, I really didn't consider it interesting, what, what, what was happening then in AI.

In fact, I steered the story [laughs] away, away from AI in a sense, and just made it a story about consciousness, not really about cognitive processes.

Because there was just so much kind of brute force, uh, in, in, uh, IBM's computer.

I think it was named, uh, Deep Blue.

So what are the prospects there of a fruitful co- cooperation?

What do you think the world is not ready for?

Well, for example, the geopolitical destabilization that could ensue, some of which I described in imagining a cyber attack that was hard to attribute, but there's also a scenario that was outlined in a paper called superintelligence strategy.

They didn't really highlight it.

It wasn't the main point of the paper, but it's in there, which is If China and the U.S., two superpowers, are racing to superintelligence and both believe that superintelligence will confer completely hegemonic power and will be used to their grave disadvantage, which is, again, that is the worldview of Dario Lange.

He wants to race to superintelligence and bring China to its knees.

then it is in the rational interest, arguably, or at least there's a real risk of this happening, of the country that's behind even by a couple of months as you approach superintelligence to stage dramatic preemptive action to disrupt the progress of the other country, you know, like could be cyber attacks, which itself could lead to World War, but in this context, but it could also be kinetic attacks on data centers and stuff.

So there's that destabilization.

Could you describe the types of phenomenon that you are known for in that?

[laughs]

I'm kinda begging, begging for people to start off with a question of purpose.

So I wanna say, in terms of what I mean by that title, several things, but one of them is, you know, by some accounts, AI is gonna turn into some kind of global superintelligence, which, uh, I'm kinda sorry to say, I don't think [chuckles] is crazy.

It could, it could happen.

It's kind of a daunting prospect.

And, and some people refer to that as a god and, and, and say it could have godlike powers and even godlike, uh, control Over us and, and if it is any sense God-like, then part of the challenge is to build a good God.

So it's a, it's like a test of what kind of God we build i- if it, you know, assuming it does reach that level.

Yeah

What about malign actors? If they wanted to use AI for a bad purpose, give me an example, who might do that and what might they do? That's also a question from Claude.

Well, the standard examples is it could help somebody build a bioweapon, kind of super hacking, a self-replicating super hacker, like taking out all the satellites in the world is another one.

But one thing those two have in common with a lot of other possible negative effects of AI is that they cannot be addressed at the level of national policy alone.

You can't keep your country safe from a pandemic started with the help of an AI in some other country by controlling AI within your borders.

Same with the hacking.

And that's why it's heartening that Trump and Xi seem to recognize this and they're going to have a dialogue.

But this is one reason that in the book, among the things I'm asking for is a level of international cooperation we've never had.

evolution distorted our view?

Okay.

Again, most people did not understand that with a certain kind of machine...

I mean, it was mapped on the human brain.

The original idea was, well, let's build something like a brain, which we did, a neural network.

But the idea had also been, and this was embodied in the model I did describe in the piece, the, the neural network.

The assumption had also been, and then we'll, like, set it up the way we think a brain works and, and, uh, you know, but, but you'll have to kind of understand, like, how we, how we process meaning or something.

And, and I just want to repeat that you d- Let me, let me describe to you- I don't know, maybe this is too technical.

And how do you kind of parse those, uh, parse those things?

Okay.

It's not just that.

Again, most people did not understand that with a certain kind of machine...

I mean, it was mapped on the human brain.

The original idea was, well, let's build something like a brain, which we did, a neural network.

But the idea had also been, and this was embodied in the model I did describe in the piece of the neural network, the assumption had also been, and then we'll like set it up the way we think a brain works and, and, uh, you know, but, but you'll have to kind of understand like how we, how we process meaning or something.

And, and I just want to repeat that you d- Let me, let me describe to you...

And how do you kind of parse those, uh, parse those things?

Why do you think most people still don't grasp the magnitude of what's coming?

coming?I think, uh, a couple of reasons.

One is I think there's a misunderstanding about what's going on with these machines, and that leads to one sense in which they are a product of evolution.

Okay, so it's commonly said that they are trained, and that's a fair word.

The... and, and the training process is referred to as a learning process, and that's true.

But it's also true that the training process is a process of evolution that, in effect, reverse engineers cognitive functionality that in our species took millions of years to evolve, okay? So a good example is the language, uh, generation that they famously do, uh, sometimes called next token prediction, next word prediction.

You know, it turns out that, uh, they developed kind of on their own in a way, uh, a system of representing the meaning of words, okay? I mean, I could elaborate on that, but it would, it would get too technical.

Mm-hmm

What are the downsides? What are the downsides?

Well, I mean...

Well, one of the fears, and I would say the first model where this is starting to seem real is this model mythos that Anthropic has not totally released.

It has released a version of it with guardrails called Fable.

And you'll notice the guardrails, if you ask much about biology, if you get very close to what you'd want to know if you're going to build a bioweapon, it'll say, sorry, can't help you.

And that's a guardrail.

But guardrails are famously surmountable, okay? Every, you know, there's a guy named Pliny the Liberator on Twitter who has a lot of followers.

And, you know, will there come a time where, you know, we're not bombing Iran because we claim we want to prevent them from getting nuclear weapons? Will we be bombing countries and saying we can't risk them getting AI?

The architects, yeah.

Yeah, nobody entirely understands them.

I mean, the key...

The thing I realized when I went back – so I had an example of a neural network.

Neural networks are at the heart of his approach of the deep learning revolution.

All these huge models are neural networks, so-called neural networks.

And I was describing one that had actually been proposed by a former collaborator of Hinton's but it didn't really represent what Hinton, it turns out, had in mind.

um questions but uh the by meaning you kind of mean like context and and uh background and possible different inflections here and and all of just the built-up kind of human understanding of things it's they're not doing any of that they're doing pattern recognition is that a way of looking at

But I'm curious from like a 50,000 foot view, what do you think this means more broadly speaking than we usually talk about between us?

Oh, I think it's a big shift in, you know, of power toward Iran.

And, you know, some people are just having trouble wrapping their mind around this.

I mean, you know, people complain, oh, Trump got such a bad deal.

You're hearing this also from liberals, by the way, Democrats who who want to score points on Trump, they'd probably believe it, but it's like, what kind of deal do you expect him to get given the position he put himself in by launching this war? Because the key thing that happened, and it is related to drones, I don't, I think the drones kind of intensified the effect and maybe made the key difference, but the new technology generally, but, you know, people long said, you know, Iran could try to, you know, stop traffic in the Strait of Hormuz.

Well, they finally actually did it.

And I think that's, to some extent, it's that contingent fact, right? Like we gave them the excuse to do it.

But, uh, that, that doesn't seem to, to be happening or if it is going to happen, it's not happening on a short timeframe for sure.

But why are you so bullish on the current paradigm?

Okay.

So I gather you have two kinds of audiences, audio and video.

This won't surprise the video audience, maybe it'll surprise the, the audio audience, but I was a journalist in 1983, okay? I'm pretty old.

And I interviewed Geoffrey Hinton and for a, a, a piece I was writing on AI, and since then, I've intermittently written about technology, not exclusively, but I, I've kind of kept up with it.

And after the large language model got my attention, and, and it was the first time AI like really got my attention in a big way.

You know, I, I wrote the Time magazine cover story on, i, uh, when the IBM computer beat Garry Kasparov for the world chess champion- championship.

So what's the case for thinking this is likely to happen over the next five or 10 years rather than kind of more gradually over the next several decades?

i gotta tell you where i stepped back it was when a preacher man told me he says look he says you have to understand that ai technology we can finally hear god's voice and i went whoa hold on you're gonna have to give me time to research that i because i didn't digest that at all and and and so how do you feel about it that the religious world is going in that direction

Well, first of all, there are people who aren't religious who think that this is going to amount to a kind of superintelligence that could ultimately govern the world.

And so it's a god in this kind of metaphorical sense, whether or not they're religious.

And I regret to say that I can't totally dismiss the possibility that it could have that kind of power.

I mean, leaving aside the literally religious question for a second, I think people need to be aware.

And look, I was a skeptic of these extravagant scenarios.

10, 15 years ago.

But who is he to you and how did you grow from his his inspiration or reasons to kind of be aware?

uh, what do you, what do you conclude? Well, I've always,

I've always argued that if you step back and view, I mean, certainly since, uh, non-zero, I even, uh, hinted at it in a book in the, in the 1980s, but, uh, In non-zero, I argue that if you step back and look at the whole, you know, unfolding of biological and then in our species, what's called technically cultural evolution to mean everything from technology to politics.

And look at look at how our own social structure evolves.

continues to kind of sustain the basic direction of evolution toward higher levels of complexity and richer complexity.

You know, our society gets globalized eventually and more and more division of labor.

You know, you step back and look at the whole thing, you got to wonder, is there a purpose unfolding? Even if, as I assume, it's a straightforward Darwinian mechanistic process.

So anyway, I've...

way less so for a lot of the jobs of the 2010s, right? And, you know, the idea that we should sort of preserve this status quo ante in amber, essentially out of fear, strikes me as not passing.

Sure.

I mean, look, I think people who engage with these things closely are just more and more impressed.

If you've been monitoring the stuff, like there was a big jump from, uh, GPT 3.5 to 4 even, but this is now, that's now three years ago, and I am just quite impressed, and they're doing things in pretty arcane realms.

Like I, I don't know if you heard about this thing about a month ago, but there are these, these set of mathematical problems that were laid down by a guy named...

I don't know what's his name.

Erdos or something.

That's wrong.

Right

uh, what do you, what do you conclude? Well, I've always,

I've always argued that if you step back and view, I mean, certainly since, uh, non-zero, I even, uh, hinted at it in a book in the, in the 1980s, but, uh, In non-zero, I argue that if you step back and look at the whole, you know, unfolding of biological and then in our species, what's called technically cultural evolution to mean everything from technology to politics.

And look at look at how our own social structure evolves.

continues to kind of sustain the basic direction of evolution toward higher levels of complexity and richer complexity.

You know, our society gets globalized eventually and more and more division of labor.

You know, you step back and look at the whole thing, you got to wonder, is there a purpose unfolding? Even if, as I assume, it's a straightforward Darwinian mechanistic process.

So anyway, I've I fooled around with that.

way less so for a lot of the jobs of the 2010s, right? And, you know, the idea that we should sort of preserve this status quo ante in amber, essentially out of fear, strikes me as not passing.

What's wrong with that two-word sentence?

What Max said was any AI company that doesn't want to be regulated, you know, if you try to impose any regulation on them whatsoever that might conceivably slow down progress, they say, but China.

In other words, this would slow us down to race with China.

And that's like the magical, you know, get out of jail free card.

I mean, I think he's right, but I will say that Dario is kind of an exception.

I think there are a lot of people who use in Silicon Valley, use the China talking point cynically to fend off regulators.

Dario Amadei, I think, is ideologically motivated.

Is that fair?

Talk to us about that.

If all goes well, I'll have a piece in the Washington Post about this by the time this comes out.

But my argument is that, first of all, skepticism of the Trump decision is proper.

And overwhelmingly in the tech community, there's been condemnation of it.

And a certain amount of skepticism is proper.

It's a new exercise of executive power.

We need a clearer explanation.

Are you concerned in that domain that people will be just relying so much on these machines and that will...

And it does create the sense that there's something, that we have a collective intelligence in a way that keeps progressing.

Yeah, and it has.

And, you know, as he pointed out, biological organization has grown to higher and higher levels.

You know, you get strands of self-replicating material, you get cells, you get complex cells, you get multi-celled life, you get societies of multi-celled organisms.

And then this one species of multi-celled organism launches the second evolution that is in some ways like biological evolution.

You know, cultural evolution is a technical term to encompass everything from technological evolution to the evolution of religious and political traditions and so on.

And anyway, that evolution is seems to carry organization to a still higher level because, you know, one day you got hunter-gatherer villages, then you got chiefdoms, you got ancient states, and now, as they are noted, we are approaching the global level of organization.

And how they relate to AI.

Uh, you describe that and, and, and you have this, uh, really arresting sentence in your book that I really love, which is, and I'm paraphrasing here, um, "It's not that we created machines that can think, it's that we created machines that create machines that think." And so just unpack what you meant by that and why this emphasis on thinking of this as a kind of an evolutionary process is helpful for your account.

Yeah.

So this gets to the kind of m- narrative framing of the book.

I mean, in 1983, and yes, I am old enough for this to be true.

I mean, if there's anybody watching this, they, they, they won't doubt that, but [chuckles] who knows? People, people hearing may, may, may have not, uh, reckoned, uh, with how long I've been around.

I was writing this piece on artificial intelligence, and I was calling around and getting background.

One of the people I called was named Geoffrey Hinton, who is now the person most commonly called as the godfather of AI.

But I worry.

And I think we see it, as these models get smarter, we also see greater sophistication in the deception.

Yeah, I do think there's very good reason to believe that certain kinds of deceptive tendencies are ingrained in us by natural selection.

As you suggested, they are fine-tuned through learning, so your kids are kind of experimenting, right, with what kinds of things they can get away with.

And, you know, y- you kinda hope that as people grow up, it'll be fairly marginal.

I mean, there are a lot of subtle forms of deception, uh, that people engage in, often without quite realizing it, you know? Like, "Sure, I'd love to do that with you someday," and they believe it, but you know it's not gonna happen, [laughs] you know? And, and there's a lot of that kind of deception.

There's a term called intellodynamics that a, uh, a doomer named Liron Shapira came up with to describe the study of just generic properties of intelligence.

Now, that phenomenon was much explored by the so-called doomer in chief, Eliezer Yudkowsky, among other people, some years ago, and it was back then that they started saying, "Look, any goal-seeking intelligence system is gonna happen upon certain, like, basic tendencies or implicit understandings about how to reach goals." One is that power is often useful in reaching goals.

Yeah.

i gotta tell you where i stepped back it was when a preacher man told me he says look he says you have to understand that ai technology we can finally hear god's voice and i went whoa hold on you're gonna have to give me time to research that i because i didn't digest that at all and and and so how do you feel about it that the religious world is going in that direction

Well, first of all, there are people who aren't religious who think that this is going to amount to a kind of superintelligence that could ultimately govern the world.

And so it's a god in this kind of metaphorical sense, whether or not they're religious.

And I regret to say that I can't totally dismiss the possibility that it could have that kind of power.

I mean, leaving aside the literally religious question for a second, I think people need to be aware.

And look, I was a skeptic of these extravagant scenarios before.

10-15 years ago you I was like I did a podcast with the doomer in chief Eliezer Yudkowsky he's now world famous as the biggest doomer and I you can go you can hear me being skeptical and say I'm like wait why would these things pursue power I don't get it I've taken a closer look over the last few years and I'm not predicting some kind of global takeover but I find these scenarios harder to dismiss and in any event what I'm confident of is huge impact okay uh this is going to be disruptive across a number of fronts and so now as for people there are people who have a uh there are secular people too in silicon valley who have what is actually a religious kind of faith that this is going to work out well they talk about the singularity everything's going to be great The singularities, you know, this acceleration of change.

But who is he to you and how did you grow from his his inspiration or reasons to kind of be aware?

First formalized by Stephen Harned in 1990, it asks, how can meaningless symbols manipulated by a computer acquire real world meaning? How, Rob?

Okay, so there's kind of, in a way, two dimensions of that question if you construe it broadly.

And the first part of the question about how these words can come to have meaning is answered by large language models in the course of their training by locating the words in what you could call semantic space, where the dimensions of space, many, many dimensions, represent features of the thing.

So, for example... you know, tigers and rattlesnakes might have similar values in terms of lethality, but very dissimilar values along the dimension of speed.

And these dimensions are, they're very implicit in the vectors, the numbers that the machines come up with for the vectors.

But we now know they're kind of back there somewhere in mathematical space, in latent space.

So is that question, And the machines, again, in a certain sense, figured that out.

So let me just ask you, so what is the deal with God?

Least valuable coauthor, it should be said.

[chuckles] Well, it was a very valuable paper.

So yeah.

So I did...

I, you know, I'm basically a journalist.

I have taught at the college level, but I am largely a journalist.

I spent my life, to some extent, making technical concepts accessible to laypeople.

... with like what's your positive vision for the future? If you, you know, if you could navigate all these pitfalls, where do you think we wanna end up?

You seem to be saying that the AIs will reflect human nature because humans designed them or fed them or trained them.

Yeah, I mean, first of all, it is a documented fact now via various kinds of lab experiments with these AIs in contrived settings.

And also with a certain amount of just observation of stuff that goes wrong in the real world, it is now a documented fact that they are inclined to use deception methods.

When it's useful, they will, I mean, a particularly dramatic experiment that I recount in the book that's gotten some attention.

They set up an AI in, you know, it was an artificial office environment.

It wasn't real.

But so far as the AI knew, yeah, there are these people.

So you're going back to bubbling lava and meteor strikes, you know? Yeah.