intelligence explosion

And I thought books are more effective for this subject.

And then, as you alluded to, my perspective didn't really change before my next book, The Intelligence Explosion, but the technology changed.

And I tried to keep up with it in that book, bring it up to date.

Thank you.

That puts a certain stake in the ground.

How tied to that stake do you feel?

That was the intelligence explosion.

I think we've created a perfect cauldron.

He said we are, in his words, pretty far from the end state of that scenario, while confirming that AI is already working alongside human researchers on model development in iterative loops.

The CEO with the deepest visibility into DeepMind is telling us, don't price in an intelligence explosion, price in a long infrastructure build.

Edo, while that conversation was happening in Silicon Valley, a congressional watchdog was releasing findings that should anchor every conversation about AI and power.

The Government Accountability Office published a report in October finding that the Pentagon is building AI-powered weapons and tools that do not consistently follow the agency's own guidelines for responsible AI.

That, I can tell you, is not a trend that is expected by the industry or wanted by the industry.

If you want to know more about what might lie in wait, I did this video on my Patreon about the intelligence explosion post recently put out by Anthropic.

What does Anthropic think will happen when AI is increasingly unleashed on the improvement of AI? I know I said things so far look predictable, but inventor of the transformer retweeted this OpenAI researcher's post which said that the rate and cost of progress today doesn't necessarily predict the speed of the recursive self-improvement loop.

This video is about what we can use today, though, so I'm gonna go back to the Fable-5 benchmarks.

But you sort of tap out those resources, uh, as it, AI becomes a larger proportion of the research field and, like, okay, you've, you've put in all of these inputs, but they just haven't yielded AGI yet.

I think that set of inputs probably would yield the kind of AI capabilities needed for intelligence explosion, but if it doesn't, after we've exhausted this current scale-up of, like, increasing-

Yeah

...

Doomsday preppers with billions of dollars in crypto wealth, remote island bunkers in the Pacific, bizarre corporate cultures and a terrifyingly real technological threshold that some of the smartest people alive say we are about to cross.

Yeah, the intelligence explosion.

Right.

But before we get into the madness, I do need to read a quick but very necessary disclaimer.

We are going to look at their private doomsday Slack channels.

Yeah, the Slack channels are fascinating.

A philosophy that literally calculates the mathematical probability of human extinction, wild billionaire bunker plots in the Pacific, and most importantly, the actual deeply technical reasons why the so-called godfathers of AI are right now warning world governments to prepare for this intelligence explosion.

So to really grasp the gravity of this, we need to start by understanding the psychological atmosphere inside these labs.

We're back with Ramez Naam, who is the founder and managing partner of Planetary VC, a computer scientist, investor, author who's written extensively about energy and technological progress.

He recently wrote a guest post on the No Opinion Substack called Where Is the Intelligence Explosion?

Mm-hmm.

Um, you know, if you've been in AI world for the last month or two, the vibes have shifted pretty dramatically, and it seems like the emergent consensus is, like, the labs are rushing towards, quote, "RSI", whatever that means, which will lead to very, very rapid runaway capabilities growth, like, on the timescale of the next year, kind of like what AI 2027 predicted, and we need a pretty major political intervention pacing the frontier to ensure that this doesn't happen.

[laughs]

So-

So where is the intelligence explosion? [laughs]

Yeah, where is the intelligence explosion?

What that means is that AI or artificial intelligence will not just be clever, but that it will get cleverer by itself, faster than humans can watch it, check it, or switch it off.

Experts call this an intelligence explosion, and a recent research paper co-authored by Nobel laureate Geoffrey Hinton, and he warns that it could threaten human control over AI and, worse, erode the checks on power within and between nations.

Those words are very carefully chosen.

It will erode the checks on power within and between nations.

who is an AI researcher and the creator of the Arc AGI benchmark.

His essay is called The Improbability or Implausibility of the Intelligence Explosion.

And it's basically the idea that intelligence is not some principle of the universe like energy.

It's an adaptation to a specific environment.

Um, but it is features, uh, Demis, Demis Hassabis, who's from Google and deep mind, uh, Dilip George, Tom Mitchell, Josha Bach, who I'm just going to put a pin in him for a second, Francesco Rossi, Richard Mala, Richard Sutton.

There's another big panel, which is called Intelligence Explosion, Science or Fiction? If an intelligence explosion happens, then what are the likely outcomes? What can we do now to maximize the probability of a positive outcome? Containment problem? Is, quote, friendly AI possible, feasible, likely to happen? These are great questions.

But this includes Nick Bostrom, who we're talking about, from Oxford, Bart Selman from Cornell, Jan Talen, who is the, I don't know if I'm saying that right, but he founded Skype.

Elon, of course, and then Shane Legg, also from DeepMind, Murray Shanahan, Werner Vinge, and Eliezer Yudkowsky.

What if automating AI R&D...

Triggers and Intelligence Explosion.

Let me share this.

So this is a paper from Jakub Pacočki, Jeffrey Hinton, Joshua Bengio, Jack Clark.

So Sam Hammond, who's been on the show.

And Tom Davidson, so a mix of academics, people at the labs themselves, lab founders, several Turing Prize winners, one Nobel Prize winner, and talking about intelligence explosion, explaining exactly what an intelligence explosion is, explaining how they would expect it to happen, explaining recursive self-improvement.

um in in detail so i think this is this is a this is the beginning of the i think the political discussion um and and academics reaching out to the political counterparties on uh this is what we are afraid of that may happen and describing the steps by which it may happen um and let's see let's see how much let's see if there are um There's sufficient alarm, I think, among academia, among politicians.

I will note one thing that Europe, for example, doesn't seem to be doing much about AI.