Yeah

... unfortunately, it seems like the, like the Tescreal, uh, thesis and that the framework that, that I, uh, developed with, uh, the computer scientist Timnit Gebru is becoming more and more pertinent.

Um, you know, precisely because, uh, the sys- the AI systems that are being developed by, you know, the companies in Silicon Valley and in London, in the case of DeepMind, um, are becoming, uh, more powerful in certain ways.

And like, as you mentioned, I mean, there, there are all of these quite alarming incidents, uh, you know, O- OpenAI's models hacking into the Australian, uh, government.

You know, messing, like you said, messing with some US, uh, government websites, the hacking, uh, um, Hugging Face, uh, incident, et cetera, et cetera.

Um, and so if the, maybe the first thing to note is Tescreal, so it, it's this, like, clunky, kind of u- ugly acronym which stands for seven, uh, different ideologies.

But the easiest way to think about it is it's just a name for the dominant orthodox worldview in Silicon Valley that simultaneously highlights the relevant intellectual history which explains what that worldview is, where it came from, what shaped it, how it's evolved over time, and so on.

Yeah

Okay

OkayLike you said, emerged in the late 1980s, early 1990s.

Um, there was, you know, somewhat famously an Extropian mailing list that brought together transhumanists from all around the world.

It's where people like Eliezer Yudkowsky, one of the leading AI doomers out there, um, you know, was able to, uh, exchange ideas with people like Nick Bostrom, uh, who's another, uh, very important, uh, figure in the formation and development of the, the tech screw worldview.

Um, so you're, you're absolutely right.

And then, you know, the other ideologies, um, right after E in the acronym, like singularitarianism, this is basically just transhumanism, a libertarian transhumanism with a focus on the singularity.

And at this point, I feel like most people listening probably know what the singularity is, right? That that wasn't the case like f- five years ago.

Yeah

And what is it of, of all the things as a philosopher, of all the, the things you could have been looking into, why did you decide to dedicate your life to studying existential risk and catastrophe?

[laughs] It's a good question.

Um, I would say that probably my childhood had a lot to do with it.

So I grew up in a very religious household.

The denomination I was a part of is, is Baptist and evangelicals in the, in the US tend to embrace a particular interpretation of scripture that is called dispensationalism.

And even though many people have not heard that big polysyllabic term, just about everyone's familiar with it, right? So the idea of the Rapture comes from the dispensationalist interpretation, which was basically invented in the 1830s, right? So everybody's familiar with, with the Rapture.

So when I was growing up, there was a huge emphasis on the Rapture being imminent, on the Antichrist rising up and taking over some, you know, international or, or supranational organization like the European Union or NATO or something.

Whether it be the psychosis we explored in season one, potential economic consequences, or the apocalypse that's currently being talked about in the news daily, how do the frontier AI labs justify these risks? These are questions world leaders are asking as I record this.

And what are you saying that they do want it to happen and they're only opportunistically hand-wringing about it to get this regulatory environment that they want to happen? I mean, how should we situate all of this?

Let me just say, they sincerely believe that there's a possibility that superintelligence, ASI, could bring about our extinction, specifically final extinction.

Again, that's what they care about.

They also, I think, realize that This extinction discourse can help them further concentrate power.

It's an opportunity for regulatory capture and so on.

What they ultimately want, and the whole reason that they're racing to build superintelligence, is because they want to bring about a cosmic post-human utopia.

They all talked about this.

Do you have thoughts and feelings about how sort of establishment, more mainstream outlets like Ezra Klein and The New York Times and Positive America are weighing in on this versus these lefty folks like Lena Kahn and Alvaro Bodea, for instance?

Here's Emile Torres on pro-extinctionism.

So you might think that pro-extinctionism, the view that humanity ought to go extinct, is a fringe view.

And that is in one sense, in a very particular sense, that is true.

In another sense, it is absolutely not true.

Pro-extinctionism is massively, massively influential in the world today.

So I'll explain.

To, to understand the, the, the difference between the two types of pro-extinctionism that you find out there, you need to distinguish between two kinds of human extinction.

That's exactly what Emile is referring to here.

Like, what exactly is the extropian belief set? What is that philosophy? What does it mean?

Modern transhumanism took its initial form as extropianism.

Extropianism is the embodiment of libertarian transhumanism.

Being a version of transhumanism, extropians want to radically re-engineer humanity ultimately to create a new post-human species that they believe will be superior to us and will be able to take over the world, run politics, do science, build new technologies.

I- in other words, like, rule the world, [laughs] right? That's the goal of this, this post-human species.

And the way that, that we would do this is by using advanced technologies.

Some of them refer to a cluster.

The more Emil looked directly at the paradise on offer, the stranger it looked.

Oh my God.

So there are a bunch of people who, who like myself, who follow this pretty closely, who are just saying like, "Oh my God, is this how the world ends?" [laughs] Like, you know, Pete Hegseth, you know, this former Fox News, like, weekend show host, uh, decides to put Claude in charge of the military, and not only does Claude decide to bomb a school or right next to a school, killing, you know, more than 180 or up to 180, uh, kids, but maybe it opts to launch some nuclear weapons, you know, in response to, uh, y- you know, growing tensions between the US and Iran or the US and, and Russia.

Um, so this is all just like a really worrisome situation, and there's a, there's an alternative, uh, world, a parallel universe, a different timeline in which you can imagine AI companies making the promises that they've made and then actually behaving in ways that are consistent with that commitment to improve the world.

Which would mean they do not, uh, collaborate with the US government or the US military or the Department of War.

Um, they make sure that their chatbots are not inducing psychosis, right?

Yeah

... reduction.

Like the window for meaningful opportunity to mitigate climate change is rapidly closing, and right as, [laughs] as it's, it's about to close, AI comes along and makes things, um, you know, that much, much worse.

Did I get that right, in a, in a nutshell? Is that, is that EA?

I mean, this is really the sibling of rationalism.

So basically, rationalists are all about optimizing irrationality, EAs are all about optimizing our morality.

And j- just as a side note, I, I think this, this idea of optimization is, in ways, like, problematic and misguided.

Like, morality is something to be optimized.

I mean, basically, Yudkowsky, in that thought experiment, I mean, he was sort of applying EA principles to an ethical problem.

The pitch that they usually give is that it's about using science and reason to figure out the best ways to do the most good, and a lot of these ideologies from a distance, at first glance, look pretty good, right? Like, "Okay, yeah, I, I, sure, of course, I want to (laughs) to be the best altruist I can be in the world, to do as much good as possible." When you look at the details, that's when things start to, to fall apart.

What message of hope, hopefully, would you leave people with when thinking about these ideas and how we find our way to a better future in spite of them and the power that they have? Let's start with you.