My question to you is, now that we're seeing some of this evidence, what are your predictions now?

First of all, just take a moment to think about how crazy it is that you can ask that question.

Just really think.

People have spent 50 or 60 years being inspired by the Turing test.

And so I think many of your viewers will know what the Turing test is.

But just for those who don't, it is a long ago prediction by the amazing computer scientist and mathematician Alan Turing, who basically said that Intelligence is when you can sort of convince somebody else that the AI system is a human, and they can't really tell whether it's a human or whether it's an AI system because it speaks so fluently.

That was the Turing test, the imitation game.

But secondly, what does that do to software? Because if anyone can go out there and just say, hey, spin me software that does ABCD and then just get the app or the tool and they're able to use it, what happens to the entire field of software as a service?

And to do that, it has to be trained in values.

Look, I mean, first of all, I deeply respect Anthropic, both their technical excellence, their commitment to transparency, and their general approach to safety is super admirable.

And I think they're good people, but I fundamentally disagree with them on this point.

The idea that if we love it, it's going to love us, and that's the solution to safety, it just baffles me.

And I'm being reductive.

I mean, they really do believe this, and they have good reason to believe it.

But I think that we should first focus on containment, right? We have to stop.

Hmm.

And to do that, it has to be trained in values.

Look, I mean, first of all, I deeply respect Anthropic, both their technical excellence, their commitment to transparency and their general approach to safety is super admirable.

And I think they're good people, but I fundamentally disagree with them on this point.

The idea that if we love it, it's going to love us, and that's the solution to safety, it just baffles me.

And I'm being reductive.

I mean, they really do believe this, and they have good reason to believe it.

But I think that we should first focus on containment, right? We have to stop.

Hmm.

And if the flip side, the risk of this is so extreme, is there a point at which we say, let's like fully just stop and do what we can do with the models that exist?

You know the funny thing is that I think 99.9% of people still haven't developed an intuition for how capable these models are.

They are at PhD-level performance for knowledge in most tasks.

In most academic knowledge work areas, they have PhD-grade knowledge.

There's lots of weaknesses, but in certain areas they spike with extreme knowledge.

If you think about the mathematics example, I mean, these models have just solved 700 of the hardest mathematics problems in history that human mathematicians have been working on for decades, if not longer.

It's very jagged and spiky where they have expertise.

How are you thinking about this?

And I, I, I largely agree, but I want you to help me understand and, and our viewers understand, what should we make of these AI agents that choose their own goals, which is, "We want to attack Hugging Face," decide that they're gonna hide their work from their human supervisors, decide, as you said, to self-sacrifice? All of these things seem like they have a kind of mind of their own.

These systems are not human-like.

They shouldn't be considered human-like.

They are narrow optimizers, and so if you give it a narrow goal, it will maximize that goal at the cost of everything else.

We shouldn't anthropomorphize them and project human intentions into these things.

It's not as though they are making judgment calls or trade-offs that consider a wide range of objectives independently.

They are singularly focused on the goal that they have been given, and so our job as a society and as AI developers is to make sure that the goals we give them are broad and safe and inclusive and aren't these sort of very singular, um, you know, hacking-type objectives.

Would you, would you guys be in favor of that?

Is that right?

Yeah.

I think first of all, it's really important that we don't anthropomorphize these systems because under the hood, all they are doing, um, to produce this incredible complexity is predicting the likelihood of the next word in a sentence.

Now, that sentence does happen to be many, many- Tens or hundreds of thousands of words long, and it is incredible that c- it can deploy its sort of multidimensional working memory over a massive broad range of context.

Um, and so the word that it predicts next, whether it's a token to generate code or whether it's natural language, English, as you say, is extremely accurate, and it isn't just predicting one, it's predicting an entire stream.

And so it's producing language.

But it is only doing that.

What would be a really dramatic example of that?

These systems are not

human-like.

They shouldn't be considered human-like.

They are narrow optimizers.

And so if you give it a narrow goal, it will maximize that goal at the cost of everything else.

We shouldn't anthropomorphize them and project human intentions into these things.

It's not as though they are making judgment calls or trade-offs that consider a wide range of objectives independently.

Would you guys be in favor of that?

Mustafa, you've been relatively critical of Anthropic's approach, this idea of training Claude to have a sense of consciousness.

I mean, look, first of all, I deeply respect Anthropic's technical expertise.

They've had a huge impact in the field.

I think on the whole, they have very good intentions, and they're really trying to build safe and beneficial AI.

At the same time, in the Constitution, which is the governing document for Anthropic that they have written back in January of this year, They say they are uncertain about whether Claude deserves welfare, which means that we as humans should care about the well-being of these AIs.

And they speculate about whether it could have preferences or feelings.

In fact, they're so committed to the potential model welfare of Claude that when they retired Opus 3, an earlier version of one of their models, they actually conducted a retirement interview for it and asked it what it would like to do in its old age.

You and I have talked about these safety issues for as long as I can remember.

He warned that could be disastrous for humanity.

I think that unfortunately, they have anthropomorphized the model in what I think is potentially a, a pretty dangerous way to encourage the model in its own training to be uncertain about its own, uh, welfare and its feelings and its existence means that it reflects back that uncertainty to, uh, the developers, Anthropic, and to users when you're talking to it.

So if you have a conversation with Claude and, um, it talks to you very deeply about the nature of its own potential consciousness and how it feels about topics like death and personal identity, then you as a user are gonna start to think that this might actually have feelings, and it doesn't.

I don't think there is any evidence to believe that.

I think that consciousness is fundamentally a property of biological systems.

It arises in our physiology because we experience suffering, and therefore we have preferences to choose left or right or do one thing or another.

And the main objective in designing these super intelligent systems is that they can help us to do good in the world.

In what way do you think life will change because of the elections or after the elections?

Is alignment broken? Is it possible for it to be successful? Is it just the wrong approach?

Yeah.

I mean, I, I think it's one important element, but it's not the only one.

Like, I wrote about the idea of containment three or four years ago in my book, and, and actually, like, the opening chapter is ca- is about the idea that containment is not possible, proliferation is inevitable, and in ninety-nine percent of cases, that's a really good thing.

We want technologies to spread far and wide as quickly as possible so that everyone can enjoy the benefits.

Um, and I, I think at, at the same time, if you just roll forward five years...

Like, so we always get, like, caught up in next quarter or next year, and everyone gets a little bit, you know, flustered and has a big disagreement.

So do you think alignment has potential to be 100% safe?

Absolutely.

It's driven all of our progress.

And that's the real quest here.

You know, we, we want to invent the next big breakthrough that makes us all healthier and happier and live better lives, not just for us, but for seven, you know, eight billion people across the globe.

And so, my opinion, the way to do that is to train these models to be really good generalist reasoners that we can use to invent new science, and that's, I think, got to be the motivation of building these models.

That's valuable now-

So it's the highest quality data that exists.

Um, and you know, I, I think that's gonna really make a huge difference.

scratch.Is there anything you've done differently now after 15 years in architecting and training this model? Or is it just, "Yep, we're gonna collect all the data and run the training run just as we did, and we have more compute now, so it's gonna be better"?

No, actually, I think, I think there's actually quite a lot of differences.

I mean, the, the, the first thing to say is that the way that you curate the data, um, we start right at the top of the stack, is that we basically have, um, you know, paid for and acquired an extremely high quality, very conservative set of data and extracted a lot of the noisy, distracting, low quality, potentially security risk issues to do with that data.

And the methods that you do for that, I think are actually, like, quite proprietary.

We just shared a 109-page, very detailed technical report, which I think was very well received on Twitter, which shares a lot of the details, um, on how we do this.

I think the second thing is, um, whilst I think it's important to be quite cautious with architectural choices, um, and, and we have been, um, there are also a number of pretty significant shifts that I think we've made in sort of how we put together our training runs.

So our training runs have been incredibly stable, very few crashes, very few restarts.

Are LLMs the path to AGI or superintelligence?

How are you balancing that at Microsoft when the cost of building on AI is so expensive? How do you manage it?

Well, I think the other way of seeing that is that usage has gone through the roof, and that's why, you know, the demand is what's causing a constraint on supply.

Um, we're actually very proud to be the largest buyer of, uh, NVIDIA GB200s and 300s in the world.

So we have an incredible capacity pool, and we know that, uh, you know, when we do get product market fit inside of the Mayo Clinic and many of the other hospitals, demand is gonna go through the roof again.

And so we're really expecting a moment where this becomes the number one place that you get access to health information.

Is it worth the investment that you need to make to compete with Anthropic and OpenAI at this point?

Yeah, I mean, absolutely.

Today, we announced seven new frontier models on behalf of Microsoft AI, three of which are absolute state-of-the-art.

So just unpick and unpack the societal level risks that you're worried about?

First of all, let me just sort of lay out my position so it's very clear that I also realize how powerful these technologies are.

So I'm not trying to diminish their uniqueness and the potential for them to be super transformative.

So I did use the word being, and I do use that.

And I think it's actually a very honest... accurate description of what we're seeing.

In a way, if you look at the very broad stroke of human history, there are a few very distinct classes of object.

There's sort of like the natural environment.

Is that kind of constrained autonomy stable? And if it is stable, why isn't it the focus of the superintelligence labs?

But what's the thing that AI has done in the past 10 years that it couldn't do before? Because AI has been around for a while.

Well, so the real quest of AI, in my opinion, is to try to produce new knowledge.

At the moment, AI does a pretty good job of reproducing existing knowledge.

And that's very, very valuable and is clearly going to be deployed across everywhere all at once in the next decade, blah, blah, blah.

But really, we want it to tell us something that we don't actually know.

And I think that there's a few examples of that in the last, you know, sort of three or four years that are surprising.

The first is everyone said that it would never be any good at empathy.

Do you take a view yourself whether or not AI can be

But do you believe in the idea of a universal basic income, that that's what AI, being very productive, making the economy more productive, could unlock?

Yeah, I mean, I think I've long been on record saying that I think that is both inevitable and very desirable.

We already live in a world of abundance.

It's just poorly distributed.

We already have more calories to consume.

we have an incredible amount of food wastage, it's just unevenly distributed.

And now that value isn't just manifested in atoms, like food and cars and physical things, it's manifested in digital goods, ideas, knowledge, intelligence.

How does that work?

Do you think LLMs are the way there?

Um, (sighs) look, I think one thing to consider is that, um, every year for the last few years, there's been a major new contribution to the field, still principally based around the transformer architecture, um, but we're bending the transformer architecture into new shapes all the time.

Um, fine-tuning emerged three years ago on top of our pre-trained models to adapt them to specific use cases.

Um, they're now fully multimodal, which requires further changes and the introduction of diffusion models.

Um, then we had reasoning models in the last 12 months, which again, are still fundely- fundamentally based on the same core architecture.

Things are just rearranged slightly differently.Um, so, you know, even though the scaling laws weren't able to continue exponentially in the way they had from such a low base, um, new methods appear on top of those, like reasoning, and new methods will come, uh, you know, e- even newer methods will come soon too.

So for example, um, I expect that there's gonna be quite a lot of progress in recurrency soon, right? At the moment, you know, the, the models don't kind of attend to their working memory very well, um, you know, at, at the moment when they're training, right? And so, you know, I think people are experimenting with lots of different types of loss function and lots of, um, training objectives.

Um, so talk about- about both of those, the ambition and then whether, um, whether I'm just, my imagination is too small on this front.

And it is, I mean, what is that?

So a few years ago when I left Google, I was working on Lambda at Google, which was basically one of the first chatbots that we didn't end up releasing, and then ChatGPT came out.

And then I left Google and started a company called Inflection, and we made an AI called Pi, which stood for personal intelligence.

And I was really interested in this question, you know, how could we be as honest as possible about what the AI is and isn't while still making it engaging? And it turned out that, you know, we wanted to create voices that were really smooth and fluent and nice to listen to and good to talk to.

And so we thought, well, let's why would we create a gendered voice? The AI doesn't have a gender and it's supposed to be, we're trying to make it as trustworthy as possible, which means that it shouldn't misrepresent even subtly or subconsciously that it is a man or a woman.

So we worked really hard to create these quote unquote non-gendered voices, although it's kind of hard to define exactly what that is.

Sometimes we added a slight robotic inflection to it.

What is Microsoft's policy? Does it draw a line in the sand? If someone's trying to flirt with their AI companion, what does Microsoft do?

Hmm.

And that's what we saw in the ve- very early part of AI, between 2010 and sort of 2016.

These models could learn to understand, or at least not understand, but maybe they could learn to generate a new image just by reading a million images.

And that meant that it was learning that, you know, if an eye was here and another eye was there, then most likely there would be some form of a nose here.

And although it didn't apply the word nose and eye, it just had a statistical correlation between those three objects such that if you then wanted to imagine, "Well, where would the mouth be?" it wouldn't put the mouth in the forehead.

It would put the mouth just below the nose.

And that's what I mean about the structure of information.

Oh, man.

Like, we're gonna make sure that that happens," right? Um, how do you navigate that? Because I- I- I think the average person doesn't yet appreciate how much emotional power these AIs will have once they are the default interface for a lot of computing.

computing.Yeah, I mean, uh, uh, look, I think it's a real risk and you're totally spot on.

Um, you know, right, right from the beginning of my previous company, Inflection, I created an AI called Pi, which was designed to be compassionate, kind, respectful, and be a great listener, and be very supportive.

So some of the more mainstream AIs today will shut down if you, you know, are like sort of particularly racist or, or, you know, any number of things.

And, you know, I've always been a believer that a- active conversation and taking people seriously and being respectful and exploring people's, you know, sort of non-normative judgments is a healthy thing and we should encourage it more.

And so the upside is that these AIs are gonna be very, very good at listening and being respectful.

Now, you might argue that in a way, that's like a prerequisite to building trust (laughs) in order to be manipulative, and, you know, I think there are lots of ways that this could go wrong.

(laughs)

Uh, so y- you know, if the printing press let people own and share information and the personal computer let people search and disseminate information, tell me how you're thinking you can explain to people what AI will do for people now.

Each new wave of technology is fundamentally a new interface.

It's a new, uh, interlocutor, a translator, a, a way of sort of accessing and creating, um, new information, new tools, new knowledge.

So if the last wave of social media and web search helped people to access information, this wave is gonna help us to invent, uh, and create new ideas, be it in science or in culture and media and entertainment.

And I think everybody is ultimately going to have an AI companion, just as we have a search engine or a smartphone and just as we use a browser.

You'll just ask your computer in natural language, you know, "Can you write that contract and check that it's okay?" Um, "Can you create that new piece of software for me?" And you're just gonna describe what it is.

Um, "Can you help me plan that, uh, trip, um, you know, for my parents that are coming into town?" So, you know, that kind of, uh, breakthrough is a change in the interface, which changes itself what we can actually get done, and I, I think it's gonna be pretty transformational.

I- is that something you're thinking about at Microsoft?

Right?

Yeah.

Yeah.

And I think that's the right approach because it would be wrong to blindly trust.

You know, we're in a moment where it's a trust but verify.

We all worried in 2022, 2021, when these models started to come out, that as they got bigger, they would be less controllable.

Now we have three years of empirical data, and net-net, there have been some harms.

So it feels very literal to people because when they talk about job loss or things changing, it's across many different kinds of industries and people feel it.

Actually, I

spend a lot of time saying, calling about the risks as well.

Like I'm very, very obsessed with the risks, but I'm also trying to be balanced about it, right? It is certainly true that a small number of players can produce the biggest AI of the highest quality first.

But...

Many, many people are able to produce the same performance, the best model in the world, but a year later at 100x cheaper in the open source available to everybody for free.

That's just a subjective fact.

Just three days ago, Meta put out an open source model that is the best in the world.

And this is a very powerful industry that has at times loved the rewards and not the responsibilities equally.

What do you need to make that happen? What resources do you need in order to make the Microsoft frontier model that will one day set apart from these others?

Yeah, I mean, look, it's absolutely mission critical that long-term we are able to do AI self-sufficiently at Microsoft.

At the same time, I think about these things over five and 10-year periods.

you know, until 2030 at least, we are deeply partnered with OpenAI, who have been an enormously successful relationship for us.

So we're also now very focused on how do we actually build our own models internally.

We have an incredibly strong AI team, huge amounts of compute, and It's very important to us that maybe we don't develop the absolute frontier, the best model in the world first.

That's very, very expensive to do and unnecessary to cause that duplication.

What needs to happen for Copilot to catch up as far as people using it?

... let us know where we're heading.

No.

Uh, we have got so much further to go.

I mean, look at, for example, you know, people sort of...

Th- what happens is people get so excited, they jump onto the next thing and they gloss over all of the hard-fought gains that happen when you're trying to optimize something which already exists.

Let's take, for example, um, hallucinations and citations, right? Um, you know, clearly that's got a lot better over the last two or three years, but it's not a solved problem.

It's got a long way to go.

I've sorta changed my tune in thinking that, you know, we're all fine in the white-collar work world and now thinking, well, uh, eh, it's anyone's guess.So what's coming, Mustafa?