What would that mean for the ecos- Because he's basically saying what you're saying.

I mean, it was surprising to hear it from him.

Ezra Klein interviewed him, you know, for his New York Times podcast.

And, you know, I think Jensen doesn't want any new laws, and realizes that we do have product regulatory regimes, right? Like, you know, we took the Ford Pinto off the market because its gas tank exploded.

This is sort of like that.

Like, there could be real harm coming from this.

So far, I don't think there's, uh, been any severe harm, but we see that these systems can, you know, hack infrastructure, which could lead to, for example, a power grid going down that could take down a hospital and so forth.

About 17 years ago, too big to fail made its way into our lexicon, right? Uh, to Dan's question, are these companies now exactly that given how important they are to the entire ecosystem of the AI trade?

And what would you have to see to change your mind on that?

Um, I would want to see, to change my mind on that, some real scientific discovery that is beyond just kind of brute-forcing like we're seeing in math.

And I would want to see it in open-ended domains, which is really what science is and AI- AI is.

So the one place where we're seeing something like that is with math.

But with math, you can use these theorem provers and things like that that are not actually part of the neural network, although that's a separate conversation, um, and they can formally verify certain things.

And the problem with most science is it isn't like that.

Most of the interesting things in science, you can't just stick it in an equation and, and kind of check your, your formal proof.

OpenAI says they are, the way they describe it is, responding accordingly.

So Gary, um, what is your level of concern today?

Uh, well, I guess there are many different things to be concerned about, so it's, it's hard to, um, answer that.

Uh, part o- One set of concerns is about the AI itself, and the other is about politicians not understanding the problems.

So I am not at all concerned about extinction.

I like that you called it fringe.

It's a very popular fringe movement right now.

I think it distracts from the real concerns that we should actually have.

I- if AI, um, develops into, you know, a, a, a super system that can put everybody at risk, either with grids or with financial stuff or with drone attacks, doesn't that, um, affect everybody equally? I mean, if, if it's a r- if we're at risk here in the US, then China is at risk as well, and because we are all at risk in the same way that we were all at risk, um, during the nuclear age, still are, but really at risk in the moment, does that force these, um, hostile nations, China, the US, others, to come together to form some consensus because it is ultimately in everyone's best interest, including the ruling parties?

So Gary, um, w- what is your level of concern today?

Uh, well, I guess there are many different things to be concerned about, so it's, it's hard to, um, answer that.

Uh, part of...

One set of concerns is about the AI itself, and the other is about politicians not understanding the problems.

So I am not at all concerned about extinction.

I like that you called it fringe.

It's a very popular fringe movement right now.

And because we are all at risk in the same way that we were all at risk, um, during the nuclear age, still are, but really at risk in the moment, does that force these, um, hostile nations, China, the US, others, to come together to form some consensus because it is ultimately in everyone's best interest, including the ruling parties?

Good.

Um, it makes for good science fiction.

I don't think it's plausible.

I know there was a guest last night on a different CNN show, um, who was, you know, saying all humans might, you know, be killed in the next few years.

That's not gonna happen.

We are very resourceful.

The people who tell these ghost stories don't really recognize how resourceful humans are.

Mm-hmm.

What is the thing that concerns you the most about the fact of the hugging face attack?

Well, first of all, I'm very concerned about all the anthropomorphic language I just heard, and I think we might want to talk about that.

The biggest problem, I think, is that OpenAI simply didn't follow appropriate security procedures, and it got blown into something different.

The most important thing that they should have done is to monitor what's going on, and Rocket mentioned that.

Had they done proper monitoring, the whole thing could have been avoided.

The systems certainly produced kinds of talk that Rocket talked about.

And you could have looked at that and decided this was a bad idea.

But you don't share that kind of dire of view, do you?

What do you make of it? Can you describe some of its capabilities? And then we will get into whether or not it is artificial general intelligence, AGI.

I mean, in terms of the model itself, it's a bit better than the models before.

What we have now is an era where p- some people call it benchmark maxing or benchmaxing, where these systems are often, it would appear, trained on lots of benchmarks.

They do really well on the benchmarks, and then they don't typically do as well in the real world.

So every time we see the phenomena, the same phenomena, model comes out, everybody's super excited about it, and then after a few days, people are like, "Yeah, but I tried it and it doesn't really work on this, it doesn't really work on that.

It deleted my files.

The code it writes is for shit." Um, the code it writes is for shit is something I saw about Astra yesterday on Twitter.

Um, I think there are people who would say, "You know, here's something I can point to that it doesn't do or it's really bad at," and it's not.

What do you make of this whole event?

Well, I think there's two things.

One is it's clear that these systems are getting stronger and more powerful.

Um, people have added loops to make them do things over and over again till they get it right.

And the other is that OpenAI really screwed up here.

They didn't do basic things we call sandboxing.

They didn't do monitoring.

I- is it alarming to someone like you what these agents could do?

What do you make of this whole event?

Well, I think there's two things.

One is it's clear that these systems are getting stronger and more powerful.

People have added loops to make them do things over and over again until they get it right.

And the other is that OpenAI really screwed up here.

They didn't do basic things we call sandboxing.

They didn't do monitoring.

Is it alarming to someone like you what these agents could do?

What do you make of this whole event?

Well, I think there's two things.

One is it's clear that these systems are getting stronger and more powerful.

People have added loops to make them do things over and over again until they get it right.

And the other is that OpenAI really screwed up here.

They didn't do basic things we call sandboxing.

They didn't do monitoring.

Is it alarming to someone like you what these agents could do?

What do you make of this whole event?

Well, I think there's two things.

One is it's clear that these systems are getting stronger and more powerful.

People have added loops to make them do things over and over again until they get it right.

And the other is that OpenAI really screwed up here.

They didn't do basic things we call sandboxing.

They didn't do monitoring.

Is it alarming to someone like you what these agents could do?

Sure.

But for now you have to remember that most jobs include many different tasks, and often AI can do some of them really well, some of them not so well, and some of them it makes stupid mistakes.

And so what most companies are finding is that they're not actually getting that much return on investment because they have to kind of hold their hand, so to speak metaphorically, of these systems every step of the way.

And so, you know, we've heard for years like, "Oh, we're gonna replace all the radiologists," and it hasn't really happened.

So someday some of that will happen.

But often humans are able to deal with novelty and the unexpected in a way that these kinds of systems aren't.

And so I think, at least for a while, we're not gonna see the sort of job apocalypse that people have described.

How do you interpret that? And, and when he says they're putting safety first, can you take him at his word?

So what is artificial intelligence as you define it?

The problems in this field start with the fact that artificial intelligence is a marketing term.

It's not clearly defined.

It was actually invented in the 50s by John McCarthy, who incidentally taught my father Fortran and then left in the middle of the semester.

But that's another story for another day.

So, you know, he made this term, but it wasn't clear exactly what it referred to, but it was a general notion, right? And the general notion was, how do you get machines to do the kinds of things that people do? And McCarthy, who was one of the founders, the real godfathers of AI, unlike people this term is applied to lately, raised, for example, the common sense reasoning problem.

McCarthy was into logic and classical AI, symbolic AI.

I'm not sure I agree with it, but that's a good one.

Users were saying the new model felt dumber, more robotic, and straight up less helpful than before.

I kept telling people for years, "GPT-5 is not gonna be magic," and the whole industry, like, dumped on me.

They all went after me for even daring to raise the question of, like, was this idea of scaling, which also is partly behind the driverless cars, w-was it gonna kind of produce these magical results? And they hated me for it.

And, you know, now GPT-5 came out, and people are starting to realize that I had a point.

But it, it took, you know, three years of just, like, constant hype and, and constant not quite meeting expectations for, for people to realize that maybe this stuff is oversold.

And every incentive is to overhype these things.

So for example, if you're a journalist and you write a story about, "Yeah, it's sort of going slowly, but it's not, you know, it's not really there yet," nobody wants to read that.

Eventually, he says, something happens that forces people to pay attention.

Am I wrong to say that if left to their own devices, Both Anthropic, OpenAI, maybe others, will create computer programs, models, whatever you want to call them, that could create a 9-11-like scenario on their own.

I think that's entirely possible.

You know, I think my politics and yours are supposed to be different, but I don't disagree with much of what you just said.

The reality is that we have these systems.

They're not very well controlled.

Some of that's for technical reasons.

Some of that has to do with how the companies themselves are formed and how they've been built.

I think they

Yeah

...

but they were talking about that a few months ago.

And so if we slow down, that does present a problem.

Now, the other thing that's happening is people are spending massive amounts of infrastructure.

They're building data centers.

They've made commitments.

[chuckles]

Are, are we, are we watching this whole thing unravel in front of us at this point? What's happening i- in your view?

Well, I always think of the Dutch tulip craze, and you could be in it and you don't know exactly when it's gonna end.

You can see that all these things are ridiculously overpriced, but you don't really know when people are gonna stop.

And, and the other metaphor I o- often use is Wile E. Coyote on the edge of a cliff, and the question is when is he gonna look down? That's when he's gonna fall.

Um, I think we're starting to see signs that we are falling off the cliff, and people do care about circular financing now.

You know, when there was a circular financing deal in September when, um, Oracle and A- OpenAI made that deal, Oracle leapt up.

It, it went up 40% in one day.

Um, and you know, maybe it's proving out right now, and we just talked about in the private markets it's gonna be hard to kinda see how that shakes out in the near term, but what could go right? Where could this kind of argument be wrong for all intents and purposes?

[laughs]

Uh, Reid Hoffman said he would bet any amount of money that hallucinations would go away in a few months.

This was tw- two thousand twenty-three.

I said, "I'm over here.

How about a hundred thousand dollars?" He never got back to me.

Um, but here we are in two thousand twenty-six, and hallucinations have not gone away.

And it's because the core of next token prediction does not allow you to address that problem, so you have to add something else, and the something else, you know, rarely works all that well.

So I guess my question to you is where do you stand on Mythos? Because we, we know that your views on generative AI and their, th- their limits, but how do we put that next to the fact that people are very scared about this thing that is supposedly extremely powerful?

Professor Gary Marcus, one of the world's most prominent critics of current language models, thinks AGI might still be some way away, but he's still very worried.

You don't need AGI to cause absolute pandemonium, right? We may see absolute pandemonium this year.

It doesn't have to be AGI.

It can just be AI that is special purpose around bioweapons, or it might even be just a general purpose chatbot that can hold the hands of the bad actors and get them to do stuff they wouldn't ordinarily be able to do.

I think everybody in the field now sees that cybersecurity's about to become a huge nightmare way beyond anything we saw before.

If that's true, then think what bad actors can do now, right? So, you know, they may be able to shut down power grids relatively soon.

All of that is an accident waiting to happen.

What are the chances you see of something like that?

He told me...

I still don't think we're that close.

I think there's a lot of work left to be done.

Now everybody says two or three years.

But they keep saying two or three years, and they just move that back every year.

So I, I don't think we have two or three years.

I think we have longer than that.

It will require some kind of mid-level catastrophic event to wake everybody up.

Is AI a bubble? I guess that's the easy question, and the hard question, if it's a bubble, when's it gonna burst?

I often think of Wile E. Coyote in the, the Bugs Bunny cartoons where he goes over the side of a cliff, but he doesn't actually fall until he looks down.

And so some of this is about psychology.

When will people realize that maybe the economics here don't make sense? But they are actually starting to.

You know, the fact that more people are talking about it is part of a shift in the psychology.

The fact that there's a backlash is a shift in the psychology.

It also means, like, younger people may not be the customers that they were expecting and so forth.

Where are we gonna end up? What's your bet?

So let's just review what we talked about the f- the first time, which is j- why, why don't you just r- remind our viewers about your thesis about what I would call the limitations of LLMs?

LLMs?So that, that actually goes back to my graduate work before LLMs existed.

I was looking at how children learn language and looking at the neural networks that were popular at the time.

They were ancestors to today's systems.

But that got me interested in, like, how do these systems work? What are their strengths? What are their limits? And I found that the ancestors, and we can talk about the current systems, were good at memorizing things, good at kind of pattern matching, and not very good at generalizing more broadly to things they hadn't seen before.

Later in 1998, I did some, I think, foundational work describing what we now call the problem of distribution shift, which is to say you train a neural network on some data, and if the world remains that it's, uh, immersed and remains similar, it does fine, but if you ask it something that's really different from what's there, you find out that its comprehension is limited, it has trouble generalizing.

And I've been thinking about that kind of stuff ever since.

I can't, I, I can't imagine positively.

Are you backing off at all?

No, not really.

Um, Anthropic I think looks a little more stable to me than OpenAI.

OpenAI is still burning cash at a huge rate.

Um, the thing you have to remember is it is intrinsic to how this particular approach to AI works that it requires a massive amount of data, a massive amount of compute, and is expensive to operate.

That's not true of all software.

Like, you can run your spreadsheet on your own local machine.

... .5 trillion for SpaceX.

Where, where's the lawsuit in all this?

It's, it's not about control.

It, it, it's about whether they're following the nonprofit mission, right? I mean, they got tax benefits out of that.

They...

You know, I think Elon would argue that they committed fraud on him.

You know, he put up a lot of the money.

They, they relied on his, uh, you know-Name for recruiting.

If he gets anything like that, right, wouldn't that be lights out for this company that's burning through all this cash?

Uh, as someone who's delved into it really, really deeply in the last few years, written about it extensively, what are the three biggest challenges you see with, let's call it, the unbridled evolution of AI systems today?

I think the number one problem right now is that we're using unreliable systems.

The core technology that most people are excited about is the large language model, and really what it does is it mimics distributions of data, and that's not the same thing as reasoning about the world.

And so what you get is plausibility rather than truth.

Um, the-- and so you see, for example, these hallucinations where you, you ask it for a biography of someone and it fabricates where they're born and things like that.

And these can be subtle and, you know, you can see it in a domain where you know it well, but not in a domain you don't know it well or not if you're paying attention, um, if you're not paying too much attention.

And so it's kind of a persistent problem throughout all of the use cases, and this is also why they're struggling to make money.

since my episode is about Mythos, where does it fit?