Like what is the through line there?

So I initially, when I was in college, what I really wanted to do is I wanted to make video games, which is maybe not an uncommon thing that people want to do when they're in college.

I spent a summer interning at a video game company that I won't name.

I didn't have such a great time and decided that, you know, contrary to my very romantic notion of what making video games was like, it's actually, you know, a corporate job like any other with all sorts of corporate job problems and decided that maybe what I should do is try something more difficult.

Maybe more difficult at a technical level and maybe easier at the organizational level, which is to do research.

But since I really liked the idea of interactive virtual environments, I worked on computer graphics.

And in computer graphics, computer graphics at some level is about simulating the universe.

But you made a bunch of interesting... architectural choices in there uh for it to be the first i would say release from the lab um take us through it kind of like take us a little bit through kind of like some of the underlying almost insights and perhaps motivations for the decisions that you made there

So how do you process this?

Yeah, that's a great question.

I guess what I would say is this, that it has actually always been the case with AI that problems that seem to us to be very cognitively demanding often end up being less difficult than ones that come to us naturally.

And there's a reason for this.

So even before the current machine learning revolution, AI systems managed to, for example, defeat the world champion at chess long before they could understand even basic natural language.

The reason for this, our brains are very capable and we've gotten very used to being able to kind of adapt them to things that they were never designed for, so to speak.

And these things are hard, and they feel very hard to us.

Are you also then saying that this discovery of generalization to these kind of tasks will potentially require different type of architecture, new version of transformer or something else versus just some kind of data and training and iterations will get us there?

You could, uh, collect like physical interactive data, and I want to hear your take on what's the best data to get, what's the worst, and what are the pros and cons?

Uh, this is, by the way, like a question that is, uh, I guess quite-- Like there's a lot of discussion in the robotics community about this question and, and some people have like very opposite o-opinions on it.

My own take on this is that, um, a lot of different data sources are easier for the model to internalize if it can ground them in a thorough physical understanding of the world.

So, uh, let me try to explain what I mean with like a few examples.

Um, if you want to learn to fly an airplane, you will probably use a, a simulator, at least during part of your training.

Uh, but the simulator makes a lot of sense to you because when you start using the simulator, you have a lot of world knowledge that you can use to ground what's going on.

Like you know that when you, uh, are using the flight simulator to learn how to fly the airplane, you're not just like playing a video game.

Um, is there some shift in the industry that made that so? Or, you know, is that something you could explain?

Could you walk us through that?

At some level, doing end-to-end control for robotic systems is a very, very old idea.

The first, for example, autonomous driving systems that used end-to-end learning, they existed in the 1980s.

ALVINN was 1986 or '87, and that was a driving system that was demonstrated to drive on highways, controlled by a neural network, and then from a camera.

The neural network was tiny.

[laughs] There are some very venerable concepts, but historically, what has been really difficult in robotic learning is that you need a system that handles the application you want to address, that is cost-effective to train for that application, meaning that you don't need like a huge amount of data for every single application you want to tackle, handles long-tail scenarios with common sense, so if something weird takes place in the world, it needs to, like, have a reasonable response to it.

And then also for the thing that it's actually supposed to do, it needs to be robust, fast, and reliable.And getting all those things together is very, very hard because with machine learning, it works best when there's a lot of data.

Schedule a demo at ridgeline.ai. Can you tell us your own personal history of approaching the problem? Maybe the origin of when you first became interested and why, and then how you've decided what to spend your personal time and attention on ever since then.

What's

up with that? Yeah, yeah.

Yeah, so I have maybe two things I can say there.

I have some bad news and some good news.

So the bad news is what you're saying is really getting at the core of a long-running challenge with video and image generation models.

Yeah.

In some ways, the idea of getting intelligent systems by predicting video is even older than the idea of getting intelligent systems by predicting text.

This is a question I've asked many guests, which is that if you look at the part of the chain that you are observing, what is the reason that China just doesn't win by default, right? If they're producing all the robots and you come up with the algorithms that make those robots super valuable, why don't they just win by default?