Ep 87: Gemini Co-Lead on World Models, RL's Next Domains & Continual Learning

I figured where I'd start was with world models because I think that was-

Mm

... just a really impressive part of, you know, uh, of yesterday, and also I think a, a pretty, you know, where Google's pretty distinct from a lot of the rest of the field.

So you obviously shipped this incredibly impressive world model in Omni yesterday.

And, you know, I think Demis has talked a lot about, you know, seeing world models as a path to AGI, and it's interesting, right? 'Cause it seems like other labs maybe are more focused on code and, you know, getting to recursive self-improvement.

Um, and so I'm wondering if it's a fair characterization and, you know, why you think, uh, you know, you and the team and Google have been somewhat uniquely focused on this, uh, world model space.

First of all, I, I guess the, the coding or, like, self-improvement angle is, is a, is it at a bit of a different layer, right? So you can certainly bet and believe that, you know, these models can, can reprogram and improve themselves, and it's something I've been actually quite actively working on at the moment.

But then the object that they improve, the model, whether it's multimodal, um, and closer or a world model, as we call it, and that even how to define that is a bit abstract.

Since day one and way before actually Gemini program started, we were working on not just language but, you know, understanding the visual world and kind of jointly modeling words in the context of, um, vision, language, video, et cetera.

So I think that part, um, you know, it's been at the core of Gemini and, and before our, our research.

And I think maybe one way to characterize it is, you know, language clearly there's a lot of information, um, collectively that we wrote about the world, and so that's clearly paid off big time.

Um, we've kind of distilled in a way all the knowledge written, um, and that is being written at the moment into, into these weights.

It was definitely convenient that we put it all on the internet too.

[chuckles]

Yes.

Exactly, right.

So and, and, and also like there's now with users, right? There's obviously like also a flywheel effect.

But at the same time, there are-- there is lots of knowledge in videos and images, and what I, I would say it kind of has happened but softly, I, I think there probably might be a big moment is how would you extract all the knowledge that you would acquire if you were to look at all the videos and images, which we certainly, um, use, right? In our, in our training mixtures.

But could that knowledge somehow, um, add value and efficiency to the language component? And I think we've seen constructive, um, sort of, um, let's say transferred learning, right? From one to the other.

We see that, and we see generalization.

But probably what I would characterize as the, the GPT moment of video and images, I'm not sure we quite have seen that.

Do you have any like, uh, uh, thoughts on like what that GPT moment might be for, for, for video and image as you, as you kind of have this intuitive feeling that it hasn't yet been reached?

Yeah.

So at the moment, we train, um, all the modalities, we mix them, and we keep enhancing the recipe.

So Omni is a, a good way to see that progress in which we not only input videos and images, we, we've seen amazing capabilities with long context, um, understanding, et cetera.