No Priors: Artificial Intelligence | Technology | Startups
Aug 1, 2024 · 46 min · 10 segments
In this episode of No Priors, hosts Sarah and Elad are joined by Oriol Vinyals, VP of Research, Deep Learning Team Lead, at Google DeepMind and Technical Co-lead of the Gemini project. Oriol shares…
Oriol VinyalsGuest
Elad GilHost
Sarah GuoHost
I mean, there's one reflection that even many years ago with friends who were kind of early, quote unquote, in the game, they said, well, get ready.

And you certainly see this, right, with open sourcing and a bit of a random search even, right? It's not like, I mean, you just selection bias, like someone does something random, but...

So I think there's the sheer size of the field that is one aspect that I think we were sort of anticipating.

But to me, that's one of the biggest changes that I've seen, that there's more brains, more different backgrounds coming into the field.

I usually tend to assign credit uh what's with what has happened to of course the scale of data and compute algorithmic advances that you can simplify but there's certainly been some that have been important in the last 10 20 years or so and then actually the accessibility right the software the open sourcing efforts those have been quite critical to then create these sort of exponentials or linear trends in log scale that we're seeing now a bit more into sort of how I see the field from maybe like, I tend to call maybe the 2000, let's say 10 to 20, like deep learning era.

So what that era did, right, is it took a set of algorithms that were general, right? The algorithms are like stochastic gradient descent, deep learning, neural networks, right? Reinforcement learning.

And then you just apply them to a domain and you get extremely good at that domain, right? So we have, you know, mastering the game of Go, beating the ImageNet challenge, becoming state-of-the-art speech recognition, state-of-the-art image generations, right? So that decade is kind of, The models themselves certainly are not general, but the algorithms are general, right? You could just take the same algorithm and then change the data set and of course tweak a couple of things and voila, you get like protein folding really enhanced, right, from the traditional methods.

And I think then the greatest insight, of course, came from realizing that, and I think that's a lucky factor for us because it makes communication with these entities much easier, but modeling language turns out to be such a powerful abstraction for generality.

So, of course, the GPT-2 paper especially posed that sort of In the abstract, it says, look, you can solve every task, not one task, by modeling language.

And then perfecting that, the whole field, of course, building up from a lot of years of research, created what is not just the algorithms are reusable, but actually the model now becomes general.

So that's why I think like, OK, AGI is getting closer, that we have powerful models first in 2010 to 20.

And I think multimodality has been more recently another amazing breakthrough that these techniques expand not to language, but also to vision, sound, videos, et cetera.

That means that we have very powerful general models that from an AGI definition standpoint, it starts to tick many boxes.

Reasoning capabilities of the models are there, but I don't think we've perfected sort of making the reasoning very crisp and accurate so that these models would not hallucinate or would not, you know, the model might solve an Olympiad mathematical problem, yet fail to then discern a very basic puzzle, right? And I think what you do with the model, this reasoning step, there's a lot of ideas and a lot of experience and a lot of algorithmic advances we've also done in the last few years over like search and so on, but we haven't quite perfected this.

And then of course the question is, you know, push the frontier and move forward in certain domains at least.

I mean, there's one reflection that even many years ago with friends who were kind of early, quote unquote, in the game, they said, well, get ready.

And you certainly see this, right, with open sourcing and a bit of a random search even, right? It's not like, I mean, you just selection bias, like someone does something random, but...

So I think there's the sheer size of the field that is one aspect that I think we were sort of anticipating.

But to me, that's one of the biggest changes that I've seen, that there's more brains, more different backgrounds coming into the field.

I usually tend to assign credit uh what's with what has happened to of course the scale of data and compute algorithmic advances that you can simplify but there's certainly been some that have been important in the last 10 20 years or so and then actually the accessibility right the software the open sourcing efforts those have been quite critical to then create these sort of exponentials or linear trends in log scale that we're seeing now a bit more into sort of how I see the field from maybe like, I tend to call maybe the 2000, let's say 10 to 20, like deep learning era.

So what that era did, right, is it took a set of algorithms that were general, right? The algorithms are like stochastic gradient descent, deep learning, neural networks, right? Reinforcement learning.

And then you just apply them to a domain and you get extremely good at that domain, right? So we have, you know, mastering the game of Go, beating the ImageNet challenge, becoming state-of-the-art speech recognition, state-of-the-art image generations, right? So that decade is kind of, The models themselves certainly are not general, but the algorithms are general, right? You could just take the same algorithm and then change the data set and of course tweak a couple of things and voila, you get like protein folding really enhanced, right, from the traditional methods.

And I think then the greatest insight, of course, came from realizing that, and I think that's a lucky factor for us because it makes communication with these entities much easier, but modeling language turns out to be such a powerful abstraction for generality.

So, of course, the GPT-2 paper especially posed that sort of In the abstract, it says, look, you can solve every task, not one task, by modeling language.

And then perfecting that, the whole field, of course, building up from a lot of years of research, created what is not just the algorithms are reusable, but actually the model now becomes general.

So that's why I think like, OK, AGI is getting closer, that we have powerful models first in 2010 to 20.

And I think multimodality has been more recently another amazing breakthrough that these techniques expand not to language, but also to vision, sound, videos, et cetera.

That means that we have very powerful general models that from an AGI definition standpoint, it starts to tick many boxes.

Reasoning capabilities of the models are there, but I don't think we've perfected sort of making the reasoning very crisp and accurate so that these models would not hallucinate or would not, you know, the model might solve an Olympiad mathematical problem, yet fail to then discern a very basic puzzle, right? And I think what you do with the model, this reasoning step, there's a lot of ideas and a lot of experience and a lot of algorithmic advances we've also done in the last few years over like search and so on, but we haven't quite perfected this.

And then of course the question is, you know, push the frontier and move forward in certain domains at least.
The rest of this transcript — segmented and speaker-labeled, so you land on the exact moment something was said
Search every transcript — by keyword, by phrase, or by meaning, across every show Radar indexes
Trends — what is surging across podcasts, measured against its own baseline
Alerts — when a name you follow appears in a newly indexed episode
No account is needed to search Radar.