Unsupervised Learning with Jacob Effron
Jun 3, 2026 · 1 hr 14 min · 10 segments
This episode with Lukasz Kaiser, co-author of the seminal "Attention Is All You Need" transformer paper and former researcher at both Google Brain and OpenAI, is a wide-ranging conversation about the…
Lukasz KaiserGuestJacob EffronHostI can think of kind of no better place to start than generalization, right? It feels like that's the, the question in the air right now.
Um, and I think in November I heard you say, you know, basically the big qu- this big question of is reasoning enough to get to generalization, or is another method needed? And I'm wondering, I guess you said that w- you know, maybe, uh, six months ago now, which is, you know, dog years in AI world, so years ago.
If you- we take the current transformers with reasoning, right? And, and, and agents, and they have access to a shell and, and, and stuff.
Like two years ago even, not to mention before transformers, I would have never believed that, you know, you just take this next word predictor, give it then chain of thought and RL that and tools, and that it will...
I now every day spend hours talking to Codex, in my case or other people just...
And it works, right? That you, you talk to it about hard problems at work, and it makes sense, and it implements things, and s- so, so that's incredible.
On the other side, hand, there is this feeling, um, that, that it is not quite like us, right? That it's not quite at the edge of what's po- that we, we all feel that it c- possibly should be even better, right? That, that we can generalize from less data, like somehow make like bigger leaps, get these concepts from way less.
I recently have this saying that y- you know, people say like Americans will do the right thing after exhausting all other options.
You need to like learn all the surface level things, and only when that doesn't explain something they will finally learn the concept.
We just get concepts from like, sometimes we make them up, and they're not great.
But s- so it does feel like there is something else that possibly could generalize much better, that could possibly have this, like a bit of a different form of understanding.
Um, but it's a feeling, right? And every time we try to put a our thumb on it, it seems to evaporate or more like ...
It, maybe it doesn't even evaporate, but it's like the transformer just catches up, right? It's, it was like ...
So, so, so, so, so both sides in this time have grown, right? Like transformers have gotten even better, but the case for something else has also gotten even better, I would say.
I can think of kind of no better place to start than generalization, right? It feels like that's the, the question in the air right now.
Um, and I think in November I heard you say, you know, basically the big qu- this big question of is reasoning enough to get to generalization, or is another method needed? And I'm wondering, I guess you said that w- you know, maybe, uh, six months ago now, which is, you know, dog years in AI world, so years ago.
If you- we take the current transformers with reasoning, right? And, and, and agents, and they have access to a shell and, and, and stuff.
Like two years ago even, not to mention before transformers, I would have never believed that, you know, you just take this next word predictor, give it then chain of thought and RL that and tools, and that it will...
I now every day spend hours talking to Codex, in my case or other people just...
And it works, right? That you, you talk to it about hard problems at work, and it makes sense, and it implements things, and s- so, so that's incredible.
On the other side, hand, there is this feeling, um, that, that it is not quite like us, right? That it's not quite at the edge of what's po- that we, we all feel that it c- possibly should be even better, right? That, that we can generalize from less data, like somehow make like bigger leaps, get these concepts from way less.
I recently have this saying that y- you know, people say like Americans will do the right thing after exhausting all other options.
You need to like learn all the surface level things, and only when that doesn't explain something they will finally learn the concept.
We just get concepts from like, sometimes we make them up, and they're not great.
But s- so it does feel like there is something else that possibly could generalize much better, that could possibly have this, like a bit of a different form of understanding.
Um, but it's a feeling, right? And every time we try to put a our thumb on it, it seems to evaporate or more like ...
It, maybe it doesn't even evaporate, but it's like the transformer just catches up, right? It's, it was like ...
So, so, so, so, so both sides in this time have grown, right? Like transformers have gotten even better, but the case for something else has also gotten even better, I would say.
The rest of this transcript — segmented and speaker-labeled, so you land on the exact moment something was said
Search every transcript — by keyword, by phrase, or by meaning, across every show Radar indexes
Trends — what is surging across podcasts, measured against its own baseline
Alerts — when a name you follow appears in a newly indexed episode
No account is needed to search Radar.