Yi Tay
2
APPEARANCES
1
PODCASTS
012
DEC 30
JAN 6
JAN 13
JAN 20
JAN 27
FEB 3
FEB 10
FEB 17
FEB 24
MAR 3
MAR 10
MAR 17
MAR 24
MAR 31
APR 7
APR 14
APR 21
APR 28
MAY 5
MAY 12
MAY 19
MAY 26
JUN 2
JUN 9
JUN 16
JUN 23
JUN 30
JUL 7
JUL 14
JUL 21
JUL 28
AUG 4
AUG 11
AUG 18
AUG 25
SEP 1
SEP 8
SEP 15
SEP 22
SEP 29
OCT 6
OCT 13
OCT 20
OCT 27
NOV 3
NOV 10
NOV 17
NOV 24
DEC 1
DEC 8
DEC 15
DEC 22
DEC 29
JAN 5
JAN 12
JAN 19
JAN 26
FEB 2
FEB 9
FEB 16
FEB 23
MAR 2
MAR 9
MAR 16
MAR 23
MAR 30
APR 6
APR 13
APR 20
APR 27
MAY 4
MAY 11
MAY 18
MAY 25
JUN 1
JUN 8
JUN 15
JUN 22
JUN 29
JUL 6
JUL 13
JUL 20
JUL 27
AUG 3
AUG 10
AUG 17
AUG 24
AUG 31
SEP 7
SEP 14
SEP 21
SEP 28
OCT 5
OCT 12
OCT 19
OCT 26
Jan 23, 2026
Captaining IMO Gold, Deep Think, On-Policy RL, Feeling the AGI in Singapore — Yi Tay
49:07
49:13
49:18
49:44
50:01
S
49:06SwyxHOST
Mm-hmm.
Yi TayGUEST
That's, that's a problem that we-- that is more of like the learning paradigm itself rather than the architecture itself.
Yi TayGUEST
I think the architecture is just basically like the interface between the learning algorithm and the tokens.
Yi TayGUEST
I think it's more about the learning algorithm itself, and this cont- like continual learning this like there's many ways to think about processing many like insanely large like contexts, right? Like two hundred million, one billion tokens or something like that, right? Like whether it's gonna be like you have a new le- learning algorithm that every time you run inference, you, you learn on it, right? Then you can technically have some kind of memory, like human being is learning as I'm talking to you, right? So that, that's also like one way.
Yi TayGUEST
The other way is like whether, okay, maybe somebody will say that, "Okay, the attention is like just too expensive for two hundred million, one billion context, so we need new architecture." Or some people will say that, "Oh, we just improve the chips like accelerators." So I think many ways to interpret it, but I think it's like if it's about...
Yi TayGUEST
There, there's a lot of fundamental things that if it's about continual learning and stuff like this, a lot of fundamental things about the learning algorithm and, and stuff like that as well.
25 MINS LATER
S
74:45SwyxHOST
Is that like, unlike I, I-- Sorry, so I don't, I don't know the, the, the, the comparative delta ver-versus T5 for you guys versus BM25, but I don't expect it to be very high, and I expect it to be a lot higher for, for a true LLM-based Raxis, depending obviously on the query set.
Captaining IMO Gold, Deep Think, On-Policy RL, Feeling the AGI in Singapore — Yi Tay 2
49:07
49:13
49:18
49:22
49:44
Yi TayGUEST
That's, that's a problem that we-- that is more of like the learning paradigm itself rather than the architecture itself.
Yi TayGUEST
I think the architecture is just basically like the interface between the learning algorithm and the tokens.
Yi TayGUEST
I think it's more about the learning algorithm itself, and this cont- like, continual learning this, like...
Yi TayGUEST
There's many ways to think about processing many, like, insanely large, like, context, right? Like 200 million, 1 billion tokens, or something like that, right? Like, whether it's going to be like you have a new l- learning algorithm that every time you run inference, you, you learn on it, right? Then you can technically have some kind of memory, like human being is learning as I'm talking to you, right? So that- that's also, like, one way.
Yi TayGUEST
The other way is like, whether, okay, maybe somebody will say that, "Okay, the attention is, like, just too expensive for 200 million, 1 billion context, so we need new architecture." Or some people will say that, "Oh, we just improve the chips, like accelerators." So I think many ways to interpret it, but I think it's like, if it's about-- there's a lot of fundamental things that if it's about continual learning and stuff, there's a lot of fundamental things about the learning algorithm and, and stuff like that as well.
25 MINS LATER
S
74:45Shawn WangHOST
Is that like, unlike I, I-- sorry, uh, so I don't, I don't know the, the, the, the comparative delta v- versus T5 for you guys, versus BM25, but I don't expect it to be very high, and I expect it to be a lot higher for, for a true LLM-based Raxis, depending obviously on the query set.