Skip to main content
Yi Tay

Yi Tay

Jan 23, 2026

SwyxHOST
49:06
Mm-hmm.
Yi Tay
Yi TayGUEST
49:06
It's not architecture itself.
Yi Tay
Yi TayGUEST
49:07
That's, that's a problem that we-- that is more of like the learning paradigm itself rather than the architecture itself.
Yi Tay
Yi TayGUEST
49:13
I think the architecture is just basically like the interface between the learning algorithm and the tokens.
Yi Tay
Yi TayGUEST
49:18
I think it's more about the learning algorithm itself, and this cont- like continual learning this like there's many ways to think about processing many like insanely large like contexts, right? Like two hundred million, one billion tokens or something like that, right? Like whether it's gonna be like you have a new le- learning algorithm that every time you run inference, you, you learn on it, right? Then you can technically have some kind of memory, like human being is learning as I'm talking to you, right? So that, that's also like one way.
Yi Tay
Yi TayGUEST
49:44
The other way is like whether, okay, maybe somebody will say that, "Okay, the attention is like just too expensive for two hundred million, one billion context, so we need new architecture." Or some people will say that, "Oh, we just improve the chips like accelerators." So I think many ways to interpret it, but I think it's like if it's about...
Yi Tay
Yi TayGUEST
50:01
There, there's a lot of fundamental things that if it's about continual learning and stuff like this, a lot of fundamental things about the learning algorithm and, and stuff like that as well.

25 MINS LATER

SwyxHOST
74:45
Is that like, unlike I, I-- Sorry, so I don't, I don't know the, the, the, the comparative delta ver-versus T5 for you guys versus BM25, but I don't expect it to be very high, and I expect it to be a lot higher for, for a true LLM-based Raxis, depending obviously on the query set.
49:06
Mm-hmm.
Yi Tay
Yi TayGUEST
49:06
It's not architecture itself.
Yi Tay
Yi TayGUEST
49:07
That's, that's a problem that we-- that is more of like the learning paradigm itself rather than the architecture itself.
Yi Tay
Yi TayGUEST
49:13
I think the architecture is just basically like the interface between the learning algorithm and the tokens.
Yi Tay
Yi TayGUEST
49:18
I think it's more about the learning algorithm itself, and this cont- like, continual learning this, like...
Yi Tay
Yi TayGUEST
49:22
There's many ways to think about processing many, like, insanely large, like, context, right? Like 200 million, 1 billion tokens, or something like that, right? Like, whether it's going to be like you have a new l- learning algorithm that every time you run inference, you, you learn on it, right? Then you can technically have some kind of memory, like human being is learning as I'm talking to you, right? So that- that's also, like, one way.
Yi Tay
Yi TayGUEST
49:44
The other way is like, whether, okay, maybe somebody will say that, "Okay, the attention is, like, just too expensive for 200 million, 1 billion context, so we need new architecture." Or some people will say that, "Oh, we just improve the chips, like accelerators." So I think many ways to interpret it, but I think it's like, if it's about-- there's a lot of fundamental things that if it's about continual learning and stuff, there's a lot of fundamental things about the learning algorithm and, and stuff like that as well.

25 MINS LATER

74:45
Is that like, unlike I, I-- sorry, uh, so I don't, I don't know the, the, the, the comparative delta v- versus T5 for you guys, versus BM25, but I don't expect it to be very high, and I expect it to be a lot higher for, for a true LLM-based Raxis, depending obviously on the query set.

We value your privacy

We use cookies to understand how you use our platform and to improve your experience. Click “Accept All” to consent, or “Decline non-essential” to opt out of non-essential cookies. Read our Privacy Policy.