Skip to main content

Search complete. 171 mentions across 26 episodes found for "BERT".

Nov 20, 2026

Gautam KamathGUEST
3:18
This is a picture of a transformer from Vaswani et al, which introduced transformer architectures.
Gautam KamathGUEST
3:24
And in particular, you might have heard of something like BERT and GPT.
Gautam KamathGUEST
3:28
These are typically trained and used in a two-step procedure.
Gautam KamathGUEST
3:31
First of all, there's a pre-training step, which is trained on a large, diverse data set, which may consist of, say, data scraped from the internet.
Rina PanigrahyGUEST
28:20
It's in the email as well that I sent.
Rina PanigrahyGUEST
28:25
So we did a very simplified version of the LSH sketch repository and attached it to a deep network, in particular to BERT.
Rina PanigrahyGUEST
28:34
We did something extremely simple, which is use an LSH table between every two layers.
Rina PanigrahyGUEST
28:43
We look at the output of the previous layer, and we use that to index into an LSH table, a locality-sensitive hash table.
Rina PanigrahyGUEST
29:28
But we want to do more experiments with concatenation.
Rina PanigrahyGUEST
29:35
So this is an observation.
Rina PanigrahyGUEST
29:38
This is what we found on BERT.
Rina PanigrahyGUEST
29:40
So we found that you will expand the capacity of a BERT model significantly by attaching a small amount of neural memory.
Percy LiangHOST
4:35
It used to be the case, maybe ten years ago, all AI researchers would just implement and train their own models.
Percy LiangHOST
4:43
And then even eight years ago, people would download pre-trained models such as BERT and fine tune them.
Percy LiangHOST
4:52
And I think a lot of today, you can get by by simply prompting a model.
Percy LiangHOST
4:58
And of course, there's nothing wrong with prompting a model.
Jeff ReihlGUEST
2:42
In 2018, we started using this technology that Google offered.
Jeff ReihlGUEST
2:49
They wrote a white paper on this technology called BERT, which was the first transformer model architecture.
Jeff ReihlGUEST
2:56
And so we started using that technology.
Jeff ReihlGUEST
2:59
So we started getting familiar with the transformer models in 2020 or 2019, I guess it was when GPT-2 came out.
Binghui PengGUEST
6:43
The second variant is just encoder-only.
Binghui PengGUEST
6:46
The most famous one is the BERT model.
Binghui PengGUEST
6:49
Recall that for encoder-only model, when you compute the attention layer, the output at position i attends to every input token.
Binghui PengGUEST
7:00
The third one is the decoder-only model, where the output at position i is determined only by previous tokens.
speaker_0HOST
7:10
So the
Binghui PengGUEST
7:10
decoder-only model is actually the one used by all modern language models, like GPT-Gemma and LLAMAS and so on.
Binghui PengGUEST
7:19
And also, from a historical point of view, we see a shift from this encoder-based model like BERT to a decoder model popularized by GPTs.
Binghui PengGUEST
7:31
And nowadays, as I said, all these modern language models are decoder-only models.
Harrison KinsleyHOST
0:28
So there's kind of two major styles of Transformers.
Harrison KinsleyHOST
0:32
You've got the OpenAI's GPT, which is Generative Pre-trained Transformer, and then you've got the BERT-style model, which is... bi-directional encoder representations transformers, something like that.
Harrison KinsleyHOST
0:46
The verbiage is slightly different, but that's what the letters stand for.
Harrison KinsleyHOST
0:50
The big difference here is the BERT model is bi-directional and the OpenAI GPT model, I think the right word for this is like autoregressive, but essentially it goes linearly.
Harrison KinsleyHOST
1:01
So in order to come up with some token, the GPT model only looks at tokens Previous to it.
Harrison KinsleyHOST
1:08
So tokens are just like numerical representations of words or sub words or even individual characters.
Harrison KinsleyHOST
1:22
But anyway, it's only going to be looking at the history of let's say if we're thinking about it in human terms.
Harrison KinsleyHOST
1:29
it's only if you know to come up with the next token it's only going to look at the previous words that led up to that point whereas the burp model it's looking at previous words it's looking at future words so the burp model kind of fills in the gaps so to speak the the gpt model kind of generates new um new stuff now you could use the the burp model for generation um And it can do that.
Harrison KinsleyHOST
2:53
So we'll make another fix to that.
Harrison KinsleyHOST
2:55
And then finally, I've got the docs up for the GPT-2 versions because this Hello World example is actually using a BERT model.
Harrison KinsleyHOST
3:04
Now, I do plan to possibly check out using a BERT model at some point.
Harrison KinsleyHOST
3:08
I think combining a BERT model and a GPT-2 model, like having them both at the same time, would make a pretty awesome autocomplete.
Harrison KinsleyHOST
3:15
So I kind of want to check them both out.
Harrison KinsleyHOST
3:17
But for now, we're going to be doing GPT-2.
Harrison KinsleyHOST
4:59
So GPT two config and a lot of the settings are, uh, let's see what they did.
Harrison KinsleyHOST
5:03
What were the settings that they did in their config vocab size? So, um, I want to say the, so for the BERT model, uh, it does have a different vocab size.
Harrison KinsleyHOST
5:13
We're going to, uh, actually, we're just going to define our own.
Harrison KinsleyHOST
5:17
Um, and we're just going to use all the defaults.

9 MINS LATER

Harrison KinsleyHOST
14:22
And then here, MLM probability, this is actually the default.
Harrison KinsleyHOST
14:27
This is just what's the probability of assigning a masked token.
Harrison KinsleyHOST
14:33
So with the BERT model, the mask can be just kind of anywhere randomly, and then the model is trained to like fill in the mask.
Harrison KinsleyHOST
14:42
And then it uses tokens before and after that mask to fill it in.
Christopher PottsHOST
4:57
That's kind of the core idea behind this contextual word representations.
Christopher PottsHOST
5:01
And then we're gonna talk about three approaches to learning such representations, ELMo, the transformer, and BERT.
Christopher PottsHOST
5:09
Uh, and then the final section which we might just leave for your own self-study, just introduces you to that notebook that I posted at the start of the quarter, which shows you some easy and practical ways that you could bring ELMo and BERT into your project, even without having to do a complete code overhaul so that you're embedded in their PyTorch or TensorFlow code.
Christopher PottsHOST
5:31
there's a bunch of associated notebooks.
Christopher PottsHOST
5:33
So in addition to contextual reps dot the notebook, um, there's also Smith 2019, which is a nice overview paper that kind of does more of providing high-level insights about how we move from word vectors to contextual word representations, but also kind of the guiding linguistic and empirical ideas behind this whole movement.

16 MINS LATER

Christopher PottsHOST
22:01
But at their core, these are things that are familiar to you.
Christopher PottsHOST
22:04
So one theme will be attention.
Christopher PottsHOST
22:06
You'll see this in the transformer and in BERT.
MinGUEST
2:16
Congratulations.
MinGUEST
2:17
They got the score of 0.7852. Is group 26 here? And what they did was they used a combination of BERT sequence classification model with oversampling.
MinGUEST
2:29
So they used transfer learning.
MinGUEST
2:32
They used the pre-trained BERT end-to-end sequence classification model, and then they trained on our training data set and tested on the evaluation set.
MinGUEST
2:41
And in order to account for the data imbalance, they used oversampling, specifically the random oversampler.
MinGUEST
2:47
They randomly sampled with replacement on the current available samples to balance out the label distribution.
Christopher PottsHOST
4:27
Uh, as far as I know, that's unprecedented that the same group won two bake-offs.
Christopher PottsHOST
4:33
Although I do remember that Lucy and Jayadev on separate teams last year finished in the top two or three on two of the bake-offs, which is also very impressive.

16 more episodes mention BERT.

Create an account to see the whole feed, search across every transcript, and follow the entities you care about.

We value your privacy

We use cookies to understand how you use our platform and to improve your experience. Click “Accept All” to consent, or “Decline non-essential” to opt out of non-essential cookies. Read our Privacy Policy.