BERT
171
MENTIONS
26
EPISODES
8
PODCASTS
Search complete. 171 mentions across 26 episodes found for "BERT".
Nov 20, 2026
Differentially Private Fine-tuning of Language Models
G
3:18Gautam KamathGUEST
This is a picture of a transformer from Vaswani et al, which introduced transformer architectures.
G
3:24Gautam KamathGUEST
And in particular, you might have heard of something like BERT and GPT.
G
3:28Gautam KamathGUEST
These are typically trained and used in a two-step procedure.
G
3:31Gautam KamathGUEST
First of all, there's a pre-training step, which is trained on a large, diverse data set, which may consist of, say, data scraped from the internet.
What Could Be the Data-Structures of the Mind?
R
28:20Rina PanigrahyGUEST
It's in the email as well that I sent.
R
28:25Rina PanigrahyGUEST
So we did a very simplified version of the LSH sketch repository and attached it to a deep network, in particular to BERT.
R
28:34Rina PanigrahyGUEST
We did something extremely simple, which is use an LSH table between every two layers.
R
28:43Rina PanigrahyGUEST
We look at the output of the previous layer, and we use that to index into an LSH table, a locality-sensitive hash table.
R
29:28Rina PanigrahyGUEST
But we want to do more experiments with concatenation.
R
29:35Rina PanigrahyGUEST
So this is an observation.
R
29:38Rina PanigrahyGUEST
This is what we found on BERT.
R
29:40Rina PanigrahyGUEST
So we found that you will expand the capacity of a BERT model significantly by attaching a small amount of neural memory.
Stanford CS336 Language Modeling from Scratch | Spring 2026 | Lecture 1: Overview, Tokenization
P
4:35Percy LiangHOST
It used to be the case, maybe ten years ago, all AI researchers would just implement and train their own models.
P
4:43Percy LiangHOST
And then even eight years ago, people would download pre-trained models such as BERT and fine tune them.
P
4:52Percy LiangHOST
And I think a lot of today, you can get by by simply prompting a model.
P
4:58Percy LiangHOST
And of course, there's nothing wrong with prompting a model.
Moving fast without compromising trust: an incredible story of innovation and agility
J
2:42Jeff ReihlGUEST
In 2018, we started using this technology that Google offered.
J
2:49Jeff ReihlGUEST
They wrote a white paper on this technology called BERT, which was the first transformer model architecture.
J
2:56Jeff ReihlGUEST
And so we started using that technology.
J
2:59Jeff ReihlGUEST
So we started getting familiar with the transformer models in 2020 or 2019, I guess it was when GPT-2 came out.
Theoretical Limitations of Multi layer Transformers
B
6:43Binghui PengGUEST
The second variant is just encoder-only.
B
6:46Binghui PengGUEST
The most famous one is the BERT model.
B
6:49Binghui PengGUEST
Recall that for encoder-only model, when you compute the attention layer, the output at position i attends to every input token.
B
7:00Binghui PengGUEST
The third one is the decoder-only model, where the output at position i is determined only by previous tokens.
S
7:10speaker_0HOST
So the
B
7:10Binghui PengGUEST
decoder-only model is actually the one used by all modern language models, like GPT-Gemma and LLAMAS and so on.
B
7:19Binghui PengGUEST
And also, from a historical point of view, we see a shift from this encoder-based model like BERT to a decoder model popularized by GPTs.
B
7:31Binghui PengGUEST
And nowadays, as I said, all these modern language models are decoder-only models.
Generative Python Transformer p.3 - Preprocessing Dataset
H
0:28Harrison KinsleyHOST
So there's kind of two major styles of Transformers.
H
0:32Harrison KinsleyHOST
You've got the OpenAI's GPT, which is Generative Pre-trained Transformer, and then you've got the BERT-style model, which is... bi-directional encoder representations transformers, something like that.
H
0:46Harrison KinsleyHOST
The verbiage is slightly different, but that's what the letters stand for.
H
0:50Harrison KinsleyHOST
The big difference here is the BERT model is bi-directional and the OpenAI GPT model, I think the right word for this is like autoregressive, but essentially it goes linearly.
H
1:01Harrison KinsleyHOST
So in order to come up with some token, the GPT model only looks at tokens Previous to it.
H
1:08Harrison KinsleyHOST
So tokens are just like numerical representations of words or sub words or even individual characters.
H
1:22Harrison KinsleyHOST
But anyway, it's only going to be looking at the history of let's say if we're thinking about it in human terms.
H
1:29Harrison KinsleyHOST
it's only if you know to come up with the next token it's only going to look at the previous words that led up to that point whereas the burp model it's looking at previous words it's looking at future words so the burp model kind of fills in the gaps so to speak the the gpt model kind of generates new um new stuff now you could use the the burp model for generation um And it can do that.
Generative Python Transformer p.4 - Tokenizing
H
2:53Harrison KinsleyHOST
So we'll make another fix to that.
H
2:55Harrison KinsleyHOST
And then finally, I've got the docs up for the GPT-2 versions because this Hello World example is actually using a BERT model.
H
3:04Harrison KinsleyHOST
Now, I do plan to possibly check out using a BERT model at some point.
H
3:08Harrison KinsleyHOST
I think combining a BERT model and a GPT-2 model, like having them both at the same time, would make a pretty awesome autocomplete.
H
3:15Harrison KinsleyHOST
So I kind of want to check them both out.
H
3:17Harrison KinsleyHOST
But for now, we're going to be doing GPT-2.
Generative Python Transformer p.5 - Training and some testing of GPT-2 model
H
4:59Harrison KinsleyHOST
So GPT two config and a lot of the settings are, uh, let's see what they did.
H
5:03Harrison KinsleyHOST
What were the settings that they did in their config vocab size? So, um, I want to say the, so for the BERT model, uh, it does have a different vocab size.
H
5:13Harrison KinsleyHOST
We're going to, uh, actually, we're just going to define our own.
H
5:17Harrison KinsleyHOST
Um, and we're just going to use all the defaults.
9 MINS LATER
H
14:22Harrison KinsleyHOST
And then here, MLM probability, this is actually the default.
H
14:27Harrison KinsleyHOST
This is just what's the probability of assigning a masked token.
H
14:33Harrison KinsleyHOST
So with the BERT model, the mask can be just kind of anywhere randomly, and then the model is trained to like fill in the mask.
H
14:42Harrison KinsleyHOST
And then it uses tokens before and after that mask to fill it in.
Lecture 14 – Contextual Vectors | Stanford CS224U: Natural Language Understanding | Spring 2019
C
4:57Christopher PottsHOST
That's kind of the core idea behind this contextual word representations.
C
5:01Christopher PottsHOST
And then we're gonna talk about three approaches to learning such representations, ELMo, the transformer, and BERT.
C
5:09Christopher PottsHOST
Uh, and then the final section which we might just leave for your own self-study, just introduces you to that notebook that I posted at the start of the quarter, which shows you some easy and practical ways that you could bring ELMo and BERT into your project, even without having to do a complete code overhaul so that you're embedded in their PyTorch or TensorFlow code.
C
5:31Christopher PottsHOST
there's a bunch of associated notebooks.
C
5:33Christopher PottsHOST
So in addition to contextual reps dot the notebook, um, there's also Smith 2019, which is a nice overview paper that kind of does more of providing high-level insights about how we move from word vectors to contextual word representations, but also kind of the guiding linguistic and empirical ideas behind this whole movement.
16 MINS LATER
C
22:01Christopher PottsHOST
But at their core, these are things that are familiar to you.
C
22:04Christopher PottsHOST
So one theme will be attention.
C
22:06Christopher PottsHOST
You'll see this in the transformer and in BERT.
Lecture 12 – Evaluation Methods | Stanford CS224U: Natural Language Understanding | Spring 2019
M
2:16MinGUEST
Congratulations.
M
2:17MinGUEST
They got the score of 0.7852. Is group 26 here? And what they did was they used a combination of BERT sequence classification model with oversampling.
M
2:29MinGUEST
So they used transfer learning.
M
2:32MinGUEST
They used the pre-trained BERT end-to-end sequence classification model, and then they trained on our training data set and tested on the evaluation set.
M
2:41MinGUEST
And in order to account for the data imbalance, they used oversampling, specifically the random oversampler.
M
2:47MinGUEST
They randomly sampled with replacement on the current available samples to balance out the label distribution.
C
4:27Christopher PottsHOST
Uh, as far as I know, that's unprecedented that the same group won two bake-offs.
C
4:33Christopher PottsHOST
Although I do remember that Lucy and Jayadev on separate teams last year finished in the top two or three on two of the bake-offs, which is also very impressive.
16 more episodes mention BERT.
Create an account to see the whole feed, search across every transcript, and follow the entities you care about.