BERT
41
MENTIONS
15
EPISODES
14
PODCASTS
Search complete. 41 mentions across 15 episodes found for "BERT".
Sep 15, 2026
Emily BenderGUEST
So that's going on.
Emily BenderGUEST
And then in maybe 2018, 2019, people started claiming that BERT, which was the first thing that I think gets called a large language model-
Adam BeckerHOST
Mm-hmm
Emily BenderGUEST
... um, comes out of Google, was understanding text.
Richard SocherGUEST
Yeah.
VibhuHOST
The sentiment also came out of some of the research, right? Like the original BERT paper was trained and towards the end of the paper they're like, "Okay, throw off the last head, train specific iterations for-"
Richard SocherGUEST
Right
VibhuHOST
... uh, you know, extractive summarization, add a head for this.
VibhuHOST
Like you should do task specific stuff.
VibhuHOST
These are like the authors that wrote attention, wrote BERT telling you this is what you're meant to do.
VibhuHOST
And like the training tasks were also very odd.
VibhuHOST
They're like-
Stephen BarnettHOST
And then by 2019, they pivoted out.
Stephen BarnettHOST
And the big hit came when Google, and I remember this from my startup experience, when I was working as founder of Util, my machine learning startup, when Google first released BERT, So BERT is bi-directional encoder representations from Transformers model on its TensorFlow platform.
Stephen BarnettHOST
And any techies will remember that very, very well.
Stephen BarnettHOST
Hugging Face didn't really like the idea, thinking that a powerful model like BERT should be more widely accessible.
Stephen BarnettHOST
So they recreated BERT, but using PyTorch, which I remember again very, very well, and released it on GitHub, which achieved more than 5,000 stars, likes, in under three months.
Stephen BarnettHOST
And so this is how the initial idea for the current Hugging Face emerged.
Stephen BarnettHOST
They asked themselves, what if sharing LLMs would be as easy as sharing code on GitHub? So this is this kind of community, open source, these things, these powerful models shouldn't all be centralized by the likes of Google.
John CooganHOST
2018, this is pre-GPT-3, and it didn't become a durable consumer business.
John CooganHOST
So in 2018, Google released BERT, which was sort of the first language model, very primitive, but, uh, people were really excited about it.
John CooganHOST
But it wasn't delivered just as weights that you could download on the internet.
John CooganHOST
It was, it was delivered as a paper from Google and, uh, and the paper was implemented in Google's TensorFlow framework, and people liked PyTorch.
John CooganHOST
So the Hugging Face con- team converted BERT from TensorFlow to PyTorch and released the conversion for free, and so developers really liked that, and that became sort of like the initial go-to-market flywheel for developer adoption.
John CooganHOST
Uh, and eventually they added more and more models, eventually thousands.
John CooganHOST
Now I think they have millions of models, which is sort of crazy, but when you think about all the forks and fine tunes, it makes sense.
John CooganHOST
and it didn't become a durable consumer business.
John CooganHOST
So in 2018, Google released BERT, which was sort of the first language model, very primitive, but people were really excited about it.
John CooganHOST
But it wasn't delivered just as weights that you could download on the internet.
John CooganHOST
It was delivered as a paper from Google.
John CooganHOST
And the paper was implemented in Google's TensorFlow framework, and people liked PyTorch, so the Hugging Face team converted BERT from TensorFlow to PyTorch and released the conversion for free.
John CooganHOST
And so developers really liked that, and that became sort of like the initial go-to-market flywheel for developer adoption.
John CooganHOST
Eventually, they added more and more models, eventually thousands.
Pierce FreemanHOST
Yep.
Pierce FreemanHOST
BERT alongside it had just dropped, which was this model.
Pierce FreemanHOST
Barely.
Pierce FreemanHOST
Barely.
Pierce FreemanHOST
Yep.
Pierce FreemanHOST
Yep.
Pierce FreemanHOST
BERT came out, which was this Google model that was from the same people that did the Transformers paper, but took this fundamental building block, which was the transformer and the attention architecture, and applied that more specifically to language modeling via a pre-trained model.
Pierce FreemanHOST
But BERT was kind of only a classification model.
HackerNoon AINARRATOR
The worst case is just, many tiny tokens.
HackerNoon AINARRATOR
The older token, some older tokenizers, like the original BERT's wordpiece, don't have a special token for things they can't represent.
HackerNoon AINARRATOR
When a model emits or ingests, information is permanently lost, you can never recover what the original text was.
HackerNoon AINARRATOR
This is strictly worse than byte-level fallback, which is why modern LLMs avoid it.
Jeremy KirkHOST
People probably do.
Jeremy KirkHOST
But as far as like that accessibility, it's there, right? So, you know, your study was like looking at using this, you know, a BERT-based framework.
Jeremy KirkHOST
We'll talk about that later.
Jeremy KirkHOST
And you created something called Darkram, which kind of class can look at Telegram.
Jeremy KirkHOST
Well, you'll tell us what it does.
Jeremy KirkHOST
First, tell us what Darkram is and tell us what a BERT-based framework is and what's the goal of it.
Sayak Saha RoyGUEST
So we built Darkram.
Sayak Saha RoyGUEST
It is an automated framework that can identify the malicious posts that are being shared on Telegram.
Alex StoreyGUEST
But, you know, the reality is, is AI has been very quietly running the engine under the hood within search for more than a decade.
Alex StoreyGUEST
So I guess the easiest way to describe BERT is, so if you remember when we used to search on the web, it was almost like you'd have to use plus signs and you'd have to use quote marks and you couldn't really type a normal sentence.
Alex StoreyGUEST
So you might use rigid keywords like weather, Sydney, tomorrow or whatever.
Alex StoreyGUEST
mechanic break repair because if you typed like a human actually spoke the whole search would break so that's when we kind of came up with BERT and it stands for wait for it bi-directional encoder representations from transformers bit of a mouth I love
Daevid RichardsHOST
the fact that you even know that
Alex StoreyGUEST
But that came around in 2019.
Alex StoreyGUEST
And so before BERT, search models read words in isolation or just from left to right.
Alex StoreyGUEST
So it meant that your tiny preposition words like for or to would be completely ignored.
Max WebsterHOST
I mean, they were on the money, like, realizing, nah, the future is in, you know, being the GitHub for open source or whatever.
David KingHOST
And I think it was BERT that actually was the big takeoff for them, right? BERT? BERT? Yeah.
David KingHOST
The BERT model?
Steve LeeHOST
Oh, yeah.
Steve LeeHOST
I
David KingHOST
think that was what really solidified their early win as a GitHub-like, if you want to call it enterprise, or kind of a hub for models.
David KingHOST
I think they were the place everybody went to go find BERT.
Steve LeeHOST
The question I'm dying to know the answer to is how much did Kevin Durant influence this pivot and strategy as an early investor? I'm
5 more episodes mention BERT.
Create an account to see the whole feed, search across every transcript, and follow the entities you care about.