Word2vec

Google's AI infrastructure evolved alongside the model.

So Dean kind of walks through Google's progression from distributed network training with their DISC belief tool, unsupervised learning on millions of YouTube frames, Word2Vec as a tool, sequence-to-sequence models, and eventually the TPU program that he talks about here.

And then he goes on to explain transformers and sparse models and pathways and how that fundamentally paved the road for larger scale AI and changes there in industry.

So he explains why transformers replace LSTMs by eliminating sequential bottlenecks through attention mechanisms.

They started the whole transformer thing.

They started tokenization back in as early as what, 2014, 2015, when they first started publishing some of the like Word2Vec, which is the predecessor of GPT-1 and GPT-2.

My hypothesis is because there's so much turf wars inside of Google where you've got, you know, people that have been in one place for a long time and there's all these internal organs and they don't they're not allowed to talk to each other and that sort of thing.

That's my hypothesis.

I published something called Gov2vec.

So after Word2vec came out from Google, Word2vec was the idea of training neural networks on a lot of text in an unsupervised way, where you're effectively just predicting the next word across a lot of text from the internet.

'Cause what I did is I took that same basic idea but adapted that methodology to specifically focusing on legal and policy text and then capturing concepts that were latent within that text.

So for example, uh, finding that if we could train on a bunch of text, the models could automatically learn the distinction between a Republican and a Democrat, the distinction between different branches of government, and things like that.

It's impacted at many levels.

Like, my background is in linguistic modeling, so it's so fascinating to see this development of what used to be like Word2Vec into large language models now, where it's so radically different-

Yeah

...