Christopher D. Manning

Welcome to Stanford CS224N, also known as Ling 284, Natural Language Processing with Deep Learning.

I'm Christopher Manning, and I'm the main instructor for this class.

So what we hope to do today is to dive right in.

So I'm going to spend about 10 minutes talking about the course, and then we're going to get straight into content for reasons I'll explain in a minute.

So how do we go from this clock of 100 days to something reasonable? That's what we're going to talk about.

So the scaling toolbox, at least as far as we were concerned, kind of ended up looking like three different parts across 16 GPUs because Percy and Chris Ray and I think Chris and Dan and Chris Manning and the NLP group decided to invest upfront in really powerful GPU machines so we could actually train on 16 GPUs at once.

For reference, 16 GPUs on AWS to just rent on an hourly basis is $56 an hour now.

$56 an hour if you want to just sit on them.

You might have heard about it.

It was developed here at Stanford, uh, by Jeffrey Pennington and Richard Socher and Chris Manning.

This is the paper.

And I guess I want to say right at the outset as a kind of meta comment, this is a lovely paper.

It might have even been number four at the time.

and worked a little with Chris Manning and Andrew Ng back there.

And soon after that, spent a little bit of time at Yahoo and started my first company.

We started off doing personalized search, eventually pivoted onto content recommendations, and we got acquired around a decade ago.

I think this design pattern is really interesting.

Unfortunately, I know somebody in the CS department, Chris Manning, who's actually an acquaintance of mine, talked up this paper a little bit.

And some people started saying, oh, throw away your database or data structure textbooks.

But actually, we never said that, really, in the paper.

I, I feel this at my alma mater.

I was talk- talking to Chris Manning from Stanford NLP Group, and he's saying it's really hard because you have top researchers who want to stay on campus.

They don't even want, they don't want the frontier lab roles, but they can't even do their research because they don't have the same access to compute-

Mm-hmm

I wanted to get your takes a little bit on this academia world.

We just had Chris Manning on recently from Stanford NLP, and talking exactly what you just mentioned around a lot of academics, a lot of professors, a lot of PhD students are leaving and often going towards big labs.

And even if you don't mean to, just if you wanna stay at the university and say ethically, whatever, morally, you really just want to be providing cutting-edge research out in the open, you can't really compete because you don't have the same compute, you don't have the same resources.

So wanted to get your takes there.