Should we slow down AI progress? | MOONSHOTS #288

All right, I'm gonna jump us in.

We have a bunch to cover in the AI space.

So let's open with a story that explains why everything is speeding up, uh, and it comes from a blog by Dwarkesh Patel.

So Dwarkesh and Jerry Han ran an experiment across six years of AI development between 2019 and 2025, asking a simple question: how much of the progress that we've been seeing is coming from better model architectures, and how much is coming from better training data, right? We've talked so much about the value of data on this pod.

So the answer, better data produces 12X improvement in compute efficiency.

Better architectures and training recipes produce 3.7X.

So data won by a factor of more than three.

So in plain English, the transformer breakthroughs get the headlines, but the quiet work of extracting, filtering, and curating what the models read has driven 3X more of the gains.

So why it matters is, you know, architectures are published and copied within months.

Data pipelines are proprietary.

If the data is a real moat and the labs with the best data engines, not the cleverest papers are gonna win, and it means the next 10X may be sitting in the data that you have someplace in your organization.

So, um, I'm gonna go to Alex and Emad here first off.

Alex, you're-- is this intuitively obvious to you?

N- not just intuitively obvious.

I actually wrote an essay on this a number of years ago-

Of course you did.

[laughs]

... called, called Data Sets Over Algorithms, arguing that the solutions to all of the grand challenges in AI historically over the past 30 years have actually been the result of putting together the correct data set, not the right algorithm.

Uh, if you look at chess, if you look at s- uh, autonomous speech recognition, if you look at Jeopardy!, these were all data set challenges.

And if you put the right training or training adjacent data set in place, that will almost always, u- usually accompanied by a competitive community and/or a benchmark, within a few years of having the right data set in place, the grand challenge gets solved.

And in some sense, the self-supervised challenge of language modeling, predicting the next token or the missing token, is the ultimate data set challenge.

So I, I don't think it's at all surprising th- this notion that curating the optimal pre-training data set for an LLM or a foundation model in, in many respects outperforms algorithmic innovation.

This is exactly what I'd expect, and also you see corollaries.

If you look at the Hutter Prize, for example, which now offers €500,000 for compressing the first one gigabyte of the English Wikipedia, all of the recent advances in compressing Wikipedia, compressing general human knowledge seem to originate from reordering the articles of Wikipedia first.

It-- basically curriculum learning, so to optimize the curriculum.