Mm-hmm

... are a function of compute, and if you know the outputs you want, it's the same as having compute to generate with a model.

And I've seen the stages of distillation, and how I see it is that previous post-training was much more reliant on the supervised fine-tuning and other, like this RLHF era, which is where post-training started.

And you could get most of your performance just by you do this next token prediction loss on the outputs of a very strong model.

It was very established that you take the best model you can, you generate completions, and you train on it, and your model's gonna be pretty good so long as you have access to the right prompts.

Prompts have always been a key to post-training.

I remember at AI2, like with Hannah, who I co-led post-training there with, we would always say like prompts are the key unit of post-training.

Mm-hmm

Yeah

... deserve a lot of credit for being very early on the scaling regime.

And that's kinda like the earlier way of building language models before this reasoning and especially agentic capabilities took off.

And like there was enough public knowledge in this area, like because a lot of the methods were known, you scale them up and industrialize them.

You-- A lot of the things that the open research does is directionally very similar in closed labs.

There's just more complexity and more people working on different domains.

And I think this like agentic work especially has become so infra heavy and just come at a time after so many people from academia, you go on leave, professors going on leave of abs and students dropping out of their PhD.

Mm

I

think it's less about like pacing is not.

I am interested in the China stuff because there's been some interesting comments, but I feel like the pacing stuff implies that it's more interesting.

metronomic or measurable than it is.

And I think that it should be more of an industry standard that like independent people that are in all of the labs.

So there's clusters of people that can see into one or multiple labs agree that the labs have done sufficient safety work for the current capabilities of their model.

So every model has different strengths and weaknesses.

We're shipping paradigm that China has been in is going to come under a increasing pressure from sort of like state security people who are just like worried about things as well as the sort of um more straightforward like ai risk stuff which you're already seeing

Mm

... going to be fascinating.

I mean, the, the, like that example that they put is just, you know, it's like everyone's biggest fear, um, i- is, is this sort of like leakage of, of, you know, state military secrets like to, um, to the Americans or not even to, you, you know, to this like enemy AI company that, that Dario comma enemy of China has set up.

For years and years, I was reading all of these, you know, it's a very small community of like, you know, people in the West who follow Chinese AI safety debates.

And for a while it was clear you could just write it off into, down to zero because this was not something that like the government really cared about.

All of a sudden she gives this speech, the hem of...

the head of the MSS writes a, um, writes a blog post.

Mm-hmm

What do you think we will see? Do you think like the future is open source models or maybe like closed ones? I think in

many ways, there's a default where people want to use open models, but it's very unlikely that the capabilities keep up.

I think the...

The primary reason is that the people building the best models, there's, in order to get like economic feedback loops as being a platform, the open model builders have a very indirect capture of value from their models back to their business versus open AI and anthropic have the fastest growing businesses of all time because they're integrated from model to harness to product.

And this is very hard to overcome.

And at the same time, I thought that the effects of this would be seen sooner, which is like the open models keep up to a remarkable degree.

I think a lot of this is just psychology and competition of the fact that once you know a model is buildable, it's a lot easier to chase it down.

What about, so like, how do you see, so like, we talked a bit about it at the beginning about the open ecosystem, but what do you think we actually should do? And what you, you try to do?

And yeah, I don't know if you have a reaction to all that.

Yeah, so my biggest reaction was that it was very inconsistent, which if you're doing safety as your top-line priority, it seems like consistency in how you approach your safety measures would be the default top-line item before you can release a model.

So that's why I was kind of like, okay, you guys are talking a bit of the talk too much when releasing this fabled model, which you have hyped up to be the most dangerous thing in AI ever.

And then you're like, well, we're going to do this safety mechanism and... two different ways.

And it's like, well, if one of them is more safe, why don't you use that for every domain? It's like, what are you doing? And to be specific, it was cybersecurity, research biology, and distillation was on the classifier side.

And then the frontier AI research was on the silent modification side.

The classifier is being very overeager at release.

the US federal government going when it comes to model restrictions and AI regulation?

Yes

... which kind of informs all of this.

The scaling law is a power law relationship between, you can think of the X-axis, so kind of what you are scaling is a combination of compute and data, which are kind of similar, and then the Y-axis is like the held-out prediction accuracy over our next tokens.

We talk about models being autoregressive.

It's like if you keep a set of w- text that the model has not seen, how accurate will it get when you're all trained? And the idea of scaling laws came when people figured out that that was a very predictable relationship, and I think that that technical term is continuing, and then the question is, like, what do users get out of it? And then there are more types of scaling where, um, OpenAI's o1 was famous for introducing inference time scaling, and I think less famously for also showing that you can scale reinforcement learning training and get kind of this log X-axis and then a linear increase in performance on Y-axis.

So there's kind of these three axes now where the traditional scaling laws are talk- talked about for pre-training, which is how big your model is and how big your dataset is, and then scaling reinforcement learning, which is like how long can you do this trial-and-error learning that we-- we'll talk about, we'll define more of this.

And then this inference time compute, which is just letting the model generate more tokens on a specific problem.

What kind of other ideas are there on post-training?

Yeah.

What the goal is, is try to do the work to compress what are complicated industry post-training recipes into something somewhat tractable that you can modify on your own and do post-training at a, what is like actual state-of-the-art level.

I think what we do relative to Frontier Labs is that we probably have a smaller amount of tasks.

I think our post-training suite for TULU is probably like 10 to 15 tasks.

But I would guess post-training at OpenAI et al you have maybe hundreds of evals.

And adding more evals is more data work and more mixing work and making sure you have these things.

But on the core evals for our suite of models from I think 870 and 405Bs based on LLaMA at the time, it's like it matches or beats Meta on these core evals.

Yeah.

Mm-hmm.

So, the long tail of if QuEN is going to keep doing this, and then it's something like Huawei, they start working with Huawei, there's like Huawei libraries, they want to support them.

And then US researchers are starting to dig down into those levels because they want to understand how these models were trained.

So, those are the people that have the most direct, uh, exposure, would be NVIDIA and AMD.

It's cheaper for them to do.

Uh, they get the benefit of the researchers keep working on their hardware and software ecosystems.

There are, like, outlandish stories that you could tell about open source AI where that type of thing could emerge.

Last question for me, uh, what are you expecting out of, uh, OpenAI's, uh, open model?

So what is pre-training and what is post-training? And what are the different flavors of things underneath post-training umbrella?

Yeah, so pre-training, I'm using some of the same words to really get the message across, is you're doing what is called autoregressive prediction to predict the next token in a series of documents.

This is done over standard practice is trillions of tokens.

So this is a ton of data that is mostly scraped from the web.

In some of DeepSeek's earlier papers, they talk about their training data being distilled for math.

I, I shouldn't use this word yet, but taken from Common Crawl, and that's a public access that anyone listening to this could go download data from the Common Crawl website.

This is a crawler that is maintained publicly.

Just people that don't know anything about AI, they show up, like, what's the actual experience? What's the use case for each one when they actually, like, type and talk to it? What, what is each good at and that kind of thing?