Mm.

-that, that are just, um, like involve, uh, like doing a much more complicated task or doing something that requires a lot more cleverness.

And, uh, you could, you could say this is like the benchmarking distribution, 'cause a lot of the most prominent benchmarks just involve doing some very hard puzzle-like task, uh, that's easy to verify.

And then there's sort of, uh, like the realism axis, where you want the model to be good in the realistic coding agent setting, where there's like multiple back and forth with the human and there's like multiple objectives.

And, uh, like I'd say, um, like the people-- like the labs who are crafting the model behavior for the first time need to push in both directions.

And to get good model behavior, you need to really push on the realism axis and have like rubrics or some kind of human feedback that's informing, uh, the reward function you use there.

Um, but, uh, I think when, if you try to do distillation naively, you end up just sort of matching the teacher on the benchmarking distribution.

Um, yeah, I don't know if you guys have thoughts on this.