Can you tell me a little bit more about those research projects specifically?

Sure.

Those are examples of cases where we had to sort of have innovative insights as to how to set up the evaluation to really be able to control and test the model's behavior in those scenarios.

So for multi-turn conversations, what we did is we took single-turn benchmarks and maybe I should back up and say, one of the things we observe about users in practice is that they often under-specify what they're trying to do.

They don't know how to give a complete specification to begin with.

We sometimes forget this as computer scientists and software engineers, but lay users might not be able to fully specify things in the first turn.

And so they kind of figure out what they're doing over multiple turns and sort of add conditions and clarifications throughout the terms.

Where do you see the science of large AI systems going next? Broadly, maybe that's a really hard question, but at least in your interests and research directions, where do you see the science of these systems going next?