How a reasoning model cracked an 80-year-old math problem - Episode 20

When you started to work on reasoning, did you have an idea of where you wanted to see this go? Like, what your expectations were? Were you looking at it purely from, "Hey, this is very cool from an academic point of view," or did you have some sort of other vision?

Uh, I think for me, the draw of reasoning when I first got excited about it was that this was something that, you know, models just obviously can't do right now.

Mm-hmm.

Um, so this was, like, end of 2023, start of 2024.

Models were, you know, struggling with grade school math problems.

Mm-hmm.

And so at that time, it was just like, can we just get the, these models to do something reasonable on m- math at all, let alone, like, have them be like, you know, like, you know, much, much better than I am at it.

Uh, I remember when I s- the first, my first day at OpenAI, uh, Nolan Brown asked me, uh, you know, when I thought models would get IMO gold.

Uh, that, that was, uh, just a benchmark we talked about.

I think at the time, a lot of people, even within research, thought that, like, you know, uh, IMO gold was out of reach this year, but maybe, like, 2026.

Mm.

I felt like, you know, I had an idea that, you know, if, if we like, if we just, like, pushed for it, maybe...

I thought, I thought we could do it by April.

Uh, it took until June to get a really good model.

Uh, but then, and then, uh, IMO rolled around and, you know, we, we were able to get gold.

And I think zooming out, uh, I think this happened a lot faster than I expected.

Mm.

And it's, I mean, it's t- it's crazy to me that it, you know, progress since then has kept up at this, uh, same sort of blistering pace.

Uh, like, it's, it was just 10 months ago, but it feels like the IMO level of problem, like, feels, like, far in, like, the rear view mirror of AI today.

Oh, Nolan Brown asked me the same question.

I mean, not about IMO gold, but about wh- whether m- whether model can solve P versus NP.

[laughs]

Mm-hmm.

I think P versus NP might be something quite hard.

Mm.