GPT-5.1

They paid for any question at the time, which is around a year ago, that the Frontier models couldn't get right.

Now, the name of that benchmark has become somewhat ironic because even without doing a web search, just using its own knowledge, so no tools, Gemini 3 Pro gets 37.5%, a huge leap above GPT 5.1. And that's a theme that you'll see recurring throughout these benchmarks.

And sticking with knowledge for a second, what about scientific knowledge in STEM subjects? That's tested in the Google-proof Q&A, GPQA Diamond.

Even the creator of this benchmark thought that model performance had plateaued, but no, Gemini 3 Pro sets a record, 92% almost.

You threw in some spatial reasoning data, didn't you? Maybe robotics data or some extra video data in that domain.

I know your secret.

Anyway, I have made Gemini 3 Pro and GPT 5.1 available on the free tier for a limited time on LM Council, so you can compare their responses side by side for your use case.

And actually, one more thing on that spatial reasoning point, because there are, of course, other benchmarks that focus on spatial reasoning, like the VPCT, and lo and behold, Gemini 3 Pro crushes the competition.

Yes, by the way, in case you're wondering, this was with reasoning effort set to extra high, not just high.

And you may be quite surprised to see it being slightly beneath GPT 5.1. That wouldn't actually be the first time for Simple Bench because GPT 5.1 itself slightly underperformed the performance of GPT 5.0, which got fifty-six point seven percent.

For other model providers, the progress is much more uniform with Opus 4.1 outperforming Opus 4, Opus 4.5 outperforming Opus 4.1, Gemini 3 outperforming Gemini 2.5, which outperformed Gemini 2, et cetera, et cetera.

If you are being extra cynical, you may wonder about benchmark maxing, where the performance in coding and mathematics and other benchmarks that are known to be highly publicized might be maximized to the detriment of the core parameter count and general knowledge, you could say, general intelligence nouse of a model.

If you need super long context up to a million tokens, Gemini 3.

Just a few more results before I leave benchmarks behind, and if you're concerned about recursive self-improvement or the singularity, well then GPT 5.2 is an incremental step forward, but no more.

On being able to successfully complete OpenAI's own pull request to a level of their standard, it got 55% versus 53% for GPT 5.1 Codex Max.

Again, on a machine learning engineering benchmark, crucial if you're gonna automate AI research, it got better than GPT 5.1 but worse than GPT 5.1 Codex Max.

For AI, if you have used two models of the same company and have bothered enough to see the difference in them.

So if you've used GPT 5.1 and then GPT 6.1, I don't know, and you've bothered to notice the differences, then you're a power user in that essence.

Yeah.

I'd say something like that.

Yeah.

And what's even more challenging is, it's, uh, moving targets 'cause, um, you know, every new model comes in, and e- even within the model, we can be using GPT 5.1 today, and GPT 5.1 in a month might give you different answers.

And there's, um, we have some intuition from human psychology as to how the way we ask the question is going to influence the answer.

These don't necessarily apply to LLMs.

Exactly.

Then in November, GPT-5.1 introduced light adaptive reasoning.

Oh, this was a huge step.

It really was because it automated that fork in the road we discussed earlier.