Gemini 3.1 Pro and the Downfall of Benchmarks: Welcome to the Vibe Era of AI

The latest and some would say greatest AI model has just been released, Gemini 3.1 Pro.

And in the 24 hours since release, as well as a short period of early access, I have tested it hundreds of times.

and of course Reddit's model card.

But here's the thing, for the average user, I want to get beyond the headline scores and try to give you a sense of why every new hot take you see on X or YouTube or TikTok or podcast seems to contradict the last one you saw.

Because there's actually a technical reason for the confusion over which model is best overall.

But I will say that there's one private benchmark, my own, that has recently seen a model pass a threshold that I think is worth talking about.

First, 30 seconds of context, because you may well know that the pre-training stage of growing or training LLMs involves training them on internet scale data.

actually now only accounts for 20% of the compute that is spent on training LLMs.

So it's the post-training stage, as I wrote about in my newsletter, where those generalist-based models are honed against internal benchmarks on specific domains.

This includes using industry source data to get particularly good at perhaps your domain.

Here's the catch.

Just a year ago, that wasn't the case.

Dario Amadei, CEO of Anthropic, said back then, the amount being spent on the second stage, RL stage, is small for all players.

Why did I give you that context though? Well, because if one of these labs have data relevant to your domain and post-train their models to optimize for high scores in that area, then your experience of that model might be quite different to what other benchmarks say.

In the older paradigm, if a model was clearly better in one domain, it was much more likely that there'd be better in many or all domains.

That just isn't the case anymore.

In fact, my second point in this newsletter was precisely an example of that.

Many of you would have heard about the intense discussion surrounding clawed code and all sorts of clawed powered agents that are now sweeping the web.

So we're seeing exponential improvement right across the board.

Well, let's take one chess puzzle benchmark made by Epoch AI and more on them later.

Five months ago, Claude Sonnet 4.5, which is their smaller model compared to Opus, scored 12%.

But just last week, Claude Opus 4.6, five months further on, scored just 10%.

That's not to knock Claude Opus 4.6.

I use it all the time.

It's an incredible model at coding.