Oct 18, 2026 · 20 min · 10 segments
Gemini is here! All 60 pages of the technical report read, plus the AlphaCode 2 bombshell paper explained and analysed. Is that paper even more consequential than Gemini? Plus the launch of AI…
And I must say in the webpage they were gunning for maximum hype because they had this human expert level here and they had Gemini being the first model to beat it.
The first problem that you might have noticed is that GPT-4's score was done five shots.
Basically it was given five examples to learn from before answering each question.
This is not the video to go into chain of thought or self-consistency, but it's a different way of measuring.
It's not an apples to apples comparison, and in the appendix of the technical report, we'll see a genuine comparison.
Also, I've had many conversations with the creators of the MMLU as I was doing my Smart GPT video, and that figure of 89.8% is very approximate.
The 10-second summary of the MMLU is that it's a multiple-choice test across 57 different subjects, from chemistry to business to mathematics to morality.
What's amazing about Gemini is that it's so good at so many things.
As we started getting to the end of the training, we started seeing that Gemini was better than any other model out there on these very, very important benchmarks.
For example, each of the 50 different subject areas that we tested on, it's as good as the best expert humans in those areas.
And frankly, if we had a slightly larger budget than the couple thousand dollars in API calls that we used, we could have reached 90% with GPT-4.
Furthermore, it's somewhat ridiculous to showcase these results to one decimal place when the test itself has maybe two to 3% of its questions being in error.
If you want more details, check out that video, but it's quite egregious.
So the fact that months after that Google is still touting results on this test is a little surprising to me.
I think it is.
It's just weird that they picked this particular benchmark.
And deep in the appendix of the paper, Google sheepishly gives a slightly more reasonable comparison between the two models on these text questions.
As you can see, depending on the prompting strategy, you get different results.
And I must say in the webpage they were gunning for maximum hype because they had this human expert level here and they had Gemini being the first model to beat it.
The first problem that you might have noticed is that GPT-4's score was done five shots.
Basically it was given five examples to learn from before answering each question.
This is not the video to go into chain of thought or self-consistency, but it's a different way of measuring.
It's not an apples to apples comparison, and in the appendix of the technical report, we'll see a genuine comparison.
Also, I've had many conversations with the creators of the MMLU as I was doing my Smart GPT video, and that figure of 89.8% is very approximate.
The 10-second summary of the MMLU is that it's a multiple-choice test across 57 different subjects, from chemistry to business to mathematics to morality.
What's amazing about Gemini is that it's so good at so many things.
As we started getting to the end of the training, we started seeing that Gemini was better than any other model out there on these very, very important benchmarks.
For example, each of the 50 different subject areas that we tested on, it's as good as the best expert humans in those areas.
And frankly, if we had a slightly larger budget than the couple thousand dollars in API calls that we used, we could have reached 90% with GPT-4.
Furthermore, it's somewhat ridiculous to showcase these results to one decimal place when the test itself has maybe two to 3% of its questions being in error.
If you want more details, check out that video, but it's quite egregious.
So the fact that months after that Google is still touting results on this test is a little surprising to me.
I think it is.
It's just weird that they picked this particular benchmark.
And deep in the appendix of the paper, Google sheepishly gives a slightly more reasonable comparison between the two models on these text questions.
As you can see, depending on the prompting strategy, you get different results.
The rest of this transcript — segmented and speaker-labeled, so you land on the exact moment something was said
Search every transcript — by keyword, by phrase, or by meaning, across every show Radar indexes
Trends — what is surging across podcasts, measured against its own baseline
Alerts — when a name you follow appears in a newly indexed episode
No account is needed to search Radar.