
Peter Gostev
19
APPEARANCES
7
PODCASTS
024
DEC 30
JAN 6
JAN 13
JAN 20
JAN 27
FEB 3
FEB 10
FEB 17
FEB 24
MAR 3
MAR 10
MAR 17
MAR 24
MAR 31
APR 7
APR 14
APR 21
APR 28
MAY 5
MAY 12
MAY 19
MAY 26
JUN 2
JUN 9
JUN 16
JUN 23
JUN 30
JUL 7
JUL 14
JUL 21
JUL 28
AUG 4
AUG 11
AUG 18
AUG 25
SEP 1
SEP 8
SEP 15
SEP 22
SEP 29
OCT 6
OCT 13
OCT 20
OCT 27
NOV 3
NOV 10
NOV 17
NOV 24
DEC 1
DEC 8
DEC 15
DEC 22
DEC 29
JAN 5
JAN 12
JAN 19
JAN 26
FEB 2
FEB 9
FEB 16
FEB 23
MAR 2
MAR 9
MAR 16
MAR 23
MAR 30
APR 6
APR 13
APR 20
APR 27
MAY 4
MAY 11
MAY 18
MAY 25
JUN 1
JUN 8
JUN 15
JUN 22
JUN 29
JUL 6
JUL 13
JUL 20
JUL 27
AUG 3
AUG 10
AUG 17
AUG 24
AUG 31
SEP 7
SEP 14
SEP 21
SEP 28
OCT 5
OCT 12
OCT 19
OCT 26
NOV 2
NOV 9
NOV 16
Nov 5, 2026
Evaluating the safety and reliability of frontier models
13:31
13:40
13:51
14:08
14:19
18:37

Cal Al-DhubaibHOST
What's one thing for these folks that you'd like for them to know about model evals that could help them move forward in this world?

Peter GostevGUEST
I think, yeah, with evals, I would say it is difficult to apply them, to translate them directly to your world.

Peter GostevGUEST
So I would say in terms of the capabilities and what maybe you should think about is that you should very seriously consider how you can use AI more to give yourself an advantage.

Peter GostevGUEST
And what I've seen a lot of companies in reaction to saying, oh, there's this report of hacking or there's this insecurity or something like that.

Peter GostevGUEST
And our response is that we are going to make the user to have cloud code ask for approval for every single thing that it does, for example.

Cal Al-DhubaibHOST
What is one thing, one trend you are most excited to see unfold over the next 12 months?
ThursdAI - Oct 29 - Gemini 4 Carbon is the new #1, Anthropic files to go public & more
24:59
25:26
59:06
Alex VolkovHOST
i want this a little bit bigger can you let's take a look so this is the overall chat right and the two preferred models on arena for chat is both gemini models which is from the last two

Peter GostevGUEST
And I would say as well, I did the whole video on this, testing the model until like 1 a.m. before the release, which was a really cool thing to be able to do.
33 MINS LATER
Alex VolkovHOST
I'm pretty sure that SpaceX is a big one, given the fact that Grokbot recently moved to Opus 5.5 and SpaceX... is mentioned by the way in the s1 as one of the providers and the amount of money that the entropic is paying spacex is also revealed finally i think i don't have the number in front of me but that's also in this one uh i don't know about the other three maybe i
ThursdAI - Oct 8 - OpenAI drops 722 math papers, Haiku 5.5 hits 10 cents & more
28:01
28:14
Alex VolkovHOST
Peter, how big is this? How big is this GitHub drop that we're getting from OpenAI with 722 manuscripts for math?

Peter GostevGUEST
We really went from just models being completely hopeless at basic maths to doing this.
14 MINS LATER
OpenAI and Anthropic Release Cheaper AI Models, Investors Visit China as Country’s AI Prowess Rises
7:16
7:28
7:37
7:41
7:47
9:24
Akash PasrichaHOST
And, and so remind me, how do these models then compare to Fable and, and Astra in terms of who they're meant for and what they can do?

Peter GostevGUEST
When you look at the core benchmarks that they released, and we, we saw some early bench- be- benchmarks come out, uh, they kind of, uh, well, number go up.

Peter GostevGUEST
You know, e- every time [laughs] all of the models are better than the previous ones.

Peter GostevGUEST
I think in practice Fable and Astra are the top tier models that will be most expensive.

Peter GostevGUEST
So when I look at our arena's agent mode, um, approximately, uh, for our median task that people do on, on the arena, uh, Fable cost $5 to complete, Astra cost $4 to complete, uh, then Opus 5 was, I think, about somewhere like $2, and, uh, Sol 5.6 was $1.
Akash PasrichaHOST
Um, what, w- what do you think? I mean, you know, d- is there any evidence that these models were, uh, a- addressing that concern at all in your view based on what you're seeing?
Peter Gostev | On benchmarking the frontier, and why human accountability remains key | #101
58:21
58:28
58:41
58:51

Peter GostevGUEST
And there's this kind of, I think, pushback against this idea of like agents or like adoption of all of this.

Peter GostevGUEST
Is that like, well, if you imagine the Kevin from the office of like, as your kind of typical employee, it's like, oh, are they going to vibe code themselves and SAP? You know, obviously not.

Peter GostevGUEST
But I think that's also underestimating the imagination and the on top potential that so many people have in different roles.

Peter GostevGUEST
They could be working in a call center, they could be working in a data department, whatever, right? All of their teams.
14 MINS LATER
Peter Gostev | On benchmarking the frontier, and why human accountability remains key | #101
58:21
58:28
58:41
58:51

Peter GostevGUEST
And there's this kind of, I think, pushback against this idea of like agents or like adoption of all of this.

Peter GostevGUEST
Is that like, well, if you imagine the Kevin from the office of like, as your kind of typical employee, it's like, oh, are they going to vibe code themselves and SAP? You know, obviously not.

Peter GostevGUEST
But I think that's also underestimating the imagination and the on top potential that so many people have in different roles.

Peter GostevGUEST
They could be working in a call center, they could be working in a data department, whatever, right? All of their teams.
14 MINS LATER
Sep 3 Part 2: Welcome to AGI - OpenAI unveils GPT-6 Astra, 99% on Arc-AGI and incredible at computer use.
25:48
26:02
L
25:37LDJHOST
It seems like it's outside of the image, but there is a little, little like one mark if you guys see the image.

Peter GostevGUEST
And I think that's an interesting question about benchmarking, right? Because I think what ArcGIS were trying to do is to say, okay, we're going to give a generic harness.

Peter GostevGUEST
But then the problem comes is that if your generic really damages some models in such a way that the actual user who uses the models experiences something completely different, then that's clearly a bad thing.
33 MINS LATER
W
58:55But yeah, I felt starting off GBT 5.5 as Hermes agent, that was for me the moment where it could do almost anything at least on the computer
Anthropic, Meta and Google Release New Models, Snowflake Shares Surge, Google Avoids Break-Up
5:32
5:40
5:58
6:06
6:12
Akash PasrichaHOST
What was your review of that model, and where do you see Gemini in the model race right now broadly?

Peter GostevGUEST
Yeah, so Gemini, it was kind of interesting and slightly confusing to see Gemini releases, because if you remember, they were going with the bigger models a few months ago, where it was maybe 3.1 Pro was the big release, and we're kind of waiting for the next one, and never quite came.

Peter GostevGUEST
And then Flash models would be, previously they were super cheap models, and now they became closer to the Pro models.

Peter GostevGUEST
And it looks like they're kind of slightly changing the approach to their whole model family.

Peter GostevGUEST
So we see pretty much Flash models while being cheaper maybe than other models.
ThursdAI - Grok 4.6 & Grok Bot at the frontier, DeepSeek v4 Pro GA, Meta opens Muse Glimmer and promises spark, Gemini gives us 3.7 flash instead of Pro & more AI news
86:09
86:30
86:41
86:47

Peter GostevGUEST
I think there's also a idea in the EU, I don't know where it comes from, but this idea that all we need to do is just to give people information and then things will magically resolve, and I think it's a kind of nice idea, but we've done this with, with cookies, right? We have these cookie acceptance banners for, what? Decade and a half.

Peter GostevGUEST
It's like, is the world better place now? And, and, uh, if you project forward from this next 10, 20 years, I don't know, it's just, it's probably not gonna mean anything.

Peter GostevGUEST
It's probably gonna add extra, I don't know, bureaucracy for the mo- for the labs.

Peter GostevGUEST
Then there's, Ben Thompson had the interesting point about like, does this kind of almost...
S
89:02speaker_8ADVERTISER
[upbeat music]
ThursdAI - Aug 06 - Google shakeup, Details on OpenAI hack, 2 new agent harnesses, 4 video models (1 Open) and 3 guest segments
104:15
104:25
104:36
104:48
105:09
Alex VolkovHOST
Peter, can you talk about like Minimax H3 a little bit and like from Arena perspective? This is an open model that can run fully on your computer that's tied to the previous like highlight.

Peter GostevGUEST
I would say the main takeaway for me in just seeing all the outputs, playing with the model and where it ranks in the leaderboard is that if, if you scroll down, it doesn't even fit in here.

Peter GostevGUEST
But if I look at our like the website, that Veo 3 that was like the best open source or closed source video model, it's ranked 26 now.

Peter GostevGUEST
So, and now this is number two or shared one yet, uh, yeah, basically first place.

Peter GostevGUEST
And then since then we had so much improvement, which is just completely unbelievable.
ThursdAI - Jul 16 - Inkling 975B open weights, Kimi K3 at 2.8T, a 27B model on a phone & Codex hits 9M
78:50
78:59
79:04
79:21
79:29

Peter GostevGUEST
I think it's in- in- inevitable that, uh, the price will be higher, right? If it's 2.8 trillion, I think, what was it? 1 trillion before.

Peter GostevGUEST
Again, there's no, like, magic there, right? If, if someone has to actually run it a- and serve it.

Peter GostevGUEST
And I, I don't know if it's a fair thing to say, probably doesn't apply to every model, but I haven't seen a lot of evidence that once the model comes out, it somehow gets optimized and, like, prices drop a lot as a, as thing, uh, as...

Peter GostevGUEST
I was kind of hoping that would happen, and I did some analysis, like, this was, like, over a year ago, so, uh, like, this, I'm not sure how much that holds.

Peter GostevGUEST
But I had, like, OpenRouter and I had two snapshots of the prices, like- Over, over some months.
18 MINS LATER
AI WorldCup (or superbowl?) GPT-5.6 lands mid-show, Zuck returns to X for Muse Spark 1.1, GPT-Live talks while it listens & Grok 4.5 trained with Cursor, Fable extended - ThursdAI - Jul 9, 2026
77:26
77:33
77:52
Alex VolkovHOST
Uh, and I would love to hear from you, how is it to actually use this model across, you know, your, your day-to-day?

Peter GostevGUEST
So the, the story is, like, I think you, you did mention it earlier that, uh, they gave some people access, uh, then they yanked it away from us, and then [laughs] they, they gave it back now.

Peter GostevGUEST
Like, I honestly, like, I was using the Codex nonstop, and then when they took it away from us, I just, like, closed Codex and, like, not opened it for, like, three days.
36 MINS LATER
ThursdAI - July 2 - LIVE from AI Engineer World's Fair 🎪 Long LIVE
4:13
4:19
4:37

Peter GostevGUEST
You know, well, actually, what I was doing is, uh, 'cause through Arena, like, I can test models.

Peter GostevGUEST
So before they find out and, uh, see how much money I spent, uh, y- uh, before everyone wakes up, I just did about, like, a hundred 3G generations.

Peter GostevGUEST
There's, like, cost, rate limits, blah, blah, blah, but it's just so cool to just be able to go do stuff, and it works so well, and it's just, it's so nice.
10 MINS LATER
How AI Models Are Really Judged, with Peter Gostev (Arena / LMArena)
13:10
13:21
13:30
13:46

Georgie HealyHOST
On that point, tell the listeners, Peter, what is the current state of play on the charts? Who's doing really well and for what reason? Give us the lowdown.

Peter GostevGUEST
If we look at the arena leaderboard, I think it's a pretty fair representation.

Peter GostevGUEST
I think we can debate some of the details, but it's interesting if we went back maybe maybe six to eight months, maybe a bit more.

Peter GostevGUEST
It's like, well, I love Sony 3.5. Why is this not higher? And it would kind of rank, I can't remember exactly, but something like top 10, top 12, something like that.
28 MINS LATER
How AI Models Are Really Judged, with Peter Gostev (Arena / LMArena)
13:10
13:21
13:30
13:46

Georgie HealyHOST
On that point, tell the listeners, Peter, what is the current state of play on the charts? Who's doing really well and for what reason? Give us the lowdown.

Peter GostevGUEST
If we look at the arena leaderboard, I think it's a pretty fair representation.

Peter GostevGUEST
I think we can debate some of the details, but it's interesting if we went back maybe maybe six to eight months, maybe a bit more.

Peter GostevGUEST
It's like, well, I love Sony 3.5. Why is this not higher? And it would kind of rank, I can't remember exactly, but something like top 10, top 12, something like that.
28 MINS LATER
Open source AI just had its 2nd DeepSeek moment (and this time the market didn't crash)
19:26
19:40

Peter GostevGUEST
So it's above 5.1, but it is kind of at the level of 5.4, like a bit 5.5, but, like, below Opus and so on.

Peter GostevGUEST
And we were doing some extra analysis, which hopefully we'll, we'll release today, about, like, how much tokens it's con- consuming.
16 MINS LATER
Fable Got Banned, Open Source Delivered: GLM-5.2, Kimi K2.7 & SpaceX Buys Cursor - June 18
24:51
24:57
25:27
Alex VolkovHOST
I can't stay for long," this is, this is the model that you wanted to cover the most, I believe, right?

Peter GostevGUEST
I can talk about both, and I, I'm actually really curious to hear what everyone else thinks, 'cause this was one of those releases that is quite confusing because if you look at the benchmarks, normally the way it goes, right, a company that is, like, caching up, right, not, not on topic, for example, release a model, they'll have a bunch of benchmarks with it, which are super cherry-picked.

Peter GostevGUEST
And then other benchmarks come out, and they all kinda, "Ah, actually it's not as good as you think," kinda thing.
26 MINS LATER
📅 ThursdAI - Jun 4 - NVIDIA drops Nemotron 3 Ultra (550B open), Microsoft becomes a frontier lab, Ideogram 4 goes open, Agent Arena & more
30:08
30:15
30:26

Peter GostevGUEST
We've been thinking about how to do this for a while, so we wanted to do it right, and I think we, we got it really right.

Peter GostevGUEST
So the way the Arena used to work is that we still can work like that, but there's a battle mode where you can go in, you put in a message, you get the responses, you vote which one is better.

Peter GostevGUEST
And this is generally captures, like, way more than you think considering how basic it sounds.
6 MINS LATER
N
36:11NistenHOST
How, how did you guys build this? What kind of difficulties did you find in, like, r- running sandboxes for the users? And h- how is it measured? Is this measured on user input or?
📅 Apr 23: OpenAI's Week: GPT-5.5, GPT-Image-2, Codex CUA + Chronicle, + Claude Design, Kimi K2.6, Qwen 3.6-27B
117:32
117:38
117:44
117:54
118:06
Alex VolkovHOST
All right, folks, let's welcome Peter Goste from Arena to the show to talk about GPT 5.5.

Peter GostevGUEST
So, uh, I haven't had huge amounts of time with it, but, uh, it was certainly fun to test.

Peter GostevGUEST
I would say the biggest thing that jumps out is that it's the first time when a model can actually properly do long-running tasks.

Peter GostevGUEST
Um, all previous models, I know they kept saying, "Oh, like, you can do it for many hours," but every time, I don't know about you guys, like, I, I just, I, I shout at it.

Peter GostevGUEST
I come up with these constructs of how it's supposed to do it, and then it never does it.
