Skip to main content
Peter Gostev

Peter Gostev

Nov 5, 2026

13:31
What's one thing for these folks that you'd like for them to know about model evals that could help them move forward in this world?
13:40
I think, yeah, with evals, I would say it is difficult to apply them, to translate them directly to your world.
13:51
So I would say in terms of the capabilities and what maybe you should think about is that you should very seriously consider how you can use AI more to give yourself an advantage.
14:08
And what I've seen a lot of companies in reaction to saying, oh, there's this report of hacking or there's this insecurity or something like that.
14:19
And our response is that we are going to make the user to have cloud code ask for approval for every single thing that it does, for example.
14:31
Or we're not going to give Codex any access to our Slack or emails.
14:36
And this is, to me, this is just you're not doing anything.
18:37
What is one thing, one trend you are most excited to see unfold over the next 12 months?
24:59
i want this a little bit bigger can you let's take a look so this is the overall chat right and the two preferred models on arena for chat is both gemini models which is from the last two
25:09
weeks Yeah, so chat, I think, is one thing.
25:12
It is a little bit, yeah, it's debatable how important that is.
25:16
But for example, this is Code Arena, which is mostly kind of front-end tasks.
25:22
And in here, we see still Carbon is a little bit behind.
25:26
And I would say as well, I did the whole video on this, testing the model until like 1 a.m. before the release, which was a really cool thing to be able to do.
25:35
And I would say that...

33 MINS LATER

59:06
I'm pretty sure that SpaceX is a big one, given the fact that Grokbot recently moved to Opus 5.5 and SpaceX... is mentioned by the way in the s1 as one of the providers and the amount of money that the entropic is paying spacex is also revealed finally i think i don't have the number in front of me but that's also in this one uh i don't know about the other three maybe i
28:01
Peter, how big is this? How big is this GitHub drop that we're getting from OpenAI with 722 manuscripts for math?
28:11
I think it's enormous.
28:14
We really went from just models being completely hopeless at basic maths to doing this.
28:23
And the Millennium Prize was a big deal.
28:26
And it was one thing.
28:27
And obviously, it was...
28:30
No, lots of agents being thrown at it.

14 MINS LATER

42:29
I would love to hear your thoughts.
7:16
And, and so remind me, how do these models then compare to Fable and, and Astra in terms of who they're meant for and what they can do?
7:26
Yeah.
7:26
So it is interesting.
7:28
When you look at the core benchmarks that they released, and we, we saw some early bench- be- benchmarks come out, uh, they kind of, uh, well, number go up.
7:37
You know, e- every time [laughs] all of the models are better than the previous ones.
7:41
I think in practice Fable and Astra are the top tier models that will be most expensive.
7:47
So when I look at our arena's agent mode, um, approximately, uh, for our median task that people do on, on the arena, uh, Fable cost $5 to complete, Astra cost $4 to complete, uh, then Opus 5 was, I think, about somewhere like $2, and, uh, Sol 5.6 was $1.
9:24
Um, what, w- what do you think? I mean, you know, d- is there any evidence that these models were, uh, a- addressing that concern at all in your view based on what you're seeing?
58:09
I completely agree.
58:11
Yeah, and I think about this kind of imagination of doing that.
58:16
And I think there's a kind of area that is unclear how things will develop.
58:21
And there's this kind of, I think, pushback against this idea of like agents or like adoption of all of this.
58:28
Is that like, well, if you imagine the Kevin from the office of like, as your kind of typical employee, it's like, oh, are they going to vibe code themselves and SAP? You know, obviously not.
58:41
But I think that's also underestimating the imagination and the on top potential that so many people have in different roles.
58:51
They could be working in a call center, they could be working in a data department, whatever, right? All of their teams.

14 MINS LATER

73:16
And does this mean that we should be moving fast and break things?
58:09
I completely agree.
58:11
Yeah, and I think about this kind of imagination of doing that.
58:16
And I think there's a kind of area that is unclear how things will develop.
58:21
And there's this kind of, I think, pushback against this idea of like agents or like adoption of all of this.
58:28
Is that like, well, if you imagine the Kevin from the office of like, as your kind of typical employee, it's like, oh, are they going to vibe code themselves and SAP? You know, obviously not.
58:41
But I think that's also underestimating the imagination and the on top potential that so many people have in different roles.
58:51
They could be working in a call center, they could be working in a data department, whatever, right? All of their teams.

14 MINS LATER

73:16
And does this mean that we should be moving fast and break things?
LDJHOST
25:37
It seems like it's outside of the image, but there is a little, little like one mark if you guys see the image.
25:46
I'm guessing it's probably something like that.
25:48
And I think that's an interesting question about benchmarking, right? Because I think what ArcGIS were trying to do is to say, okay, we're going to give a generic harness.
25:57
So it's like for like, which I think there is an argument in that.
26:01
Just keep it generic.
26:02
But then the problem comes is that if your generic really damages some models in such a way that the actual user who uses the models experiences something completely different, then that's clearly a bad thing.
26:17
And I remember Meta was actually doing this.

33 MINS LATER

58:55
But yeah, I felt starting off GBT 5.5 as Hermes agent, that was for me the moment where it could do almost anything at least on the computer
5:32
What was your review of that model, and where do you see Gemini in the model race right now broadly?
5:40
Yeah, so Gemini, it was kind of interesting and slightly confusing to see Gemini releases, because if you remember, they were going with the bigger models a few months ago, where it was maybe 3.1 Pro was the big release, and we're kind of waiting for the next one, and never quite came.
5:58
And then Flash models would be, previously they were super cheap models, and now they became closer to the Pro models.
6:06
And it looks like they're kind of slightly changing the approach to their whole model family.
6:12
So we see pretty much Flash models while being cheaper maybe than other models.
6:17
They're certainly cheaper than like a Fable-level model.
6:20
They're certainly not as cheap as Luna model.
10:13
price? How do they rank on price?
86:06
The people get punished, and the bad guys do their bad stuff anyway.
86:08
Yeah.
86:09
I think there's also a idea in the EU, I don't know where it comes from, but this idea that all we need to do is just to give people information and then things will magically resolve, and I think it's a kind of nice idea, but we've done this with, with cookies, right? We have these cookie acceptance banners for, what? Decade and a half.
86:30
It's like, is the world better place now? And, and, uh, if you project forward from this next 10, 20 years, I don't know, it's just, it's probably not gonna mean anything.
86:41
It's probably gonna add extra, I don't know, bureaucracy for the mo- for the labs.
86:46
It's...
86:47
Then there's, Ben Thompson had the interesting point about like, does this kind of almost...
speaker_8ADVERTISER
89:02
[upbeat music]
104:15
Peter, can you talk about like Minimax H3 a little bit and like from Arena perspective? This is an open model that can run fully on your computer that's tied to the previous like highlight.
104:25
I would say the main takeaway for me in just seeing all the outputs, playing with the model and where it ranks in the leaderboard is that if, if you scroll down, it doesn't even fit in here.
104:36
But if I look at our like the website, that Veo 3 that was like the best open source or closed source video model, it's ranked 26 now.
104:48
So, and now this is number two or shared one yet, uh, yeah, basically first place.
105:05
Mm-hmm.
105:05
This is like, my gosh, amazing, like video is solved.
105:09
And then since then we had so much improvement, which is just completely unbelievable.
105:14
So yeah, amazing.
78:49
I think, yeah
78:50
...
78:50
I think it's in- in- inevitable that, uh, the price will be higher, right? If it's 2.8 trillion, I think, what was it? 1 trillion before.
78:59
Again, there's no, like, magic there, right? If, if someone has to actually run it a- and serve it.
79:04
And I, I don't know if it's a fair thing to say, probably doesn't apply to every model, but I haven't seen a lot of evidence that once the model comes out, it somehow gets optimized and, like, prices drop a lot as a, as thing, uh, as...
79:21
I was kind of hoping that would happen, and I did some analysis, like, this was, like, over a year ago, so, uh, like, this, I'm not sure how much that holds.
79:29
But I had, like, OpenRouter and I had two snapshots of the prices, like- Over, over some months.

18 MINS LATER

97:09
Otherwise-
77:26
Uh, and I would love to hear from you, how is it to actually use this model across, you know, your, your day-to-day?
77:33
Yeah.
77:33
So the, the story is, like, I think you, you did mention it earlier that, uh, they gave some people access, uh, then they yanked it away from us, and then [laughs] they, they gave it back now.
77:46
So the-- just the feeling of losing it was so crushing.
77:52
Like, I honestly, like, I was using the Codex nonstop, and then when they took it away from us, I just, like, closed Codex and, like, not opened it for, like, three days.
78:02
Well, not literally, but you know.
78:04
But not, at least outside of my work.

36 MINS LATER

113:41
Yes
4:09
Are you done? Are you, are you maxing out on your credits [laughs]?
4:13
You know, well, actually, what I was doing is, uh, 'cause through Arena, like, I can test models.
4:19
So before they find out and, uh, see how much money I spent, uh, y- uh, before everyone wakes up, I just did about, like, a hundred 3G generations.
4:31
So I had so much fun doing this.
4:33
This is honestly...
4:35
I know there's, like, not everything is good about it.
4:37
There's, like, cost, rate limits, blah, blah, blah, but it's just so cool to just be able to go do stuff, and it works so well, and it's just, it's so nice.

10 MINS LATER

15:08
Yeah.
13:10
On that point, tell the listeners, Peter, what is the current state of play on the charts? Who's doing really well and for what reason? Give us the lowdown.
13:21
If we look at the arena leaderboard, I think it's a pretty fair representation.
13:26
It kind of matches my vibes, more or less.
13:30
I think we can debate some of the details, but it's interesting if we went back maybe maybe six to eight months, maybe a bit more.
13:40
I remember we had cloud models lower down the rankings.
13:44
And I remember the feedback we used to get.
13:46
It's like, well, I love Sony 3.5. Why is this not higher? And it would kind of rank, I can't remember exactly, but something like top 10, top 12, something like that.

28 MINS LATER

42:25
Then great.
13:10
On that point, tell the listeners, Peter, what is the current state of play on the charts? Who's doing really well and for what reason? Give us the lowdown.
13:21
If we look at the arena leaderboard, I think it's a pretty fair representation.
13:26
It kind of matches my vibes, more or less.
13:30
I think we can debate some of the details, but it's interesting if we went back maybe maybe six to eight months, maybe a bit more.
13:40
I remember we had cloud models lower down the rankings.
13:44
And I remember the feedback we used to get.
13:46
It's like, well, I love Sony 3.5. Why is this not higher? And it would kind of rank, I can't remember exactly, but something like top 10, top 12, something like that.

28 MINS LATER

42:25
Then great.
19:23
Mm-hmm.
19:23
So we've got GLM 5.2 here.
19:26
So it's above 5.1, but it is kind of at the level of 5.4, like a bit 5.5, but, like, below Opus and so on.
19:35
So to me, that feels about right.
19:38
So it's a really solid model.
19:40
And we were doing some extra analysis, which hopefully we'll, we'll release today, about, like, how much tokens it's con- consuming.
19:48
And we-- I saw some reports, I don't know, maybe you guys saw as well.

16 MINS LATER

35:46
Yeah
24:51
I can't stay for long," this is, this is the model that you wanted to cover the most, I believe, right?
24:56
Yeah.
24:57
I can talk about both, and I, I'm actually really curious to hear what everyone else thinks, 'cause this was one of those releases that is quite confusing because if you look at the benchmarks, normally the way it goes, right, a company that is, like, caching up, right, not, not on topic, for example, release a model, they'll have a bunch of benchmarks with it, which are super cherry-picked.
25:19
Maybe it's like sometimes it crosses the line into, like, abuse or not.
25:24
But normally it's like, you know, a few good benchmarks.
25:27
And then other benchmarks come out, and they all kinda, "Ah, actually it's not as good as you think," kinda thing.
25:32
That's normally, that's what I normally see.

26 MINS LATER

51:05
Peter, thoughts on, on, um, on like the jagged frontier?
30:03
It's great to see you.
30:04
Yeah, good to see you guys.
30:06
Yeah, this is super exciting.
30:08
We've been thinking about how to do this for a while, so we wanted to do it right, and I think we, we got it really right.
30:15
So the way the Arena used to work is that we still can work like that, but there's a battle mode where you can go in, you put in a message, you get the responses, you vote which one is better.
30:26
And this is generally captures, like, way more than you think considering how basic it sounds.
30:32
The fact that it works at all is amazing.

6 MINS LATER

NistenHOST
36:11
How, how did you guys build this? What kind of difficulties did you find in, like, r- running sandboxes for the users? And h- how is it measured? Is this measured on user input or?
117:32
All right, folks, let's welcome Peter Goste from Arena to the show to talk about GPT 5.5.
117:38
Yeah.
117:38
So, uh, I haven't had huge amounts of time with it, but, uh, it was certainly fun to test.
117:44
I would say the biggest thing that jumps out is that it's the first time when a model can actually properly do long-running tasks.
117:54
Um, all previous models, I know they kept saying, "Oh, like, you can do it for many hours," but every time, I don't know about you guys, like, I, I just, I, I shout at it.
118:05
I do anything I can think of.
118:06
I come up with these constructs of how it's supposed to do it, and then it never does it.
121:09
Mm.

We value your privacy

We use cookies to understand how you use our platform and to improve your experience. Click “Accept All” to consent, or “Decline non-essential” to opt out of non-essential cookies. Read our Privacy Policy.