Benchmark

A little bit.

It's entry into a slightly different topic, but the latest in our Better Know a Benchmark series.

I am actually really excited to talk about Exploit Gym, which was the benchmark that those agents were trying to hack.

It's a little bit complicated.

It's an East Coaster, you know what I mean? You've got to take a side sometimes in Australia.

So how many teams are left? Not many.

Is it four? Sydney?

Sydney?

It'll be, hang on, wouldn't it be four? Because we've got two teams playing next week and then the week after they go into the grand final.

That's

right, four.

Yeah, so four

I was watching them to see what lovely projects the good doctor throws our way.

And the new ones I found were Benchmark, your Atari 8-bit computers using BASIC.

takes 15 to 30 minutes, and learn about scanlines using Peek in Atari BASIC.

5 to 10 minutes.

Math and Science, there's a table here in the text.

The heading row contains five columns, which read, Benchmark, GPT 5.5, GPT 6 Ustra, Delta, PP, N, Ustra.

See the original text for the table content.

Subheading.

Subheading.

SWE and Cyber, there's a table here in the text.

The heading row contains five columns, which read, Benchmark, GPT 5.5, GPT 6 Astra, Delta, PP, N, Astra.

See the original text for the table content.

But you might see a kink in that graph or like de-deceleration to some extent if models just get way more token efficiency Like, like if the next version of Gemini or Three Point Eight Flash ahhh, is a Hundred X more token efficient you might see a little bit less growth in token volumes but they might be doing more valuable work than ever sort of an interesting nuance you got to adapt to the modern times I suppose.

Jack Altman saw the Benchmarks.[laughing]and said Sam exclamation point I guess even a blind squirrel can find a nut sometimes congrats .

That's good that's good.We should move on to the jobs report because even though AIs getting better so is the US employment picture.

The U S added a hundred and sixty two thousand jobs in August sheesh That is a lot of jobs.

变数。

......那个 EU 或者是那个 memory 的 bandwidth 那样去，去比较啦。那很快应该都会有-有-有 benchmark 出来的，就是那些，就是总会有，呃，有些钱是，呃，拿来做，拍-拍这些 benchmark 片的人，是啦。

是啊，赞助的。

做这些 benchmark。

那样。

哈，那样。

Uh, we had the idea for AI infrastructure.

So we got started, but we almost immediately did the fundraise with, uh, Benchmark.

So we really appreciated all the support m- just like Eric had given us over the years.

Uh, we believe, uh, at, at least Eric was telling us that this is the biggest check that he wrote for Benchmark at le- when we started.

Um, so we were glad to have Benchmark, uh, at the start in '22.

And like, uh, like in '22, we started in '22 September.

We started looking at different ideas around recommendation systems.

Parts of that conversation.

So I do a series on the Patreon called By the Benchmarks.

And I start with Geekbench scores.

But then I want you to see, look at how these phones ranked in Geekbench.