METR

Think about this again, okay? So they learned less and they weren't meaningfully faster, right? So the caveats here, because I always give the caveats, small sample and they measured right after the task, not six months later.

but it was published by a company that actually sells AI, which makes the whole thing much more interesting than not, right? The second case is METR, meter, July 2025.

You have heard this one on the show before and the one before that.

16 experienced open source maintainers, 246 tasks on their own repositories, With AI, they were 19% slower, and they believed AI had made them 20% faster.

And that internal wargaming actually surfaced several other weaknesses.

Plus, they brought in third party safety evaluators, METR and Redwood Research, and they found even more vulnerabilities.

It really proves that internal OPSEC just isn't enough anymore.

You need that external ecosystem of auditing.

What they missed, and if you read the report, it's very fascinating.

METR.org is an organization that sort of was a third party that came in and wrote a report about what happened.

And it's available on their website.

Anybody can download it and read it.

However we do need to measure both in terms of capability and economic impact how these models are progressing so that we can see what their impact will be in the coming years.

So in terms of measuring capability, what we cover today is we are looking at the task of how long or how complex of a task can these models accomplish? And the second question we ask is, what is the economic value of the task that these models can do, or can they do real-world tasks that humans can currently do? And the two papers that we are referencing here is, or the two metrics that we are referencing here is that METR refers to the time horizon metric calibrated by human professionals and GDP-VAL compares the win rate of model performing a task relative to experts.

So both of them are equally valid metrics in certain ways and the insights are very similar but the trends that they measure in terms of how models perform are quite different.

So let's start with the first one.

And that relates then to the question of context.

what is the duration of the task that the model can complete? So that's just the duration question.

And then the second question is, what's the percentage success? Can it do so reliably? So METR, this particular benchmark, focuses on duration of the task that model can complete and then it associates a reliability metric of like, will it complete it with 50% success or will it complete it with 80% success and so on.

And the universal anchor for these tasks ends up being the, how much time will it take humans to complete if they were professionals in solving these tasks? So there is three task suites that are used to design this particular benchmark.

Which is terrifying.

And that brings us to the METR audit from over the summer.

Outside auditors spent three weeks inside OpenAI and they discovered this systemic blind spot.

Yeah, the monitoring gap.

which could speed up medicine so much, but there are security gaps.

METR discovered a now patched open AI monitoring gap where 10% of internal codex traffic went unmonitored from May to July.

Wait, 10% went completely unmonitored?

Yeah, because the triggers only fired when control was returned to the user.