Jun 1, 2026 · 32 min · 10 segments
Knowing when an AI agent has failed sounds straightforward — until it isn't. Agents have a frustrating habit of finishing confidently while quietly doing the wrong thing, or looping endlessly without…
Hi, hi, hi, and welcome.
We are talking about agents again today.
And so today, episode seven, we're gonna be talking about evaluations.
We talked last time about failure modes.
That conversation assumed that there was some way of measuring what a failure even was.
We talked about things like compound errors or coordination problems, and as you may be aware, or as you might have picked up from some of that conversation, a lot of agent failures, they don't look super obvious.
Uh, they look like the agent finishing confidently, but it actually didn't do what it said it did, or going around in circles, uh, not creating, like, an obvious crash, for instance.
And so when you don't have that glaring failure mode like, "Hey, this went badly," if failure is hard to see, then how do you measure it systematically? That's what we're gonna be talking about today.
How do you know whether an agent is working? You're listening to Linear Digressions.
[rock music] Now, if you're listening to this, I assume that you've used an AI chatbot before, at least.
Read the full transcript.
Create an account to read the whole episode, search across every transcript, and follow the shows you care about.
Hi, hi, hi, and welcome.
We are talking about agents again today.
And so today, episode seven, we're gonna be talking about evaluations.
We talked last time about failure modes.
That conversation assumed that there was some way of measuring what a failure even was.
We talked about things like compound errors or coordination problems, and as you may be aware, or as you might have picked up from some of that conversation, a lot of agent failures, they don't look super obvious.
Uh, they look like the agent finishing confidently, but it actually didn't do what it said it did, or going around in circles, uh, not creating, like, an obvious crash, for instance.
And so when you don't have that glaring failure mode like, "Hey, this went badly," if failure is hard to see, then how do you measure it systematically? That's what we're gonna be talking about today.
How do you know whether an agent is working? You're listening to Linear Digressions.
[rock music] Now, if you're listening to this, I assume that you've used an AI chatbot before, at least.
