This is really key to understanding how digital products can better serve users.

I'm very excited to be here, um, and talk to you about, uh, how we've been looking at AI agents and how to, how to measure, evaluate, and improve them.

Um, I...

And, and the key challenge we've, uh, we've seen working with our companies is that, um, uh, a-a-agent builder teams, um, spend a lot of time building evals and testing, testing agents, but still, when they go live, there are a lot of cases it doesn't work for the users.

So question, the challenge of the industry is: how do we actually capture the right signals? How do we improve, uh, AI agents? Now, before I go into that, I'll share a little bit of background about myself and, and the company, uh, uh, where I'm at, uh, Conviva.

Um, so we've been, uh, Conviva's been around twenty years.

I'm one of the co-founders.

This is where innovation in user experience measurement can really make a difference, especially for operations teams.

The question is, "If a fraud dispute can look perfect in the agent transcript but still fail the user journey, what is the first signal you'd add to catch that gap earlier?"

Yeah, great, great question.

Um, so what we believe is the best way to catch, um, these kinds of issues, and it can be fraud signal, it could be, could be all sorts of different error scenarios where the agent didn't quite do what it was supposed to do, is, is to, one, capture as many of the user signals as we can ca- get from the entire digital surface.

Um, so and, and many times these are signals that occur after the conversation with the agent.

So this could be in the website, um, that the user goes in, the user is already in, but then actions they may take in the website after the, a conversation.

So in that case, they would go and check if their particular dispute was filed, could be any type of customer support ticket.

Um, it could also be if it was a purchase or a sign-up, um, for example, then to see if they actually went and did that in the website.

And Moya's asking, um, "When you say users are the best evaluators of agents, what user behavior signal has been most valuable that trace-by-trace reviews usually miss?"

And Moya is asking, when you say users are the best evaluators of agents, what user behavior signal has been most valuable that trace-by-trace reviews usually miss?

Yeah, quick point.

So I think there are two classes of signals.

One is signals within the conversation that you would see in the trace, but a lot of the current tools are more focused on the agent's response, evaluating and judging the agent's response to see if that was a good response to the user prompt.

We believe that's useful, but it's not complete.

The signals, I'll give you a few examples that we see even just from the conversation turns.

Things like clarifications, user is clarifying back to the agent, no, this is not exactly what I was looking for, I want something different, or repeating that same ask multiple times.

This is from Mark, and Mark is asking, for teams already spending heavily on evals, what's the smallest change you'd make to shift from conversation-only scoring to full journey measurement?

The question is, if a fraud dispute can look perfect in the agent transcript but still fail the user journey, what is the first signal you'd add to catch that gap earlier?

Yeah, great, great question.

Um, so what we believe is the best way to catch, um, these kinds of issues, and it can be fraud signal, it could be, could be all sorts of different er- scenarios where the agent didn't quite do what it was supposed to do, is, is to, one, capture as many of the user signals as we can ca- get from the entire digital surface Um, so and, and many times these are signals that occur after the conversation with the agent.

So this could be in the website, um, that the user goes in, the user is already in, but then actions they may take in the website after the, a conversation.

So in that case, they would go and check if their particular dispute was filed.

Could be any type of customer support ticket.

Um, it could also be if it was a purchase or a sign-up, um, for example, then to see if they actually went and did that in the website.

And Moya is asking, um, "When you say users are the best evaluators of agents, what user behavior signal has been most valuable that trace-by-trace reviews usually miss?"