Ep. #91, Every Failure Becomes an Eval with Janaki Vivrekar

[upbeat music] You don't always have customers writing into you like, "Hey, I would really like this particular tool to be more accurate." Like, they're not gonna give you that feedback up front.

You kind of need to be able to figure it out from the overall response quality.

That's really similar to, like, how pretty much all product development works, but it's a little bit amplified in this world of AI, where there's a lot less transparency into how these agentic systems work.

[upbeat music] I'm Ken Rimpel.

And I'm Jessica Kerr.

You're listening to Observability Cast, a monthly series where we talk about how we can make production systems more observable, more reliable, and easier to maintain.