So what's one lesson across all of those journeys or all of those chapters that most directly shaped how you approach brain trust and like, why did you begin brain trust really?

Yeah, you know, interestingly, the first thing I worked on was a relational database company called MemSQL.

And one of the issues that we had is we would try to test a bunch of SQL queries and find like correctness issues and performance issues.

Back then, performance meant speed, by the way.

Nowadays, when people talk about performance, they mean accuracy.

But we would try to find issues.

And then, of course, customers would use our product and they would find new issues that we didn't expect.

How do you think eval products would fit into it? Maybe from a policy enforcement lens more than an instrumentation lens.

Tell me, tell me about your approach to, to software engineering in the age of AI.

You know, I spend a lot of time working on software for doing evals and observability, and that's kind of shaped my own perspective about software engineering.

Like, now, now that models are so good at actually writing code, one of the best things that we can do is create really hard evals.

And not, I'm not talking about, like, AI evals.

I mean things like why is this query so slow? And if you create the right tests and success criteria for a model, then it can be really creative, and it can work on the stuff in the background and actually try to improve a bunch of things.

So, um, one of the things that I spend a lot of time on right now is making the queries that people run in our product faster.

And people can just write arbitrary queries.

Any tips or tricks for how you're managing your, your fleet of agents that you think are unique?

So like, what is, what does best-in-class look like if you're a software company that's integrating LLMs? Like, what does best-in-class testing look like? Who, who, who's, like, doing this at the highest level, uh, that you can see?

Yeah, I mean, I, honestly, I think Ramp is a great example of a company that does this very well.

They use a bunch of different models for different use cases.

And I think there's two things you shouldn't do.

One is just use, uh, the same model and be very afraid to change it.

There are some companies we meet that are still using GPT-3.5 on Azure.

Um-

Mm-hmm.