Observability

Do you find that there's a parallel between what agencies maybe should be doing now and maybe things that have been done in the past around other areas?

Well, a lot of it's foundational in my opinion, right? Observability has always been a thing now for a long period of time, especially through sort of the cloud boom.

And this extends back even into virtualization, right? When you get into these technologies that allow very rapid and very easy to do sprawl, if you will, of resources.

It was always important to know what you have and where it's at and therefore where your data resides.

Yeah, we are probably quite fortunate, I think, in that Snowflake said, well, look, you guys are doing well as a separate company.

Observability is a different business to sort of traditional AI data cloud pipelines that are warehousing data engineering, all that kind of stuff.

So why don't you keep running the thing over here? You keep going to the software engineering leaders, CTOs and so on, selling observability, and then let's figure out where the leverage points are.

I mean, Snowflake does have great relationships and so many big accounts.

we could have a podcast episode on versioning and probably go and lament about that one issue for 60 minutes and operationally, you know, distributed computing is hard.

Uh, Observability across microservices is hard.

And then with all of this, the complexities of large enterprises get magnified,

right?

And what's interesting is I was around for the first version of AIOps about nine or 10 years ago.

Observability, APM, we called it then, Application Performance Monitoring Tools, based on machine learning algorithms rather than neural networks or generative AI.

you know, we're trying to apply analytics based on automation.

And what a lot of people found was kind of the same thing we're seeing now, but more broadly was that, you know, garbage in garbage out with these things.

Policy determines what it's allowed to do.

Observability determines whether we can understand what happened.

Ownership determines who is responsible for operating it.

Lifecycle management determines what happens when the agent changes, becomes inactive, or should no longer have access.

This creates a more practical definition of autonomy.

Autonomy is not simply the AI can do everything, it is the AI can operate independently within a clearly defined authority boundary.

Observability has to capture decisions, not just logs.

This is another area where my view of AI observability has changed.

Agent relationships themselves are becoming operational data.

Observability has to move beyond logs.

This is where my own experience with production systems strongly influences how we think about agents.

A normal application log might tell you that's useful.

For example, some policies can be distributed to local enforcement points.

Identity can be validated close to the workload.

Observability data can be collected locally and aggregated centrally.

High-risk authorization can call a central policy service.

And I worry-- I, I saw an article just recently about a rash of VP engineering and CTOs just quitting because they're so burned out 'cause they were being asked to do impossible things, and they were getting headcount taken away, and everybody's just like, "Well, why can't, why can't you do it? This other person on, on X did it." Like, "Look, why aren't you 10X-ing yet?"

[gentle music] All right, so today we have Charity Majors, co-founder and CTO at H- Honeycomb.io. Tagline is Observability Built for the AI Era.

She is also, uh, one of the authors of the O'Reilly book, "Observability Engineering," which is now out with a new edition.

So Charity, welcome to the podcast.

Thank you.