Observability

That is where incidents get ugly.

Observability has dependencies too.

Logs need storage.

Metrics need CPU and memory.

This is incredibly useful if you need to display a long-term trend alongside real-time metrics with low sampling rates.

Meanwhile, CloudTrace and the Observability API now fully support VPC service controls in general availability.

Finally, let's look at cloud storage and heavy-duty AI computing.

First, Cloud Storage Intelligence Advisor is officially generally available.

I think we might make it.

Today's title and topic, Visibility, Transparency, Observability.

No commas necessary.

If you're not familiar with the Let's Innovate for Vegas audio program, or perhaps the Innovate for Vegas Foundation, I'll give you the quick rundown.

So transparency can drop off for some things.

You know what I'm talking about, those cameras, among other things.

Observability, that's a little bit more down deep into the inner workings.

That's a rarity.

This lets anyone in your organization build multi-step automated workflows to streamline tasks, which can run on schedule, manually on demand, or directly via app mention in chat conversations.

To help you monitor all of these agents, Google has added two new telemetry views to the Observability tab.

Latency and error rate.

Latency shows you P50 and P95 metrics for time to first token, time to first answer, and time to last token.

That brings us to the layer, which is the eighth layer, and that is verification and observability.

I am treating this layer as one because they are same discipline pointed to two questions.

Observability asks, can we reconstruct what happened? And verification asks, was it actually right? And you can't answer the second one without the first.

that sequence is a trajectory the full path that produced the outcome not just the final answer when a double refund lands on your desk tomorrow the agent made a mistake is useless did it pull the slate data did it call the refund twice or did the tool fire twice on one call was approval granted or quietly bypassed which sub-agent made the call now the verification half our customer our agent reports refund process successfully the customer has been credited very confident well formatted possible and potentially completely false because the question isn't did the agent produce a good sounding answer it is is the money actually back in that customer's account that's the difference between checking the output and checking the outcome the real state of the world and that is a query against your payment system not a judgment about a sentence And even that is not enough because an agent can reach the right outcome down an unacceptable path, issuing the correct refund, but pulling out four other customers records to get there.

So you verify on your several axes.

41.

Observability vs.

Monitoring.

What's the difference? Monitoring has been a basic system to track the health of servers for years.

Applications in 2022.

74.

Production Observability for Multi-Agent AI with Chaos plus Otel plus SIGNOZ.

Learn how to monitor multi-agent AI systems with OpenTelemetry and SIGNOS using Chaos on Kubernetes, tracing LLM calls, tool usage, and agent delegation.

Key findings include insufficient data inventory, a lack of unified controls, and challenges in monitoring data flows across hybrid environments.

Security leaders should accelerate efforts to map data assets, unified controls across environments, and leverage AI observability tools to gain visibility into how data is being used and where it's flowing.

Observability is becoming a core component of AI risk management, especially in regulated sectors.

Indian banks, for example, are investing in observability solutions to manage the risks associated with AI in production environments.

This reflects a broader trend toward operationalizing AI while maintaining visibility into model behavior, data usage, and compliance.

Theksha Vichinder III said, Data protection and observability are foundational to managing AI risk.

As employee behavior and identity systems become primary attack vectors, organizations must invest in robust data classification, access controls, and monitoring capabilities.

Observability tools can provide the visibility needed to detect and respond to anomalous activity, support compliance, and assure stakeholders that AI systems are operating securely.

Yeah, yeah.

Observability.

Uh, that was a struggle.

[chuckles] Uh, we do a lot in Harper with, like, recording analytics of what's going on, uh, you know, how many requests we're getting, what is the load, uh-