Site reliability engineering

typically on conversations also depend on the size of your team but for me it was once a month which wasn't too bad that basically means we have a pager and if something goes wrong if there's a very serious alert that happens then they will just ping you off hours and bring you into a bridge line where you basically have to figure out what's going on and fight the incident, basically.

But a lot of this stuff is usually working with the IT team as well as the SRE team, which is the Site Reliability Engineering team.

I find that they are also very involved in cybersecurity incidents alongside all the SOC analysts and the incident responders.

So that would probably be a typical week.

in this situation, I would say like the situation two years ago, for a reason, it was like some kind of like the best optimum, like the best optimum.

And also, because if you say this, you could say, yeah, the CEO could be the, with cloud and agents could be like the data analyst, the data engineer, the SRE, the software engineer, and so on.

And so everyone is just CEO at the top level and do stuff.

No, I don't think so.

The renewable energy industry needs folks who can build lifecycle communication systems that actually work.

We need people that have managed distributed device fleets at scale, designed APIs and telemetry systems, ran incident response and SRE operations, created dashboards that translate data into action.

The energy industry, it desperately needs these exact skills and they have no idea how to find people who have them.

So what happens? You're sitting there thinking, wow, I guess I'll just apply to some other tech company.

That's where someone with your background becomes essential.

Building the orchestration layer that makes hardware stretch further.

Here's a friendly call-out to ex-DevOps, SRE, data engineers, product people.

If you're listening, this is where you come in.

Like your platform engineer, that is a DevOps person.

And it was SRE for a while too, right? So like this is just job title shuffling.

That's all that's happening with DBAs.

They exist somewhere in a platform engineering team under a different job title.

Correct, Joe.

So typically the dashboard, like let's come to the SRE, right? Typically they are the one ensuring every application is healthy.

They typically look at the response time, error rate, availability, any restarts, right? Whereas in this case, what is happening, I mean, the reasoning, in this case we cannot i mean in addition to the response time now it's a challenge right now we have to measure the reasoning time right because your reasoning time can differ based on the user query and the context how the context i mean how the contact context is growing at some point what happened your system prompt might bloat so when the system prompt might bloat that time it and it will end up loading the conversation again so so i mean so when it so in this case for example there i mean agent answering even though it made some of the check box like a response time looks good your throughput looks good no errors but however what if it takes five times of tokens what if it's take five times of token compared to my baseline so that token ended up showing in the cost budget not into the i mean not into the not into the regular performance matrix So that is also like another key matrix, like how the cost is inflated.

And similarly, and also another primary metric, we wanted to make sure one agent, I mean, basically this one agent is not going to impact the entire platform or infrastructure because one agent end up taking all the token limits on the one AI provider.

That is very critical as well to simulate the production type behavior.

All right.

So do you need then to have SRE set up first to do these open telemetry type traces so you really understand performance and production to get good results? So you're able to trace, you know, what good looks like in production? And what about that? It's like you're kind of running blind?

Yeah, definitely.