Agentic Performance Engineering ft. Perfloop

Okay.

I think that makes sense and I also experience that with the work I do.

If there's a separate agent with a fresh context window, it's not biased, it's not polluted from whatever the previous agents have done and it's always much cleaner in terms of output as well.

So I think it makes sense.

In terms of like different AI models, what has been your experience? Like, do you use like specific models from, you know, let's say GPD models or Claude or, you know, some other Grok models? for certain things? And what has been your experience? Which one is, you know, which one is the most balanced one which works best for these kind of work?

So we take a little bit the approach that, do you know AMP, the coding agent? Yeah.

So like, AMP's philosophy has been that, you know, different models are good at different things.

Yeah.

Right? And that, you know, they offer basically a set of agents that you can invoke for different jobs, like, you know, code review or advice or finding an answer in the code base or searching something on the internet, right? And so...

We have a little bit of that too, essentially.

And that's why the thing that matters for us is that we have the way to assess any new model that comes out for a specific job quickly right and so like i've used as you have been seeing you know it's it's dizzying like the the rate of releases from the labs right um and every time there is something that looks interesting for for perfloop to uh to test like i just put a coding agent running our evals with this new model, right? And then doing a side-by-side comparison on all of the metrics that we care about for that specific job.

So it's very fluid, right? Like the models that we use for different roles in the system.

You know, we have a few different jobs.

You know, we have like the research agent.

We have the triage agent.

We have the proposer on the casework.

We have the model building that maps your code and generates hypotheses from those plot paths and creates this architecture that is structured.

We have the verifier, the verification layer.

Yeah, it's extremely fluid, you know, like which models we're using.

And it's a constant cat and mouse game on trying to get the best cost performance point across the whole system, right? And there's a lot of variables there, right? Like, for instance, if on the model building side, you're generating a lot of false positive hypotheses, right? right, then that's going to be the most expensive thing you can do, right? Because generate this hypothesis.

If you have autonomy on, then the triage agent is going to pick them up, going to tell the case to be worked.

And then you're going to spend a lot of time and tokens disproving a false positive.

And then it turns out, OK, this is refuted.

So you just spent a lot of time to get no result.

It's a negative, right? You need to put a lot of emphasis on trying to reduce false positives early in the pipeline.

Okay.

I think that makes sense and I also experience that with the work I do.

If there's a separate agent with a fresh context window, it's not biased, it's not polluted from whatever the previous agents have done and it's always much cleaner in terms of output as well.

So I think it makes sense.

In terms of like different AI models, what has been your experience? Like, do you use like specific models from, you know, let's say GPD models or Claude or, you know, some other Grok models? for certain things? And what has been your experience? Which one is, you know, which one is the most balanced one which works best for these kind of work?

So we take a little bit the approach that, do you know AMP, the coding agent? Yeah.

So like, AMP's philosophy has been that, you know, different models are good at different things.

Yeah.

Right? And that, you know, they offer basically a set of agents that you can invoke for different jobs, like, you know, code review or advice or finding an answer in the code base or searching something on the internet, right? And so...

We have a little bit of that too, essentially.

And that's why the thing that matters for us is that we have the way to assess any new model that comes out for a specific job quickly right and so like i've used as you have been seeing you know it's it's dizzying like the the rate of releases from the labs right um and every time there is something that looks interesting for for perfloop to uh to test like i just put a coding agent running our evals with this new model, right? And then doing a side-by-side comparison on all of the metrics that we care about for that specific job.

So it's very fluid, right? Like the models that we use for different roles in the system.

You know, we have a few different jobs.

You know, we have like the research agent.

We have the triage agent.

We have the proposer on the casework.

We have the model building that maps your code and generates hypotheses from those plot paths and creates this architecture that is structured.

We have the verifier, the verification layer.

Yeah, it's extremely fluid, you know, like which models we're using.

And it's a constant cat and mouse game on trying to get the best cost performance point across the whole system, right? And there's a lot of variables there, right? Like, for instance, if on the model building side, you're generating a lot of false positive hypotheses, right? right, then that's going to be the most expensive thing you can do, right? Because generate this hypothesis.

If you have autonomy on, then the triage agent is going to pick them up, going to tell the case to be worked.

And then you're going to spend a lot of time and tokens disproving a false positive.

And then it turns out, OK, this is refuted.

So you just spent a lot of time to get no result.

It's a negative, right? You need to put a lot of emphasis on trying to reduce false positives early in the pipeline.