Pareto distribution

And when you batch fewer requests through the system, as you might imagine, the cost goes up.

And you'll see this nice Pareto curve that's on Semi-analysis is inference X, which kind of explains this in a bit more detail where I can batch more and I can reduce, I can put more users through the system, but the trade-off is every user is going to have a slower experience.

And so...

The question is really where on this Pareto curve do you as an enterprise want to balance? And you have full flexibility and configurability of the system.

And then the challenge is every chip has a different Pareto curve where on GPUs, They struggle with the faster performance and are not able to batch efficiently.

On some of the SRAM solutions, you can go really fast, but you're not able to really get a high throughput per chip.

And the real question is, how do you balance that speed versus cost in a way that makes sense, which is really what we're trying to do with our RDUs, especially on these really, really big models.

It's the organizing principle.

It's a culmination of, of the other three, and I'm gonna let you try and explain, um, the Pareto tail because it's such a powerful thing.

It, it is a powerful thing, and I, I...

It's obviously whoever I speak to says, "This last principle is very tough." [laughs] We had Richard Quest here, as you know.

Said, "How was the book, Richard?" He said, "Demanding." [laughs]

[laughs]

And I realized it was the Pareto tail that I think he was talking about.

Just hear me out on this.