Static random-access memory

Yeah.

Well, it comes down to a lot of, uh, technical implementation details, like the fact that if you, if you look at the lowest level, the type of memory that is used on the silicon itself is called SRAM, static RAM.

And SRAM is made out of, you know, six transistors with, uh, a bit line and word line, some other control logic around it.

But that SRAM cell has not scaled in terms of the, the sizing of that with Moore's law over the past about 15 years.

So they have grown or shrunk, I should say, mu-much slower than just a group of transistors that you'll use for, for other purposes.

And I would say that there's just been a lot more architectural advancements that could happen on the compute side, while an S60 SRAM more or less has not changed in 30 or 40 years from a, like an architectural, you know, primitive perspective.

And so, you know, that's on the input side of raw technical capabilities on fabrication, et cetera, have not been able to improve.

But I would also say that there wasn't the right motivations for most of that, that decade.

The decode is really about pulling in the data and running it through the cache and getting the tokens out, right? And that is very memory bandwidth limited.

So how much bandwidth you can get from your memory, whether it be HBM or SRAM, really like dictates how fast that is, right? To do a full inference, you need to do both of those things.

You need to do one and you need to do the other.

There's been a lot of talk about disaggregating those.

The wipe has to complete faster than an attacker can reach the die.

That's why HSMs use battery-backed SRAM for key storage rather than flash.

SRAM is volatile.

Pull power, and the keys are gone.

The battery is there to keep the keys alive during normal operation, but the tamper circuit can cut that battery connection instantly, and the SRAM loses state in microseconds.

Flash would be a disaster for this.

Flash retains data without power.

You'd have to actively overwrite it, and that takes time, and the attacker is drilling towards you while you're trying to overwrite.

And the key is where this happens.

It happens directly within the high-speed GPU static RAM or SRAM.

Okay, let's clarify that hardware distinction for the listener.

Yeah.

The global VRAM, the twenty-four gigabytes you see on the box of your RTX forty-ninety, that is the DRAM.

It is huge, but relatively slow to read and write to.

But SRAM is the L1 and L2 cache physically etched right next to the GPU processing cores.

It's microscopic in capacity, maybe just a few megabytes, but it is blazingly fast.

So Tilas pre-fabricates about 98 to 100 base processing layers.

This includes all the standard transistors, the on-chip SRAM memory, the PCIe interfaces, all of that.

The basic plumbing?

Right.

Okay, that's the short-term memory the AI needs to remember The specific conversation you are currently having with it, correct?

Correct.

Because Talos eliminated off-chip external memory to save power, all the dynamic state for your conversation must fit inside the 128 to 256 megabytes of on-chip SRAM memory.

Megabytes? That is tiny.

Uh, we can get, um, uh, pro- essentially computational efficiency, signal efficiencies, right? Signal, signal time efficiencies, um, by stacking memory in, uh, um, with stacking memory with logic in certain ways, right? So, like, all of this is not really just about stacking transistors-

You can also logic fold the SRAM, the caching itself, right?

Exactly, right.

So, so one side of the cache, they occupy smaller space.

Yeah.

Um, and if you, if you ...

And this is especially relevant if you look at, uh, how much performance gain is being squeezed out of iterative node shrinks after three nanometers, right? Um, SRAM scaling has basically stopped, right? And in fact, you see this with, uh, you know, the announced performance levels of TSMC two nanometer, right? And the per- the expected performance gains from TSMC 1.5, uh, 1.4 nanometers and then, um ...

Was it one point ...

It's about 50 times larger than a standard GPU.

And what that allows us to do is have a huge amount of both SRAM, which is super fast memory, and compute on the same piece of silicon.

That's what ultimately allows us to deliver the 30 times faster speed that you shared about CS4.

What's unique about CS4 is this is a brand new system.

in a single chip anymore so how how has that played out for you in terms of like what sort of parallelism scale uh you know strategies are you having to develop does it look similar to what other players are having to do just with bigger components or are there different um you know kind of pareto frontiers that become possible because of the nature of the cerebrus system

So the way that we run inference on Cerebrus is completely different than with GPUs.

We actually store all of the weights of a model directly on the chip within the SRAM.

So what allows us to do that is because we have this wafer scale integration, we have a huge amount of SRAM, 44 gigs of SRAM per chip.

You know, you were pretty clear that the, the bottleneck is still in memory pricing and supply.

Um, NVHBM, what, what's the pitch there on what that allows an XPU-based architecture to do? A lot of the specialized inference platforms might be SRAM-based, for example.

And, and I just wanna throw this in, Jensen, as I may, 'cause no one's asked you for quite some time.

What's the sort of percentage or proportion of content of the server design NVIDIA's now pushing on its latest Vera Rubin generation platform, and how that, that kinda changes w- in this XPU case study where you have a hybrid AI factory essentially?

And I think the conclusion is that they're basically winning on both sides of the curve, so both fast tokens and cheap tokens, which is...

It's just so interesting because we've seen so many other, um, companies make claims about how they're gonna beat NVIDIA, and they just pick one of those, right? Groq or Cerebras or any of the other startups that are gonna focus on SRAM, like call it a D-matrix that's coming up with stuff, or SambaNova, where we've even seen some results on benchmarks, you know, on the InferentX benchmark or something similar to it.

They're saying they're going for fast tokens, just decode speed, low batch size.

They don't worry about throughput.

Um, maybe the other thing is that it has an L1 cache, which is quite funny.

Like all of these accelerators do not have L1 caches now.

They rely so much on L2, um, namely SRAM.

We hear SRAM all the time.