Static random-access memory

Level that's closest to the processor.

This is often called SRAM.

And the memory bandwidth, the amount of memory they can move back and forth into the processor is huge.

It can be like 20 terabytes per second.

HBM is going to be other elements on that board.

And as a result, because it's a little bit farther away, it's going to be a little bit slower to read in and out.

Where you had 20 terabytes per second for SRAM, you might have something like three or four terabytes per second for HBM.

These are numbers for the H100, by the way, the NVIDIA data center board.

L'introduzione della tecnologia 3D V-Cache da parte di AMD ha segnato un punto di svolta nelle prestazioni di calcolo per carichi di lavoro intensivi e nel gaming ad alto frame rate, triplicando la memoria cache L3 tramite stacking verticale.

Tuttavia, nelle architetture Ryzen 5000X3D e 7000X3D basate su Zen 3 e Zen 4, il die di memoria SRAM supplementare da 64 MB veniva fisicamente impilato al di sopra del core complex di CCD.

Questa disposizione stratificata creava una severa barriera termica: il silicio della cache, posizionato tra i core computazionali caldi e il diffusore di calore integrato in rame IHS, fungeva da isolante termico.

Di conseguenza, AMD fu costretta a limitare rigidamente le tensioni operative di alimentazione e a bloccare le frequenze di clock massime, impedendo l'overclocking manuale e penalizzando le prestazioni pure nei carichi single thread di tipo produttivo.

L'inversione strutturale dell'architettura Zen 5 a 64 MB.

Con il debutto del processore octa-core Ryzen 7 9800X3D fondato sui transistor a 4 nm di TSMC per i core Zen 5, i progettisti hanno completamente rivoluzionato la topologia del die attraverso il packaging avanzato 3D V-Cache di seconda generazione.

Nel nuovo schema fisico, il die contenente i 64 MB di memoria SRAM viene collocato al di sotto del CCD computazionale, a diretto contatto con il substrato del package organico.

I core di calcolo Zen 5 poggiano ora superiormente, direttamente interfacciati con la pasta saldante e l'heat spreader superficiale.

You want to have the sort of hierarchy of memory and use it effectively.

I think Grok, which was recently acquired by NVIDIA, was a design where you just had the giant amounts of SRAM.

And for certain workloads like inference, this is very helpful.

But still, for most accelerators, you're going to have this hierarchy that you're going to really have to respect in order to make anything run fast.

Yeah, I mean

physical distance is a big part of it.

I mean they are, if I believe, if I remember right, they're both SRAM.

Like that's one of the reasons why global memory is much slower.

The industry realized how big these models are going to get and how many chips are going to be required.

So a lot of them did start with the SRAM based approach, which actually the matrix also has in their technology and we'll get to that.

But clearly you went straight for inference.

So that is a unique problem that you identified early on for sure.

So after many years of transistor scaling, we knew that the compute is very small, but the wires had not scaled at the same rate.

So it was data movement that was a problem.

So our initial idea was, can we take an SRAM array and add compute into the SRAM array itself? So unlike a traditional SRAM where you only activate one word line at a time, we ended up designing an array where we activated all of the word lines.

And so that was a pretty good operation for a matrix calculation, matrix vector-specific calculation.

Right,

and starting with an SRAM chip is very telling.

Why is that?

SRAM, or Static Random Access Memory, is kind of the Camari in the coal mine for a new fab.

It is the most densely packed foundational structure you can print.

So if you mess that up...

If your fab can't produce a working SRAM chip with high yields, it has absolutely zero chance of printing a complex AI logic processor.

Wow, okay.

For the various circuit types, the delay didn't really vary that much.

FPGAs were slower, but more or less uniformly slower, except for SRAMs.

SRAMs were particularly a little bit faster.

But the big difference was in area.

So things like random logic, multiplexers were particularly inefficient.

And additionally, even though that this, in our specific analysis, there is this memory constraint, and that's also largely a function of that we're looking at a certain type of chip, you know, hallway type accelerators.

Chinese chip companies have been experimenting with different sorts of memory arrangements, such as using DDR5 or DDR memory, and also SRAM.

instead.

And so if there are significant constraints on one type of memory chip, that does not constrain other types of chip architectures emerging from China that tries to mitigate this.

It seems like 2026 is the year of further memory innovation.

NVIDIA's announcement of the Grok LPU was another way of using SRAM rather than HPN for most part, pulling in more of that.

Cerebras went the other way and actually is doing wafer-scale computing with a lot more memory on die, of course.

Each has its challenges as to what size of models optimize for how it works.

I think both NVIDIA's Quark LPU and Cerebus wafer-scale processing, they both point to the need to have memory as fast and as close to the computer as possible.

So that's certainly one way of implementing the solution.

The problem with both is that the SRAM embedded in the wafer doesn't scale much anymore.

It's kind of roughly stuck at about 40 megabits per millimeter square.

It has level shifting in it, so you can use it with 3-volt or 5-volt microcontrollers.

It has the boost converter stuff in it, and it also has SRAM.

And we put 256 kilobits of SRAM on there.

So you can control up to 4.2 inch displays.

If you do the math, these need basically 256 kilobits.

This is shiny.

And so you can drive up to 4.2-inch E-Ink displays.

You don't have to use the SRAM, but what's nice is that even chips with very low memory can drive very large displays.

So can we put some numbers on that? How many features would you need on a chip to create various types of conventional memory cell as opposed to a Memrister-based memory cell?

One of the comparisons that I can make off the top of my head is that an SRAM cell, for example, uses six transistors, so that's six devices for a single binary piece of memory.

Whereas a memristor is a single element, a single device that holds actually a continuous memory state.

So not only is the density per cell increased in a memristor, but also the number of devices you need in order to have an equivalent unit of memory is much less.