#494 – Jensen Huang: NVIDIA – The $4 Trillion Company & the AI Revolution

[gentle music] You've propelled Nvidia into a, uh, new era in AI, moving beyond its focus on chip-scale design to now rack scale design.

And I think it's fair to say that, uh, winning for Nvidia for a long time used to be about building the best GPU possible, and you still do.

But now you've expanded that to extreme co-design of GPU, CPU, memory, networking, storage, power, cooling, software, the rack itself, the pod that you've announced, and even the data center.

So let's talk about extreme co-design.

What, uh, is the hardest part of, uh, co-designing a system with that many complex components and design variables?

Yeah, thanks for that question.

So first of all, the reason why extreme co-design is necessary is because the problem no longer fits inside one computer to be accelerated by one GPU.

The problem that you're trying to solve is you would like to go faster than the number of computers that you add.

So you added, uh, you know, ten thousand computers, but you would like it to go a million times faster.

Then all of a sudden, you have to take the algorithm, you have to break up the algorithm, you have to refactor it, you have to shard the pipeline, you have to shard the data, you have to shard the model.

Now, all of a sudden, when you distribute the problem this way, not just scaling up the problem, but you're distributing the problem, then everything gets in the way.

This is the Amdahl's law problem, where, uh, the amount of speed up you have for something depends on how much of the total workload it is.

And so if computation represents fifty percent of the problem, and I sped up computation infinitely, like a million times, you know, I only sped up the total workload by a factor of two.

Now, all of a sudden, not only do you have to distribute the com-computation, you have to, you know, shard the pipeline somehow, uh, you also have to solve the networking problem because you've got all of these computers are all connected together.

And so distributed computing at the scale that we do, the CPU is a problem, the GPU is a problem, the networking is a problem, the switching is a problem, and distributing the workload across all these computers are a problem.

It's just a massively complex computer science problem.

And so we just gotta bring every technology to bear.

Otherwise, we scale up linearly, or we scale up based on, uh, the capabilities of Moore's Law, which has largely slowed because Dennard scaling has slowed.

I'm sure there's trade-offs there.

Plus, you have a complete disparate disciplines here.

I'm sure you have specialists in each one of these, high bandwidth memory, the, the network and the NVLink, the NICs, the, the optics and the copper that you're doing, the power delivery, the cooling, all of that.

I mean, there's like world experts in each of those.

How do you get them in a room together to figure out-

That's why my staff is so large.

[laughs] What's the pro-