117. Multi-Terabyte GPUs the Only Answer for AI Scaling

One of the ongoing themes seems to be that AI models are getting larger.

These large language models are just getting larger and larger.

We thought that billion parameter models were going to be the biggest and baddest, but now we're starting to see multi-trillion parameter models.

One of the things you may not recognize is that those parameters have to be stored in memory in a GPU somewhere.

And so the larger the model, the more memory that GPU needs.

You may also have noticed that memory prices have gone up a lot in the last year or so as people have been contracting to build huge AI data centers and taking up all of the available capacity for that memory.

leads us to wonder how long can this massive expansion and growth go? Is this the right way for us to be continuing to pursue AI? And I think as we start getting to the point where real business is being done with AI, companies are using AI to make decisions and to build AI processes into their applications, The scalability here starts to become a real challenge and the cost of producing an answer becomes a vital part of the discussion.

And is it a good idea to keep building bigger and bigger and bigger GPUs as we go along?

I tend to think this is a time where we need to ask ourselves about what it is we're trying to solve.

And that's the generic product management type answer.

But a lot of times, I think we are thinking that we need to scale up as opposed to scale up.

And I think there's an opportunity to essentially work smarter versus harder in the sense that Harder meaning spending more capital and building for the skyscraper deployment.

In reality, it kind of brings into question how much AI is done at the edge versus centralized.

And I think for the training side of things, it may make sense, but I also think there's an opportunity for us to pull back and ask the question.

The frontier model vendors are really pushing us forward.

And that's good and bad because they're not necessarily thinking about optimizing for every enterprise, if you will.

They're optimizing for biggest, baddest, right? Like we are invincible.

And at some point, there's an inflection point of, is this cost prohibitive? And what are the real ROIs on this investment?

I think part of it is that going for the larger model is just bragging rights at this point without actually coming up with a necessary use for it.

And while certainly having the largest model for bragging rights is a goal, if that's what you're trying to achieve, it doesn't necessarily accomplish the goals of the everyday person trying to make use of the AI models.

Yeah, I have a real-world example of that, Andy.

I was watching myself and the coding I've been doing and the work I've been doing across, in this case, like Opus 4.5, 4.6, 4.7, 4.8, and I tend to gravitate toward the newer one partly because it is better and because it's part of the plan.

What I've been trying to do lately is step back and look at what models are acceptable for subsets of the task.

How do we divide up tasks into subsets? What's good enough from six, seven months ago, which in AI time is like a decade ago.

And what do we actually need the front end for, the cutting edge?

One of the ongoing themes seems to be that AI models are getting larger.

These large language models are just getting larger and larger.

We thought that billion parameter models were going to be the biggest and baddest, but now we're starting to see multi-trillion parameter models.

One of the things you may not recognize is that those parameters have to be stored in memory in a GPU somewhere.

And so the larger the model, the more memory that GPU needs.

You may also have noticed that memory prices have gone up a lot in the last year or so as people have been contracting to build huge AI data centers and taking up all of the available capacity for that memory.

leads us to wonder how long can this massive expansion and growth go? Is this the right way for us to be continuing to pursue AI? And I think as we start getting to the point where real business is being done with AI, companies are using AI to make decisions and to build AI processes into their applications, The scalability here starts to become a real challenge and the cost of producing an answer becomes a vital part of the discussion.

And is it a good idea to keep building bigger and bigger and bigger GPUs as we go along?

I tend to think this is a time where we need to ask ourselves about what it is we're trying to solve.

And that's the generic product management type answer.

But a lot of times, I think we are thinking that we need to scale up as opposed to scale up.

And I think there's an opportunity to essentially work smarter versus harder in the sense that Harder meaning spending more capital and building for the skyscraper deployment.

In reality, it kind of brings into question how much AI is done at the edge versus centralized.

And I think for the training side of things, it may make sense, but I also think there's an opportunity for us to pull back and ask the question.

The frontier model vendors are really pushing us forward.

And that's good and bad because they're not necessarily thinking about optimizing for every enterprise, if you will.

They're optimizing for biggest, baddest, right? Like we are invincible.

And at some point, there's an inflection point of, is this cost prohibitive? And what are the real ROIs on this investment?

I think part of it is that going for the larger model is just bragging rights at this point without actually coming up with a necessary use for it.

And while certainly having the largest model for bragging rights is a goal, if that's what you're trying to achieve, it doesn't necessarily accomplish the goals of the everyday person trying to make use of the AI models.

Yeah, I have a real-world example of that, Andy.

I was watching myself and the coding I've been doing and the work I've been doing across, in this case, like Opus 4.5, 4.6, 4.7, 4.8, and I tend to gravitate toward the newer one partly because it is better and because it's part of the plan.

What I've been trying to do lately is step back and look at what models are acceptable for subsets of the task.

How do we divide up tasks into subsets? What's good enough from six, seven months ago, which in AI time is like a decade ago.

And what do we actually need the front end for, the cutting edge?