EP 38: The Local AI Stack Nobody Talks About (But Should)

So let's get straight to the big question that frames everything else.

The cloud versus on-prem debate is heating up.

What was the moment you real- realized running AI locally was not just viable, but actually preferable for certain use cases?

Yeah.

F- for me, there wasn't, like, one specific moment that made me want to pursue more of the on-prem, more of the local models as opposed to the cloud providers.

It really started as just sort of a general distrust with the, the cloud providers.

It was not clear to me how my data was going to be used.

I was uncomfortable with taking my data sets and actually moving it to the cloud.

And, you know, there's plenty of evidence that there's other people that are struggling with this.

I saw a survey not long ago, ninety percent of its respondents were worried about the AI providers using their data without their consent, and that was sort of like the bubbling feeling inside of me is like, I don't know how this is gonna be used.

I'm not sure that it's smart for me to use these cloud providers in this way.

And then you can consider some of the news headlines that are out there.

Just last...

Oh, no, it was this week, maybe even yesterday, actually.

It was yesterday when Claude went down for hours, and that's not really a trust factor.

That's just, you know, an unfortunate circumstance from Claude.

But it was just a few weeks ago that Claude had a major leak and some of their software got published to the internet completely by accident, and they had to go into damage control to do that.

And, and, and I've mentioned two things about, about anth-- uh, about Claude and Anthropic.

You know, in, in my personal opinion, they may be the most trustworthy of the, of the model providers.

Like, I have a...

You know, I, I really like Dario Amodei.

I think that he is a very sharp guy, and of all the players in this game, I trust him the most.

He is, you know, for any of your listeners that follow this to- sort of thing, know that he's within a hot seat with the Department of Defense.

You know, he was the one that set some hard lines on where, why, how he wanted his technology to be used and how he specifically did not want it to be used.

Um, and they felt like very reasonable hard lines, you know, like not mass surveillance on Americans, you know, as an example.

So let's get straight to the big question that frames everything else.

The cloud versus on-prem debate is heating up.

What was the moment you real- realized running AI locally was not just viable, but actually preferable for certain use cases?

Yeah.

F- for me, there wasn't, like, one specific moment that made me want to pursue more of the on-prem, more of the local models as opposed to the cloud providers.

It really started as just sort of a general distrust with the, the cloud providers.

It was not clear to me how my data was going to be used.

I was uncomfortable with taking my data sets and actually moving it to the cloud.

And, you know, there's plenty of evidence that there's other people that are struggling with this.

I saw a survey not long ago, ninety percent of its respondents were worried about the AI providers using their data without their consent, and that was sort of like the bubbling feeling inside of me is like, I don't know how this is gonna be used.

I'm not sure that it's smart for me to use these cloud providers in this way.

And then you can consider some of the news headlines that are out there.

Just last...

Oh, no, it was this week, maybe even yesterday, actually.

It was yesterday when Claude went down for hours, and that's not really a trust factor.

That's just, you know, an unfortunate circumstance from Claude.

But it was just a few weeks ago that Claude had a major leak and some of their software got published to the internet completely by accident, and they had to go into damage control to do that.

And, and, and I've mentioned two things about, about anth-- uh, about Claude and Anthropic.

You know, in, in my personal opinion, they may be the most trustworthy of the, of the model providers.

Like, I have a...

You know, I, I really like Dario Amodei.

I think that he is a very sharp guy, and of all the players in this game, I trust him the most.

He is, you know, for any of your listeners that follow this to- sort of thing, know that he's within a hot seat with the Department of Defense.

You know, he was the one that set some hard lines on where, why, how he wanted his technology to be used and how he specifically did not want it to be used.

Um, and they felt like very reasonable hard lines, you know, like not mass surveillance on Americans, you know, as an example.