llama.cpp

And they have a Xeon, and they have two GPUs, and they're, like, small GPUs, but it's big enough that you can, you can put the parts of the model that need to go fast on there.

So something like Ollama or Llama.cpp, they'll do that automatically.

They'll take the encoder, the decoder layers, the KV cache, and stick that on the GPUs, run everything else out of the system RAM.

And suddenly you can run these huge models, or even from the hard drive, right? You don't even need the RAM to hold the model.

Third is the intelligence engine, which can be any local model like Llama Three or Gemma.

Fourth is the inference engine itself, like Llama CPP or VLM.

And the fifth primitive covers tools and memory using a standard protocol, along with learning systems like direct preference optimization or LoRA to make the agent persistently improve over time.

The absolute coolest part of the presentation by John was the optimization loop To bridge the gap between local and cloud models, they actually use a cloud model to automatically optimize the local stack configuration.

But I don't think it was enough because it was doing the confusing things that the other models had done earlier, which is telling you that it couldn't reach the API, giving you some weird nil error and stopping.

So Rocky tells me about llama.cpp. And lama.cpp is an alternative to olama that it allows you to host models.

But the cool thing about it is that it can take the compute heavy stuff and run that in VRAM, and then it can use your system RAM for the rest of it.

Now, I don't know if it swaps things in and out, system RAM, VRAM, or it just runs some things that don't require such a heavy load.

So the episode would be 43, and that would be on last week, September 2nd.

So if you want to see what that does, I'm going to have a live demo of it.

Going to try it, but llama.cpp. Know it, learn it, love it.

Cool.

You know how I have all these older sixteen gig, uh, GPUs sitting around, like forty of them, that I thought were no longer useful? Well, it turns out suddenly they're useful again because some brilliant person came up with this idea of what's called KV cache streaming.

It's a fork of Llama CPP, which is the inference engine that serves models, and it turns out you can take this Alibaba Qwen twenty-seven B model, and I'll, I'll stop geeking out here in a second.

You can take this model, and you can run it on these sixteen gig cards with multiple concurrency lanes because of KV cache streaming, and it's freaking fast.

So all of a sudden, there's new life, uh, in my old GPUs, and I've been working to put them back into production in my mini data center, which is serving up all the, you know, all the book writing for the BrightLearn engine and, uh, all the document processing for BrightAnswers.ai, et cetera.

But I would also say if I were interested in starting a new company, I might run out and say I am starting a competitor to Hugging Face that's going to focus on non-NVIDIA hardware.

I don't think Hugging Face is hardware-specific, though, because the only thing they have that's hardware-specific is they support Llama C++, which does have hardware-specific builds.

But you go to any Hugging Face transformer repo, which is what I use, and they say, OK, great.

Here is how to run it on all 15 different kinds of hardware.