Half-precision floating-point format

FP8 support is the newest frontier for high-efficiency AI workflows.

It allows models to run, uh, with half the memory or F- FP16.

Remember, FP16 is basically also half the memory of [chuckles] floating point 32.

Legacy enterprise cards absolutely completely blind to this optimization.

And the bill is real.

GPT-3, a hundred and seventy-five billion parameters, two thousand token context, about nine gigabytes of KV cache per request in FP16.

Ninety-six layers, two thousand and forty-eight tokens, two vectors, twelve thousand two hundred and eighty-eight dimensions, two bytes each.

Nine gigabytes per conversation before you've stored a single weight in gradient adjacent anything

The cache became the thing you optimize around, not the model Give me the memory hierarchy part, because I know you've been waiting.

Standard attention materializes the full score matrix N by N in high bandwidth memory.

At four thousand and ninety-six tokens in FP16, the scores in the softmax output are thirty-two mebibytes each.

Four intermediate trips through memory move about a hundred and twenty-eight mebibytes per layer per head, and the actual answer, the output tensor, is one mebibyte So ninety-nine percent of the traffic is scaffolding.

Memory bandwidth utilization, which is really DRAM activity, is one thousand and five.

Then the pipe activities, FP64, FP1002, FP16 at one thousand and six, one thousand and seven, one thousand and eight.

PCIe transmit and receive bytes at one hundred and nine and ten ten.

NVLink transmit and receive bytes at one one and one twelve.

So by compressing it, it cuts the memory reads in half.

So Blackwell supports FP8, KV natively, Apple stuck at FP16.

I'm gonna give kudos to Matt Kamusi who provided this analogy.

He said, imagine a highway memory bandwidth.

As Sony has been struggling for any good news recently, QSSR came out of nowhere and it is genuinely a pleasant surprise.

Somehow Sony squeezed a PSSR-derived model onto a 2020 GPU just using FP16.

Notably, a similar challenge AMD took on with the promise of releasing FSR 4 and RDNA 2.

I think that's supposed to come out early next year.

...

when they have sign- they do have more teraflops and I believe support, uh, one of the other, um, compute types that PlayStation 5 doesn't.

They wouldn't have to use FP16.

So they'd actually have quite a leg up with an upscaler versus base PS5 and yet Xbox is nowhere.

I guess we should kick off by talking about, you know, what is QSSR? And it's really interesting, right? Because fundamentally, the PlayStation 5 Pro has got dedicated machine learning instructions, Int8 support, which the base PlayStation 5 doesn't have.

So instead, what QSSR is, is effectively a version, a refactoring of PSSR streamlined for FP16, which PS5 does support with the benefit of rapid-packed math.

So we actually had the opportunity to pose some questions to Sony's Daniel Craig, not that one, principal engineer, graphics R&D SIE.

So basically, he describes it as a drop-in replacement for titles that already support PSSR.

But we'll talk about performance later.

Anything to add to that, Oliver?

Oh, not too much, just to say that the general approach of basically porting over PSR from INT8 to FP16 is surprising, but that does seem like, you know, obviously that's the way that you have to go here.

FP16 is a much more general and more precise data format.

Choose Ternary Bonsai 27 BGGUF when it's reported 7.

2GB footprint, 95% FP16 intelligence retention claim, or approximately 26 tokens per second on an Apple M5 Pro laptop matches your deployment target.

The supplied comparison data does not establish a direct quality or speed test between that model and this crack build.

ternary bonsai-8b gguf choose when you need higher model capacity 27b class reasoning vision support and the crack refusal profile choose ternary bonsai-8b gguf when lower memory use and faster local inference matter more than 27b class capability the supplied information identifies the 8b model as a ternary 1 58 bit gguf model bit does not provide its file size or speed here Ternary Bonsai 1.7 BGGUF Choose for quality, reasoning depth, vision, and tool use experiments.

Choose Ternary Bonsai 1.7 BGGUF for lightweight edge deployment, quick tests, or workloads where a 27B model is too large.

The supplied material does not provide a direct benchmark, file size, or tokens per second comparison.

bonsai 27 bgg uf choose when you want the newer bonsai 2 family ternary packaging vision support and explicit refusal circuit removal choose bonsai 27 bgg uf when asmaller approximately 3 9 gigabytes binary model and it's reported approximately 44 tokens per second on an apple m5 pro laptop are more important than the crack model's stated capability preservation target The alternative reports approximately 90% of FP16 intelligence retained, while this model's supplied MMLU comparison is against its ternary base rather than FP16.

Technical specifications, has these confirmed properties, task, text-to-text generation, pipeline, parameter scale, 2.7b, architecture, hybrid attention plus SSM using gated delta net, Depth.

Int8 is the sweet spot.

Float16 is workable if you have a TPU.

Float32 is desktop territory.

Int4 is aggressive but works for tiny models.

Well, it's pretty quickly realized that having 32 bits precision was super overkill for the things that you're wanting to represent and, you know, it's both cost more from a storage and computational perspective than lower precision.

So we went to FP16, you know, Google developed BF16, you know, a little rejiggering of those bits.

Went to FP8, now we're at FP4, you know, in popular systems.

And so we've been reducing the precision quite quickly, but that does lead to, you know, for lack of better word, some brain loss when, when these models run, just because you are now trying to encode the, the same information into fewer bits.

And so there's been a lot of interesting schemes to say, "Okay, I'm going to take this group of FP16 values or BF16 values, and I'm going to quantize those, so I'm going to, you know, use a truncate and rounding that, that down to, let's say, int4 values." So now you, you actually saved 75% of your total size of that group of values.

You shrunk that down from 16 bits to 4 bits.

But just doing that naively will mean that on a lot of benchmark scores, you'll have them get 20, 30% worse.

So you, you get that 75% savings in, in space, but y- you kind of lobotomize the model.