n-gram

The 1980s and 90s.

Google Ngram is really not accurate, but the Ngram chart, a sharp dramatic spike in mid-80s, peaking in the mid-90s.

And then it's a steady decline, mostly taken over by actually, what are the, what words do you think have replaced? Uh, this is an interesting thing.

And why, what words do you think replaced dweeb specifically? Oh, wow.

So a lot of efficiencies to be found there.

Another thing is that some of the AI models now use a lookup table, which I think is generally referred to as an Ngram table.

And for a model like DeepSeq version 4.1 Flash, which I'm installing right now, that table can be quite large, like 70 gigabytes in size.

It turns out that table doesn't need to be loaded into GPU RAM either.

And you can go even further than that now.

We're not going to talk about it in this podcast, but like I've been running QEN 3.8 Flash Next, where you've got this new kind of emerging Ngram table, which is essentially like a database that's even simpler for the model to access.

and it doesn't take as much bandwidth to do so.

So that Ngram table now can be loaded onto your actual storage like SSD and it not bottleneck your performance of the model, making you have smaller weights on RAM and more on SSD.

I think it takes it kind of another step further, and I wanted to simplify it here today, but there's cool stuff happening in what weights you need to access.

Well, to your point,

And so just to clarify a couple of things, right, so the semantic IDs that we generated, that's, you know, described in the paper, yes there are three level semantic ids but the features that we ended up using even in these like aggregate dense features were based on semantic ids uh like the n grams of semantic ids so our semantic id code book size is like 512 cubed today so like each level has 512 possible ids and in order to like scale that effectively and also build dense features that are meaningful, that are not too sparse effectively.

We went down the route of actually building Ngram tokens and then using those Ngram tokens to aggregate our engagement features at the item and consumer level.

Exactly like you said, it could be something like an Ngram that represents all the way down to strawberries, like a strawberry cluster of items based on the semantic IDs.

And then based on the engagement, we at the item level, we would look at the engagement across the surfaces and also at the consumer item level engagement, right? So just those features by themselves, like where impactful for our model, like actually added much more values such that we could even, I think we ended up removing like around 17 to 20 features from our original model

that were taxonomy based.

Even though they thought the same or may have thought the same about blacks.

Consider this Google Ngram on the Ngram racial discrimination.

Giving an index of the counts of books in the Google American English Corpus, they used the phrase at least once.

As you can see, it spikes in 1948 with the desegregation of the armed forces by Truman.

So within a few years, uh, things picked up in the 1960s, and the phrase separation of church and state was suddenly everywhere.

So take a look at this chart from, uh, Google's Ngram, which, uh, I think kind of illustrates it.

It tracks all mentions of the phrase separation of church and state in books and periodicals going back to the 1800s, and as you can see, you know, pretty much no one was using the phrase until about the 1960s.

Very similar trajectory to the Trail of Tears, which also became a household name very suddenly in the same decade.

Persistent SWA KV is about one-eighth of.

Other components: single-pass MHC with a MegaMHC kernel, Ngram conditional memory with one hundred and ninety-six B parameters and token-based sparse lookup.

D-Spark speculative decoding with semi-autoregressive drafts and confidence scheduled verification.

Vision stack: DeepSeekViT trained from scratch, 2D Rope, and pixel unshuffled downsampling.