Okapi BM25

When we were working on this sort of transition off of Elasticsearch, we, we h- did a bunch of work to evaluate a different technology, which was reverse indices in Postgres.

So our thesis was, okay, we will run Elasticsearch, basically, Elasticsearch style Lucene reverse indices, BM25 indices, in Postgres.

And there was this company, this startup, this YC startup that was doing it.

Brilliant team, really top-notch, great technology.

For that, they use Vectorize and AI search.

This handles your HNSW dense vector indexing combined with BM25 keyword matching for your hybrid search.

And then they wrap all of this in their AI gateway, which handles edge caching, rate limiting, observability.

Got it.

Precisely.

And that's just the state coordination.

When we look at the RxZ support, Cloudflare's AI search handles managed hybrid search out of the box, so dense vectors combined with BM25.

Which is great.

The one that is ranked low, the analysis agent can like analyze the goal document, analyze the query, see what got ranked above it by using its tools and say, oh, like it seems like things that were ranked higher, I see this pattern.

And I have a theory that something in the query, like we were using BM25 as a retriever.

So it's very important to have like overlapping terms, obviously.

So, oh, it seems like a really important thing term that's not common in other documents in the goal document um our representation program currently just left it out for some reason like we deleted it that's like a very simple case and the analysis agent kind of like looks at that and can kind of reason about the feedback it's getting from using these tools and exploring the data set to figure out what's wrong with the current representation.

And it was kind of like messing it up.

Cause the N 25 is just like a lexical matching retriever.

Um, And I think the reason that worked well with BM25 is because it's very easy to see just in text what is going wrong.

Like it's a bag of words.

How does it actually do that?

Specifically, it uses an algorithm called BM25.

And BM25 doesn't just do a dumb find in page search.

It calculates term frequency versus inverse document frequency.

OK, you're gonna have to break that down for me.

It asks a mathematical question.

How many times does this exact part number show up in this specific document compared to how rare it is across the entire company database? If the part number 4F9920X is very rare globally, but shows up 10 times in one specific PDF, BM25 heavily weighs that PDF as the correct answer.

Oh, so it understands the rarity and the density of the exact keyword.

That insight is, especially in the world of search, earlier in the year, I had Doug Turnbull on the WeVA podcast, who's like legend in the search community.

And he was really excited about RLMs and writing search programs, the way you can, like, because like with Grep and BM25, the way you can search and then get that feedback.

It's so powerful, and it can write this particular regular expression to, you know, command to find the files.

It's such a captivating world in search.

Uh, the, the very first thing on my playbook is, is evals and measurement, and I know-

BM25 as a baseline, perhaps.

Yeah, just, just get a good baseline in place and measure how good it is to see where you're gonna find things that work really well and things that don't work really well.

And that's honestly the biggest gap most organizations have with search.

And once you're building a custom frontend anyway, the argument for keeping things simple with Markdown files starts to feel... quaint.

Why not just use Pinecone? Because Pinecone costs money and adds operational complexity and ties you to a specific vendor and requires you to manage embedding models and re-indexing pipelines and all of that for what? For a few thousand tokens of project memory, the BM25 algorithm, which is just a fancy bag-of-words search, no embeddings, no GPU, performs comparably to dense retrieval for moderate-sized document collections.

You can run it on a laptop.

For tens of thousands of tokens of markdown files, BM25 or a tiny local embedding model like all, MiniLML6v2 is more than enough.

You don't need a vector database until you're dealing with millions of documents.

So the market has

When the user asks a question, you embed the question, compute cosine similarity against all the stored vectors, and pull the top k results.

The whole thing runs in milliseconds for a few thousand documents.