Search Party: How Two Rival Ideas About Retrieval Learned to Share

RAG, retrieval, BM25, embeddings, production

For decades, search split into two camps: the literalists who matched words, and the semanticists who matched meaning. The peace treaty between them is now the quiet backbone of production AI.

A user typed five words into a system we had built: show me prd_itm_rev last quarter. The machine, fluent in the language of meaning, thought for a moment and returned five documents about product revenue. Every one of them was reasonable. Not one of them mentioned prd_itm_rev, the actual column the user needed. The model was not broken. It had simply been educated, like a clever graduate who grasps the gist of a question while missing the one fact that matters.

That small failure is a parable for a much older argument — one that has divided the field of information retrieval for half a century, and which, after years of stalemate, has lately been settled by compromise.

Two schools, two centuries of stubbornness

The first school is the literalists. Their great instrument, BM25, was forged in the 1970s and 1980s out of the Okapi project and has the unfashionable virtue of still working. It treats a document as a bag of words and asks a blunt question: do the query's terms appear here, and how rarely do they appear elsewhere? Rare words count for more; common ones for almost nothing. It is fast, it needs no training, and it is gloriously literal. Ask it for prd_itm_rev and it will find prd_itm_rev. Ask it for "car" when the document says "automobile" and it will shrug.

The second school is the semanticists. Their instrument is the embedding: a neural network that folds language into a geometric space where things that mean the same land near one another. To this school, "car" and "automobile" are neighbours, and "how to fix a bike" sits comfortably beside "repair bicycle". It is a marvellous trick. It is also expensive, occasionally vague, and prone to sailing straight past a rare technical term it has never properly met.

Each school, in other words, is brilliant precisely where the other is blind.

The literalist (BM25) The semanticist (embeddings)
Matches exact words meaning
Synonyms no yes
Rare codes, IDs yes often missed
Speed sub-millisecond 5–50ms
Training data none a whole corpus

For years, engineers were asked to pick a side. The fashion of the moment favoured the semanticists, and so many systems — ours among them, briefly — shipped with embeddings alone, only to discover the awkward truth that customers keep typing exact things into search boxes.

BM25 exact words prd_itm_rev SKU-1234 error strings Embeddings meaning car ≈ automobile fix bike ≈ repair bicycle Hybrid best recall
Each method owns a different failure. The overlap is where production systems want to live.

The peace treaty

The resolution, as so often in economics, was not victory but trade. Run both retrievers. Let the literalist and the semanticist each nominate their favourite documents, then combine the two lists. The strengths overlap; the blind spots do not.

The difficulty is diplomatic. BM25 hands you raw relevance scores; embeddings hand you cosine similarities; the two currencies do not convert. Try to add them directly and you get nonsense, like settling a bill half in pounds and half in degrees Celsius.

The elegant fix is to ignore the scores entirely and look only at the rankings. Reciprocal Rank Fusion — the workhorse of modern hybrid search — does exactly this:

RRF(doc) = Σ  1 / (k + rank_in_each_list)

A document both retrievers rank highly rises to the top. A document only one of them found still earns a seat, just a humbler one. The constant k is set to 60 by long convention, and in our experience there has never been a pressing reason to argue with it. No normalisation, no exchange rate, no haggling.

BM25 ranking Embedding ranking Fused (RRF) 1 · doc A 2 · doc B 3 · doc C 4 · doc D 1 · doc A 2 · doc C 3 · doc B 4 · doc E 1 · doc A 2 · doc B 3 · doc C top in both → wins
Fusion works on rank position, not raw score. A document both retrievers favour is lifted above one that only a single retriever loved.
def rrf_fusion(sparse, dense, k=60, w_sparse=1.0, w_dense=1.0, top_n=20):
    scores = {}
    for rank, (doc_idx, _) in enumerate(sparse, start=1):
        scores[doc_idx] = scores.get(doc_idx, 0.0) + w_sparse / (k + rank)
    for rank, (doc_idx, _) in enumerate(dense, start=1):
        scores[doc_idx] = scores.get(doc_idx, 0.0) + w_dense / (k + rank)
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)[:top_n]

When one camp deserves a louder vote — keywords, say, in a corpus thick with codes and identifiers — a weight tips the balance. We hand the literalists about a third more sway when searching schemas and source code.

An assembly line, not a single machine

A production system runs this as a pipeline of three stages, each one narrowing the field, in the manner of a firm that interviews fifty candidates to hire five.

Query BM25top 50 Dense ANNtop 50 RRFtop 20 Cross-encodertop 5 LLM recall — cast wide precision — the shortlist
Fifty candidates interviewed, five hired. Each stage trades breadth for precision.

First, cast a wide net: both retrievers return around fifty candidates each. Recall is the only goal here; miss a relevant document now and no later stage can rescue it. Second, merge the lists with RRF down to twenty. Third — and this is where much of the quality hides — send the survivors to a cross-encoder, a model that reads the query and each candidate together, as a pair, rather than judging them at a distance. It is the expensive senior interviewer, brought in only for the shortlist. We once cut this stage to save a few milliseconds and quietly put it back within the week.

The whole arrangement, assembled:

from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer, CrossEncoder
import faiss

class HybridRetriever:
    def __init__(self, documents):
        self.documents = documents
        self.bm25 = BM25Okapi([d.lower().split() for d in documents])
        self.embed = SentenceTransformer("all-MiniLM-L6-v2")
        emb = self.embed.encode(documents, normalize_embeddings=True).astype("float32")
        self.index = faiss.IndexHNSWFlat(emb.shape[1], 32)
        self.index.add(emb)
        self.reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")

    def retrieve(self, query, top_p=5):
        toks = query.lower().split()
        sparse = sorted(enumerate(self.bm25.get_scores(toks)),
                        key=lambda x: x[1], reverse=True)[:50]
        q = self.embed.encode([query], normalize_embeddings=True).astype("float32")
        dist, idx = self.index.search(q, 50)
        dense = list(zip(idx[0].tolist(), dist[0].tolist()))

        cands = [i for i, _ in rrf_fusion(sparse, dense)]
        pairs = [(query, self.documents[i]) for i in cands]
        ranked = sorted(zip(cands, self.reranker.predict(pairs)),
                        key=lambda x: x[1], reverse=True)
        return [self.documents[i] for i, _ in ranked[:top_p]]

Those who already keep house with LangChain will find its EnsembleRetriever does the fusion for them. Those whose database speaks hybrid natively — Weaviate, Qdrant, Elasticsearch, Azure AI Search — may dispense with the plumbing altogether, since these systems quietly maintain a word index and a vector index in parallel and reconcile the two behind the curtain.

The price of balance

How much weight each camp deserves is, like most things, a matter of context rather than principle.

Conversational, FAQ         lean semantic   (~0.7)
Technical documentation     even-handed     (~0.5)
Code and identifiers        lean literal    (~0.35)
Legal, medical, exact IDs   lean literal    (~0.3)

The frontier, naturally, is to stop guessing. A 2025 proposal called Dynamic Alpha Tuning lets a language model inspect each query and set the balance on the fly — generous to the semanticists for a chatty question, deferential to the literalists for an error code. A fixed weight, after all, serves the average query and shortchanges every actual one. We have not yet trusted it in production, but the logic is sound and the direction unmistakable.

The usual suspects

Every elegant theory keeps a folder of operational embarrassments. Ours:

  • The empty-handed literalist. If not one query term appears anywhere, BM25 returns nothing at all. Fall back to embeddings rather than serving a blank page.
  • The tokeniser that drifts. Index your words one way and query them another, and recall bleeds away silently. We lost a day to exactly this; the lesson was cheap only in hindsight.
  • The stale index. An in-memory keyword index does not refresh itself when documents change. Managed databases tend to; libraries do not.
  • The mistaken identity. A cross-encoder ranks; it does not write. Confuse the reranker with the generator and the architecture grows muddy fast.

A note on keeping score

The temptation is to judge the whole contraption by its final answer. Resist it: a bad answer might be a retrieval failure or a generation failure, and the two demand different cures. We measure retrieval on its own terms — ranking quality on a fixed set of questions — and generation on whether its claims are actually supported by what was retrieved. Assemble that test set before tuning anything, or every adjustment becomes a matter of faith.

The settled verdict

For any system that must field both precise terminology and loose human phrasing — which is to say, almost all of them — the combination of keyword search, rank fusion and a cross-encoder is no longer the clever optimisation. It is the sensible default, the thing you reach for first and depart from only with reason. The semanticists were never wrong, exactly. They were merely incomplete, as the literalists had been before them. The lasting lesson, here as elsewhere, is that the productive move was to stop choosing.