What hybrid retrieval actually means (and when BM25 still wins)

Dense embeddings feel like magic until you need to recall a specific invoice number.

Hybrid retrieval — combining the semantic richness of dense embeddings with the exact-match precision of BM25.

Dense vector retrieval has a particular failure mode that doesn't show up in general-purpose benchmarks but surfaces immediately in production on domain-specific data. The failure mode is precision on low-frequency exact terms: invoice numbers, reference codes, specific dates, unique identifiers, regulatory clause numbers. Anything that looks like a specific token sequence rather than a semantic concept.

The intuition for why is straightforward. Embedding models are trained to capture semantic similarity. A document that contains 'invoice INV-2024-00447' and a document that discusses 'invoice processing workflows' will have semantically similar embeddings, because both are about invoices. But if your user is querying for INV-2024-00447 specifically, the second document is irrelevant. The embedding model can't distinguish between 'related to invoices' and 'contains this specific invoice number' — and for this class of queries, that distinction is the entire job.

BM25 handles this trivially. It's a term frequency model: the document that contains the exact query terms ranks higher. 'INV-2024-00447' in the query hits 'INV-2024-00447' in the document as an exact match. The semantic richness that makes dense retrieval valuable is irrelevant here.

Dense vs BM25 on an exact-identifier query — the embedding model conflates semantic relatedness with specific recall; BM25 does not.

Hybrid retrieval combines both signals. The simplest approach is score fusion: run dense retrieval, run BM25, and combine the ranked lists. Reciprocal Rank Fusion (RRF) is the canonical method — it combines rankings rather than raw scores, which avoids the normalisation problems that arise when you try to add scores from two different systems with different scaling.

Reciprocal Rank Fusion — RRF(d) = Σ 1/(k + rank(d)) across retriever lists, where k=60 is the standard smoothing constant.

The practical decision is what to fuse and at what stage. I've found that fusing at the retrieval stage (before reranking) works better than fusing at the reranking stage, because the reranker gets to see a candidate set that includes both semantically relevant documents and exactly matching documents, and can make better judgements about what actually answers the query.

One thing worth naming: hybrid retrieval is not always better than pure dense retrieval. For tasks where semantic similarity is the right retrieval signal — recommendation, clustering, general Q&A over unstructured text — BM25 can add noise without adding signal. The decision to use hybrid retrieval should be driven by an analysis of your query distribution. If a significant fraction of your queries are looking for specific named entities or exact values, the investment pays off.

Run both on your actual query distribution with your actual data before deciding. The benchmark that matters is yours.

All Thinking