AI Systems2024

Production RAG Infrastructure for Document Intelligence

End-to-end retrieval-augmented generation pipeline with hybrid dense-sparse retrieval, MMR reranking, and a streaming inference layer. Deployed for a fintech firm with 40K+ document corpus.

View Source
Pythontext-embedding-3-smallpgvectorFastAPINext.js
Key Metrics
Hallucination rate18% → 3%
p95 retrieval latency340 ms
Document corpus40 K docs
Impact

Domain hallucination rate dropped from 18% to under 3% on financial compliance queries. p95 retrieval latency of 340ms on a 40K document corpus.

Screenshots / Product Walkthrough
Query Debug View
Retrieval Score Breakdown
Streaming Answer UI
Problem Statement

Semantic similarity alone was retrieving relevant-sounding chunks that missed precise facts.

Pure dense retrieval with cosine similarity consistently retrieved thematically related documents while missing exact numerical values, dates, and regulatory clauses — the specific facts that matter most in financial document queries. A query for a specific SEBI circular date would return three documents about SEBI regulations, none containing the actual date. The system felt intelligent but was factually unreliable.

Solution

Hybrid dense-sparse retrieval with MMR reranking to balance precision and diversity.

Combined dense embedding search (text-embedding-3-small) with BM25 sparse retrieval, then fused the ranked lists using RRF (Reciprocal Rank Fusion). Added MMR reranking on the combined results to prevent redundant chunks from filling the context window. The hybrid approach lets BM25 anchor on exact terms while the dense model handles semantic variation — they are complementary, not competing.

Tradeoffs & Decisions

80ms additional retrieval latency in exchange for a measurable drop in hallucination rate.

The hybrid pipeline adds roughly 80ms to retrieval time compared to pure vector search. This was acceptable because the accuracy improvement was non-negotiable for the compliance use case — a hallucinated regulatory citation has legal consequences. The p95 latency sits at 340ms end-to-end. If latency had been a harder constraint, I would have explored caching the BM25 index differently or tiering the query path.

Architecture
  1. User query
  2. Query preprocessor — intent parsing + normalisation
  3. Dense retrieval — text-embedding-3-small + pgvector (k=20)
  4. Sparse retrieval — BM25 index (k=20)
  5. RRF fusion — reciprocal rank combination
  6. MMR reranking — diversity filter (k=5)
  7. Context assembly — chunk stitching + metadata
  8. LLM inference — streaming response to client
← All projectsNext projectLLM Evaluation Harness →