Enterprise Knowledge Search — Context Engine
Semantic search and document intelligence layer across 200K+ internal documents, with natural language queries, role-based access filtering, and a streaming answer interface.
Average research task time reduced from 45 minutes to 8 minutes across 500 enterprise users. Pipeline handles 200K+ documents across 4 source systems with daily incremental updates.
Enterprise knowledge lives in incompatible silos — and unified ingestion was the real problem.
The search problem looked like a retrieval problem. It was actually an ingestion problem. SharePoint, Confluence, Notion, and email archives all have different document structures, permission models, and API behaviours. Building per-source search meant 80% of the work was adapting to each source format, with retrieval quality varying entirely based on how well each connector parsed documents. A query for "Q3 budget approval" returned different results depending on which source happened to index first.
A canonical document schema with per-source connectors that normalise before chunking.
Built a single ingestion interface: every document, regardless of source, is normalised into a canonical schema (title, body, metadata, permissions, source ID) before entering the processing pipeline. Chunking, embedding, and indexing operate on canonical documents with no knowledge of the source system. Each connector is responsible for one thing: mapping source format to the canonical schema.
Connector development is slower, but the retrieval system is permanently source-agnostic.
Writing a connector correctly takes 3-4 days — field mapping, permission normalisation, incremental sync logic. A direct integration would have been faster to ship but would have infected the retrieval layer with source-specific assumptions. As soon as you build retrieval that assumes SharePoint document structure, adding a Notion connector breaks it. The canonical schema is the cost of not paying that price repeatedly.
- Source systems — SharePoint / Confluence / Notion / Email
- Per-source connectors — normalise to canonical document schema
- Chunking pipeline — semantic + fixed-size with overlap
- Embedding — text-embedding-3-small
- pgvector index — with role-based access metadata
- Query processor — intent parsing + access filter
- Hybrid retrieval + MMR reranking
- Streaming answer — GPT-4o with source citations
- Next.js frontend — real-time streaming UI