Building Production-Grade RAG Systems

Why RAG demos fail in production

A weekend RAG demo is roughly forty lines: load documents, split them, embed them, do a top-k cosine search, stuff the results into a prompt. It works because the corpus is small, clean, and the questions are the ones you thought of while building it.

Production breaks all three assumptions. The corpus is large, messy, multi-format, and changes hourly. Questions are ambiguous, multi-hop, and often unanswerable from the corpus at all. And every answer now carries a cost: latency budgets, token spend, and the reputational damage of a confident hallucination in front of a customer.

The failure modes cluster into five categories:

• Retrieval misses. The right chunk exists but never enters the top-k. Almost always a chunking or embedding-mismatch problem, not a model problem.
• Context dilution. Twenty mediocre chunks crowd out the two good ones and the model averages them into mush.
• Stale indexes. The vector store lags the source of truth, so the system answers from last month's policy.
• Unbounded latency. Sequential embed → search → rerank → generate chains that nobody budgeted end-to-end.
• No evaluation. Without a graded question set, every change is a vibe check and regressions ship silently.

A reference RAG architecture

Treat RAG as two independent systems that share a store, not one pipeline. The write path is a data-engineering problem; the read path is a latency problem. Keeping them separate lets you re-index without touching serving, and tune serving without a backfill.

Write path (asynchronous)

1. Source connectors and webhooks emit change events per document.
2. A worker parses, normalises, and deduplicates content into a canonical text form.
3. Structure-aware chunking produces chunks plus metadata (source, section, timestamp, tenant, ACL).
4. Embeddings are generated in batches and upserted into the vector store, keyed by a deterministic chunk ID so re-processing is idempotent.
5. The same chunks land in a lexical index (BM25/Postgres full-text) for hybrid search.

Read path (synchronous, budgeted)

1. Query understanding: rewrite the question with conversation context, expand acronyms, extract metadata filters.
2. Hybrid retrieval: dense vector search plus lexical search, filtered by tenant and permissions at the query level — never after retrieval.
3. Fusion and reranking down to a small, high-precision context set.
4. Grounded generation with citations and an explicit "insufficient context" escape hatch.
5. Post-processing: citation validation, PII redaction, telemetry.

In practice I run this as a FastAPI service with a queue-backed worker pool, Postgres for canonical documents and metadata, and Pinecone or FAISS for vectors depending on whether the deployment is managed or on-premise.

Ingestion and chunking

Chunking decides your ceiling. No reranker recovers a chunk that split a table in half. Fixed 512-token windows are a fine baseline, but structure-aware chunking beats them on every corpus I have measured.

• Split on semantic boundaries — headings, list groups, speaker turns, table rows — before falling back to token windows.
• Overlap 10–15% so a sentence spanning a boundary survives in at least one chunk.
• Prepend a breadcrumb ("Employee Handbook › Leave Policy › Carryover") to each chunk. It costs a few tokens and measurably improves both embedding quality and the model's ability to cite.
• Keep chunks addressable — store the source URI and character offsets so a citation can deep-link back to the original.
• Version your embedding model. Store the model name on every vector; switching embeddings means a full reindex, and mixed-model indexes silently degrade recall.

For conversational corpora the unit is different again. Meeting transcripts chunk best by topical segment rather than by time or token count: group consecutive speaker turns until the topic shifts, then attach participants and timestamps as metadata so "what did engineering commit to last Thursday" can be filtered before it is searched.

Hybrid retrieval, filtering, and reranking

Dense retrieval understands paraphrase; lexical retrieval nails identifiers, error codes, product SKUs, and surnames. Running both and fusing the ranked lists (reciprocal rank fusion works well and needs no tuning) is the single highest-leverage upgrade over a naive vector search.

Then narrow aggressively:

• Filter inside the query. Tenant IDs and ACLs belong in the vector store's metadata filter, not in a post-retrieval loop. Filtering afterwards is both a correctness bug and a recall bug.
• Over-retrieve, then rerank. Pull 40–60 candidates, run a cross-encoder reranker, keep the top 4–8. Precision at the context window matters far more than recall at k=50.
• Enforce a context budget. Cap total context tokens and drop the tail. More context is not more accuracy past a surprisingly low threshold.
• Deduplicate near-identical chunks — boilerplate headers and repeated policy text otherwise eat the whole window.

LLM orchestration and grounded generation

Orchestration is where RAG stops being retrieval and starts being a system. The rules I hold to:

• Make the prompt a contract. Numbered context blocks with IDs, an instruction to cite those IDs inline, and explicit permission to answer "not in the provided context."
• Validate citations programmatically. If a cited ID is not in the context set, the answer is rejected or retried. This alone removes most fabricated references.
• Route by intent. Not every question needs retrieval. Small-talk, arithmetic, and metadata lookups ("who attended?") should bypass the vector store entirely — cheaper and faster.
• Structure the output. JSON schemas or tool calls for anything downstream systems consume; free text only for humans.
• Stream, and degrade gracefully. Timeouts on every hop, a smaller fallback model, and a cached-or-partial answer beat a spinner.
• Log the whole trace. Query, rewritten query, candidate IDs, reranked IDs, final prompt, response, latency per hop. You cannot debug a RAG system you cannot replay.

Evaluation: the part everyone skips

You need a graded question set before you need a better model. A hundred real questions with known-correct source chunks is enough to turn tuning from guesswork into engineering.

• Retrieval metrics: recall@k and MRR against the known-correct chunk. Diagnose here first — most "the LLM is wrong" tickets are retrieval failures.
• Generation metrics: groundedness (is every claim supported by context?), answer relevance, and citation accuracy, scored by an LLM judge with spot-check human review.
• Negative cases: questions the corpus cannot answer. The correct behaviour is refusal, and it must be measured or the model will happily invent.
• Run it in CI. Any change to chunking, embeddings, reranker, or prompt re-runs the suite. Ship on the numbers, not on a demo that felt good.

Latency, cost, and scale

Budget the read path hop by hop — for example 40 ms query embedding, 60 ms hybrid search, 120 ms rerank, and the remainder for generation — and alert on p95 per hop rather than end-to-end only.

• Cache embeddings for repeated queries and cache full answers for high-frequency questions with a short TTL.
• Run retrieval branches concurrently; dense and lexical search have no dependency on each other.
• Batch embedding generation on the write path — per-document API calls are the usual reason a backfill takes days.
• Push heavy work asynchronous: summaries, tagging, and CRM updates belong in a queue, not in the request.
• Track cost per answered question, not tokens per call. It's the number that survives contact with a finance review.

Lessons from shipping Sally and TekInsight

Sally is a meeting intelligence platform at Liquid Technologies. It autonomously joins Google Meet and Teams calls, ingests transcripts through webhook-triggered LLM orchestration pipelines, and serves a conversational Q&A engine over a Pinecone index with GPT-4 synthesis. Two things mattered most: segmenting transcripts by topic rather than by fixed windows, and attaching participant and timestamp metadata so time-scoped questions filter before they search. The Q&A layer improved knowledge retrieval speed by 65%, and pushing summary generation and CRM sync onto the async path kept the interactive query fast.

TekInsight is an enterprise RAG platform built on FAISS and LangChain with JWT multi-tenant auth, deployed on IIS. Here the hard constraint was isolation: tenant scoping had to happen inside the retrieval query, and the index had to be partitioned per tenant so no ranking path could ever cross a boundary. Combining that with over-retrieval plus reranking cut document retrieval time by 60% and query latency by 40% against the original naive top-k implementation.

The through-line across both: the model was rarely the bottleneck. Chunking strategy, hybrid retrieval, metadata filtering, and a graded evaluation set delivered the gains — and they are the same four levers I would reach for first on any new RAG system.