Five Things We Learned Shipping RAG Systems

1. Chunking strategy matters more than the model

Before we tuned the LLM, before we tried re-ranking — chunking was the thing that determined whether retrieval worked at all.

Fixed-size chunks (500 tokens) with overlap work as a starting point. But documents with clear structure (headings, sections) benefit from semantic chunking that respects those boundaries.

# Naive chunking misses context
chunks = split_fixed(doc, size=500, overlap=50)

# Semantic chunking preserves document structure
chunks = split_by_headings(doc, max_size=800)

2. Evaluation before optimisation

We built an evaluation set before touching the retrieval pipeline. 50-100 question-answer pairs, manually verified, covering the spread of expected queries.

Without this, every change is a guess. With it, you can measure retrieval precision, answer faithfulness, and catch regressions.

3. Citations are not optional

Every RAG answer needs to point back to its sources. Not "based on your documents" — actual document names, section references, page numbers where possible.

People don't trust answers they can't check. We learned this the hard way.

4. Guardrails need to be specific

"Don't hallucinate" is not a guardrail. Specific topic boundaries, refusal patterns for out-of-scope queries, and confidence thresholds are guardrails.

We use a classifier that detects when the retrieved context is insufficient for the query. When confidence is low, the system says "I don't have enough information" rather than guessing.

5. Monitor retrieval, not just generation

Most RAG monitoring focuses on the LLM output. But if retrieval is returning irrelevant chunks, no amount of prompt engineering fixes the answer.

We track retrieval latency, top-k relevance scores, zero-result queries, and whether users thumbs-up or thumbs-down the answers.