Retrieval-augmented generation (RAG) is the default pattern for grounding a language model in your own data. The demo is trivial: embed some documents, retrieve the top matches, stuff them in the prompt. Production is where it gets interesting.
1. Retrieval quality beats model quality
A smaller model with great retrieval will out-answer a frontier model with poor retrieval. Invest in chunking, metadata and ranking before you reach for a bigger model.
2. Chunk for meaning, not for size
Fixed 500-token chunks shred context. Chunk along semantic boundaries — sections, headings, list items — and keep enough overlap to preserve meaning.
3. Evaluate, or you are flying blind
Build an eval set of real questions with known-good answers early. Without it, every prompt tweak is a guess and every regression is invisible.
4. Cite sources, always
Returning the source passages alongside the answer does two things: it lets users verify, and it makes hallucinations obvious. Trust comes from traceability.
5. Guardrails are a feature
Decide what the assistant must not do and enforce it — refuse out-of-scope questions, never invent figures, escalate to a human when confidence is low.
6. Watch your latency and cost
Retrieval, reranking and generation each add latency and cost. Cache aggressively, stream responses, and use the smallest model that passes your evals.
7. Keep a human in the loop where it matters
For high-stakes outputs, route to a human reviewer. The goal is not to remove people — it is to make them dramatically more productive.
“A trustworthy AI feature is mostly retrieval, evaluation and guardrails. The model is the easy part.”
Elena García
Head of AI
Leads generative-AI engineering at CRUDTree — RAG, agents, evals and the unglamorous work that makes AI features trustworthy.