RAG systems usually underperform for one reason: teams optimise components in isolation. High-quality RAG requires coordinated decisions across ingestion, indexing, retrieval, orchestration, and evaluation.
Core trade-off map
1. Recall vs precision
- Higher recall can increase coverage but introduces noisy context.
- Higher precision reduces noise but risks missing key evidence.
Practical approach: tune retrieval for the decision type. Compliance workflows usually prefer precision and explicit citations. Research workflows can tolerate broader recall.
2. Latency vs grounding quality
More retrieval steps, reranking, and larger context windows often improve answer quality but increase response time and cost.
Practical approach: define separate latency budgets by user workflow (interactive assistant vs back-office generation).
3. Simplicity vs control
A single-chain architecture can ship quickly but struggles with policy enforcement and failure handling at scale.
Practical approach: introduce orchestration and guardrails incrementally, not all at once.
Architecture decisions that matter most
- Chunking strategy by document type, not one global chunk size.
- Index freshness policy with explicit SLAs for updates.
- Retrieval observability including hit rate, citation coverage, and abstain rate.
- Fallback path when confidence is low (abstain, route to human, or route to search).
Evaluation before scale
RAG quality cannot be inferred from user demos. Build a repeatable evaluation set:
- representative queries by workflow
- expected answer shape and evidence requirements
- failure taxonomy (hallucination, stale context, missing citation, wrong reasoning)
Run evaluation on every meaningful change: embedding model, reranker, prompt template, chunking, or retrieval parameters.
A phased implementation pattern
- Phase 1: single domain + narrow query set + strict citation policy.
- Phase 2: multi-domain retrieval with confidence routing.
- Phase 3: production SLOs for quality and latency with operational on-call ownership.
This progression reduces rework and helps teams avoid architecture lock-in.