RAG evaluation
Evaluate RAG pipelines end-to-end — retrieval quality, faithfulness, and answer quality.
A RAG (retrieval-augmented generation) pipeline has more failure modes than a plain LLM call. Bad retrieval kills good models; great retrieval can't save a hallucinating one. RAG evaluation scores each stage so you know where the failure actually is.
The shape of the work
Define the three dimensions.
Retrieval quality — did we get back relevant chunks?
Faithfulness — did the answer ground itself in the retrieved chunks (not invent)?
Answer quality — did the final answer actually help the user?
Pick scorers and judges.
Retrieval: classical IR metrics (precision@k, recall@k) as scorers; or an LLM judge that grades chunk relevance.
Faithfulness: an LLM judge that compares answer claims against the retrieved chunks. GEPA-optimize against a labeled set.
Answer quality: an LLM judge for end-to-end helpfulness, again GEPA-optimized.
Run as a trace evaluation. Your pipeline emits a trace with retrieval span + generation span. Stratix grades each.
Compare configurations. Try a different retriever, a different chunk size, a different model — re-run, see the score deltas per dimension.
Lock in CI gates. Faithfulness regressions are particularly costly; gate them.
Why it works on Stratix
Trace-first. Multi-span traces are first-class; you can score retrieval and generation separately.
Judge optimization. GEPA tunes judges to match human labels for faithfulness — usually the trickiest dimension.
Compare-models. Try the same retrieval feeding three different generators; see which generator is most faithful to the same chunks.
Tools you'll use
Stratix Premium — Agent Evaluation (RAG is a small agent)
Outcomes you should see
You'll know this is working when:
Faithfulness judge agreement with humans crosses 90% after GEPA optimization.
Retrieval and answer scores diverge meaningfully — when answer quality drops, you can tell whether retrieval or generation is at fault.
Hallucination rate trends downward over consecutive releases, not just bounces.
Per-chunk-strategy comparisons take <1 day instead of multi-week experiments.
Anti-patterns
Scoring only the final answer. You'll know quality dropped; you won't know which stage caused it.
Faithfulness judge without optimization. Out-of-the-box LLM judges over-credit confident-sounding hallucinations. GEPA-optimize against a labeled set.
No retrieval ground truth. If you can't say "these are the right chunks," your retrieval scores are just plausibility scores.
Where to next
Last updated
Was this helpful?