Evaluate a RAG system
Build a test set, measure retrieval and answers separately, and catch regressions before users do.
- Build a representative evaluation set
- Compute precision@k, recall@k, MRR and nDCG
- Measure answer faithfulness and know the limits of automatic judges
You can’t improve what you don’t measure - and RAG has many knobs (chunk size, k, hybrid weights, reranker, prompt). Start with an evaluation set: 50-200 realistic questions, each with the chunks that answer it and a reference answer. Include paraphrases, multi-part questions, questions from different user roles, and unanswerable ones the system should decline.
Then measure the two stages separately, so you know where a failure came from:
- Retrieval: did the right chunks come back? → precision@k, recall@k, MRR, nDCG.
- Generation: given the chunks, is the answer correct, relevant and faithful (every claim supported by the context)?
Try it
Up to 15 XPScore two retrievers
The question is “How do I reset a forgotten password?” and 3 chunks in the corpus are relevant. Compare the systems, move k, and click results to change their relevance judgments. Which metric rewards getting a relevant result to the very top?
Precision@3
33%
Share of the top k that is relevant
Recall@3
33%
Share of all 3 relevant chunks found in the top k
MRR
33%
1 / rank of the first relevant result
nDCG@3
23%
Rewards relevant results near the top
Click a result to change its relevance judgment. Results below k are faded - they don’t count.
Judging answers
Answers are harder to score than rankings. Teams combine:
- Faithfulness / groundedness: the share of claims the context supports.
- Answer relevance and correctness against the reference answer.
- LLM-as-judge: a model grades answers with a rubric. It scales, but it has biases - check its grades against a sample of human ratings before trusting it.
- Abstention: on unanswerable questions, did it decline?
Simple word-overlap checks are cheap but blind to meaning: “is not covered” and “is covered” share almost every word. The exercise below shows exactly that failure.
Run the evaluation on every change (a new chunk size, a new model) like a test suite, and track latency and cost alongside quality.
Key takeaways
Build an evaluation set with expected chunks, reference answers and unanswerable questions.
Measure retrieval (recall@k, MRR, nDCG) separately from generation (faithfulness, correctness).
Use LLM judges with care, and run evaluations on every change.
Lesson quiz
6 questions · pass with 5 correct · up to 50 XP
Passing this quiz completes the lesson and keeps your streak going. Questions you miss come back in review sessions later.
Practice: write Python
Write Python in the editor and run it against sample inputs. Python runs locally in your browser using a WebAssembly runtime.
Evaluate retrieval over a test set
The first line is k. Each following line is question|retrieved ids|relevant ids (ids comma-separated, retrieved in rank order). Print question recall@k=R rr=X for each (2 decimals; the reciprocal rank counts only the top k), then mean recall@k=R MRR@k=X.
- Three questions
Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.
A naive faithfulness check
The first line is the retrieved context; each following line is an answer sentence. For each sentence, take its words longer than 3 letters (lowercase [a-z0-9]+) and compute the share that appear among the context’s words. Print 0.75 supported: sentence when the share is at least 0.6, otherwise ... unsupported: ... (2 decimals), then faithfulness: S/T.
Then look closely at the second sentence: is it really supported?
- Warranty answer
Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.
Questions about this lesson
Stuck? Ask. Figured something out? Share it. Explaining is one of the best ways to learn.
Loading posts…