Start with a question and its evidence.
A useful test case names a question, the source passage needed to answer it and the behavior expected when that evidence is absent. Keep the source version with the test. Otherwise a document update can silently change what “correct” means.
Inspect retrieval before judging the answer.
Look at the passages the system actually retrieved. If the required evidence is missing, rewriting the answer prompt may hide the problem without fixing it. Compare retrieval candidates against the same questions and source collection.
Evaluate the response separately.
When the evidence is available, review whether the answer is supported, whether it addresses the question and whether its citations point to the relevant passage. A citation that links to an entire document is harder to inspect than one tied to a specific passage.
Include questions that should not be answered.
Add cases with missing information, conflicting versions and sources the user cannot access. Define the expected behavior for each case. A useful system must sometimes ask for clarification or say that the available evidence is insufficient.
Keep a small, versioned evaluation record.
Record the question, expected source, retrieved passages, generated answer and reviewer decision. Preserve the configuration used for each run. The first goal is to make changes comparable, not to produce one impressive aggregate score.
Further reading
This note presents our suggested working method. For broader architectural guidance, see Microsoft: Design and develop a RAG solution ↗.
PUT IT TO WORK
Review production readiness ↗