How can we test whether a RAG system properly handles queries requiring multiple pieces of evidence? (Consider having test queries where leaving out one retrieved piece would make the answer incorrect.)

To test whether a RAG system correctly handles queries requiring multiple pieces of evidence, start by designing test cases that explicitly demand integration of distinct facts from separate sources. For example, a query like, “What are the environmental impacts of electric vehicles and how do they compare to gasoline cars?” requires evidence on EV battery production (e.g., lithium mining impacts) and gasoline car emissions (e.g., CO2 per mile). If the system retrieves only one document (e.g., battery data but not emissions comparisons), the answer will be incomplete or incorrect. Test cases should validate that all necessary evidence is retrieved and synthesized accurately.

Next, simulate scenarios where partial or irrelevant retrieval could mislead the answer. For instance, create a knowledge base with overlapping but incomplete documents. Suppose a query asks, “How did the Treaty of Versailles contribute to WWII and what economic policies followed it?” If the system retrieves a document on the treaty’s territorial clauses but misses one on post-treaty hyperinflation in Germany, the answer might blame territorial disputes alone while ignoring economic factors. To detect this, track whether the system’s response includes all critical points defined in a pre-built checklist (e.g., territorial changes, reparations, economic collapse). Automated checks can flag missing components, while manual review ensures nuanced connections between facts are preserved.

Finally, test adversarial cases where omitting a single document alters correctness. For example, a medical query like, “What are the first-line treatments for hypertension in diabetic patients?” requires combining guidelines for hypertension (e.g., ACE inhibitors) and diabetes-specific considerations (e.g., avoiding beta-blockers if hypoglycemia is a risk). If the system retrieves general hypertension guidelines but misses diabetes-specific advice, the answer could recommend unsafe options. To validate robustness, systematically remove one critical document from the retrieval pool and verify if the system’s answer becomes incorrect. This “ablation” approach isolates dependencies, ensuring the model isn’t over-relying on partial data or making unsupported assumptions. Pair this with metrics like precision (correct inclusions) and recall (missing facts) to quantify performance gaps.