Evaluating multi-hop question answering (QA) systems requires metrics that assess both the correctness of the final answer and the reasoning process used to integrate information from multiple sources. Traditional metrics like exact match (EM) or F1 score, which focus on surface-level text overlap with a reference answer, are insufficient because they don’t verify whether the model connected information across documents. Instead, effective evaluation should measure answer correctness, reasoning trace quality, and robustness to irrelevant or conflicting information.
First, answer correctness must account for whether the final answer logically combines facts from multiple documents. For example, if a question asks, “What disease is caused by both vitamin C deficiency and exposure to contaminated water?” the correct answer (“scurvy” from document A and “cholera” from document B) requires integrating two distinct facts. Metrics here could include human evaluation of answer validity or decomposition into sub-questions (e.g., verifying each hop separately). Automated methods might use entailment models to check if the answer logically follows from combined evidence. Datasets like HotpotQA include “supporting facts” annotations to validate intermediate reasoning steps, which can be used to measure accuracy at each hop.
Second, reasoning trace quality evaluates whether the model identifies and connects relevant information across documents. This can be measured by tracking the model’s intermediate steps, such as retrieved documents or generated explanations. For instance, a system might first retrieve a document about vitamin deficiencies and another about waterborne diseases, then explicitly link them to infer the answer. Metrics here include precision/recall of retrieved documents or coherence of generated reasoning chains. Tools like attention visualization or chain-of-thought prompting can help developers inspect if the model’s focus aligns with expected connections. Adversarial tests, where irrelevant documents are added, can also measure robustness to distractions.
Finally, logical consistency and coverage ensure the model avoids contradictions and fully addresses all parts of the question. For example, if a model answers “scurvy” but fails to mention “cholera,” it’s partially correct but incomplete. Metrics like BLEURT or ROUGE-L can assess answer quality, while structured formats (e.g., JSON outputs with evidence citations) enable automated checks for coverage. Human evaluators might score answers on a scale (e.g., 0-2) based on completeness and logical soundness. By combining automated checks with human judgment, developers can holistically assess whether a multi-hop QA system truly synthesizes information rather than relying on shallow patterns.
