How might user expectations differ for multi-hop questions (like expecting more detailed answers) and how should evaluation metrics reflect satisfaction for these complex queries?

User expectations for multi-hop questions differ from simple queries in three key ways: depth, logical coherence, and source reliability. Unlike single-step questions that require direct fact retrieval, multi-hop questions demand connecting information from multiple sources or reasoning steps. For example, answering “What was the average temperature in Paris when the first SpaceX rocket landed?” requires finding both the rocket landing date (2015) and Paris weather data for that period. Users expect answers to explicitly show how these pieces connect, not just present the final result. They also anticipate explanations of potential ambiguities, like whether “first successful landing” refers to Falcon 9’s 2015 milestone or earlier attempts.

Evaluation metrics must prioritize different factors than those used for simple QA. Traditional metrics like BLEU or exact match fail here because they focus on surface-level text similarity rather than reasoning validity. A better approach combines three elements:

1. Step correctness: Verify each logical hop (e.g., confirming the rocket landing date before weather lookup) using factual checks or grounded citations.
2. Answer cohesion: Measure whether the final answer synthesizes information coherently, avoiding contradictions between steps. Tools like logical entailment models or dependency parsing can help assess this.
3. User follow-up likelihood: Track implicit signals like whether users immediately ask clarifying questions or rephrase the query, indicating unresolved gaps.

For developers, this means moving beyond single-score metrics. A multi-hop evaluation system might combine:

• Automated fact verification at each reasoning step (using tools like Wikidata or domain-specific knowledge graphs)
• Human ratings for clarity in connecting concepts (e.g., on a 1–5 scale: “Does the answer explain why Step A leads to Step B?”)
• Session-level analysis tracking if users needed additional searches to validate intermediate results.

For example, an answer to “How did GDPR affect the adoption of AWS in German healthcare?” should explicitly link GDPR’s data residency rules to AWS’s Frankfurt region rollout, then connect that to healthcare migration patterns. An evaluation would penalize answers that correctly state both facts but fail to show causation, even if all facts are technically accurate. This granular approach better reflects user satisfaction with complex reasoning than traditional metrics.