Evaluating a multi-step retrieval RAG system differs from a single-step approach primarily in how you track intermediate retrieval accuracy and assess its impact on final answer correctness. In a single-step system, you evaluate retrieval quality once (e.g., relevance of retrieved documents) and directly measure how well the generated answer aligns with ground truth. For multi-step systems, you must evaluate each retrieval phase (e.g., initial query, refined follow-up queries) to ensure accuracy at every stage, as errors in early steps can compound and degrade final results. This layered evaluation helps identify where improvements are needed—whether in query refinement, context expansion, or filtering.
In practice, single-step retrieval evaluation focuses on metrics like precision@k (e.g., how many of the top 5 documents are relevant) and recall, paired with answer correctness metrics like exact match or F1 score. For multi-step systems, you’d track similar metrics for each retrieval step. For example, if the first step retrieves broad context and the second step narrows it down, you’d measure precision@k for both steps independently. Additionally, you might analyze how intermediate outputs (e.g., summarized context from the first retrieval) influence the second query’s effectiveness. Final answer correctness would still rely on standard metrics, but you’d correlate it with intermediate results to diagnose issues—like a correct final answer relying on flawed intermediate steps (e.g., a correct guess despite poor retrieval).
A concrete example: Suppose a user asks, “What causes X disease, and how does it differ from Y?” A single-step system might retrieve general medical articles, missing specific distinctions. If 3/5 documents are irrelevant, the generated answer might be incorrect. In a multi-step system, the first retrieval could focus on disease causes, and the second on differentiation. If the first step retrieves 4/5 relevant documents but the second step fetches 1/5 due to a poorly refined query, the final answer might correctly explain causes but fail contrasts. Tracking both steps reveals the second retrieval as the bottleneck. This granularity helps developers target fixes—like improving query refinement logic—rather than overhauling the entire system. Multi-step evaluation thus adds depth but requires more instrumentation to trace errors through the pipeline.
