What modifications to the prompt or system might be needed to handle multi-step retrieval (such as giving the model the ability to output a follow-up question), and how to evaluate this new ability?

To handle multi-step retrieval, the system must be modified to support iterative interactions and context tracking. First, prompts should explicitly instruct the model to identify gaps in information and generate follow-up questions when needed. For example, a prompt might include templates like, “If the query lacks details, ask one clarifying question to narrow the scope.” The model’s output format could be adjusted to separate answers from follow-up queries (e.g., using delimiters like ## Follow-up:). Additionally, the system needs state management to retain context across steps. This could involve appending previous interactions to the prompt or using a session-based cache. For instance, a travel assistant might first answer a user’s question about flights, then ask, “Do you need hotel recommendations for the same dates?” without requiring the user to repeat dates.

Evaluating this capability requires metrics that assess both the relevance of follow-up questions and their impact on task success. Human evaluators can rate whether follow-up questions are logical (e.g., a medical chatbot asking about symptoms after a user mentions “headache”). Automated tests could measure if follow-up interactions improve answer accuracy compared to single-step retrieval. For example, if a user asks, “How do I fix my computer?” and the model responds, "What error message are you seeing?", the system should track whether the second-step answer (after the user provides the error) resolves the issue. Task completion rate and reduction in user clarification requests (e.g., “Can you be more specific?”) are practical success indicators.

Developers should implement these changes incrementally. Start by adding a single follow-up step to the prompt and test with real-world scenarios, such as troubleshooting technical issues. For instance, a user query like “My app crashes” could trigger a follow-up question about the OS version. Tools like unit tests can validate if the follow-up logic fires correctly, while A/B testing can compare user satisfaction between single-step and multi-step flows. To avoid overcomplicating the system, set limits on the number of follow-ups (e.g., max two questions per query) and ensure the model prioritizes critical information gaps. Monitoring tools should log cases where follow-ups fail to improve outcomes, allowing iterative refinement.