What does it indicate if a RAG system’s retriever achieves high recall@5, but the end-to-end question answering accuracy remains low?

If a RAG system’s retriever achieves high recall@5 but end-to-end question answering (QA) accuracy remains low, it suggests a disconnect between the retriever’s ability to fetch relevant documents and the generator’s capacity to synthesize accurate answers from them. Recall@5 measures how often at least one correct document appears in the top five retrieved results. High recall here means the retriever is successfully identifying relevant context. However, low QA accuracy indicates the generator either struggles to interpret the retrieved content, the retrieved data lacks sufficient detail, or the system fails to prioritize the most critical information within those documents. The root issue likely lies in the interaction between retrieval and generation components, rather than the retriever alone.

One common reason is poor alignment between retrieved content and the generator’s requirements. For example, the retriever might return passages that are broadly related to the query but lack specific details needed for precise answers. Imagine a user asks, “What causes lithium-ion batteries to degrade?” The retriever fetches five documents discussing battery chemistry, but only one briefly mentions “solid electrolyte interface growth” as a degradation factor. If the generator isn’t trained to identify and emphasize that detail, it might produce a vague or incorrect answer. Additionally, noisy or redundant information in the top five results could overwhelm the generator, leading it to conflate concepts or prioritize less relevant details. This highlights the importance of not just retrieving relevant documents but ensuring they contain clear, concise, and complementary information.

To address this, developers should first verify the quality of retrieved content. Tools like manual sampling or automated relevance scoring can identify whether the top-five documents consistently contain answer-worthy passages. Next, evaluate the generator’s performance when given explicitly correct context. If accuracy improves, the issue lies in retrieval precision or ranking (e.g., correct documents are present but buried in noise). If accuracy remains low, the generator may need fine-tuning on tasks that require synthesizing multi-document inputs or extracting fine-grained details. Adjusting the number of documents passed to the generator (e.g., using top-2 instead of top-5) or implementing a re-ranker to prioritize the most answer-rich passages could also bridge the gap between retrieval and generation.