If a RAG system’s answers are poor, how can we determine whether the fault lies with retrieval or generation? (Hint: evaluate retrieval accuracy separately with metrics like recall@K.)

To determine whether a RAG system’s poor performance stems from retrieval or generation, start by isolating and evaluating each component separately. First, assess retrieval accuracy using metrics like recall@K (the proportion of relevant documents retrieved in the top K results) or precision@K (how many of the top K documents are relevant). For example, if a user asks, “What causes climate change?” and the system retrieves documents about weather patterns instead of greenhouse gases, the recall@5 score would be low. This indicates retrieval is failing to surface correct context, directly impacting the generator’s ability to produce accurate answers. Tools like labeled datasets or manual checks can validate whether retrieved documents align with the query’s intent. If retrieval consistently underperforms, improving embedding models, adjusting chunk sizes, or refining search algorithms may be necessary.

Next, test the generation component by providing it with manually curated, relevant documents and evaluating the output quality. If the generator still produces incorrect or incoherent answers despite good input, the issue likely lies here. For instance, if the system correctly retrieves a document stating, “CO2 emissions are the primary driver of climate change,” but the generated answer says, “Methane is the main cause,” the generator is misinterpreting or hallucinating. This can be measured using metrics like BLEU score (for text similarity) or factual accuracy checks. Common fixes include fine-tuning the language model on domain-specific data, adjusting temperature/prompting strategies, or adding post-processing rules to constrain outputs.

Finally, analyze the interaction between retrieval and generation. Even with decent recall@K, poor ranking (e.g., burying critical documents at position 5 in a K=5 setup) can lead the generator to prioritize less relevant text. Tools like attention heatmaps or saliency scores can reveal whether the generator is focusing on the right parts of retrieved content. For example, if a medical RAG system retrieves correct drug dosage guidelines but the generator overlooks a key sentence about contraindications, the problem may involve both ranking and generation. Iteratively refining both components—such as re-ranking retrieved documents or adding context-weighting in prompts—can address these gaps. By systematically isolating and stress-testing each stage, developers can pinpoint root causes and allocate resources effectively.