When evaluating a RAG (Retrieval-Augmented Generation) system, the metrics for retrieval and generation should generally be analyzed both separately and in combination. This dual approach ensures you diagnose weaknesses in each component while also understanding their combined impact. Presenting them separately is critical for debugging: poor generation quality might stem from faulty retrieval, and vice versa. However, aggregating metrics can provide a holistic view of system performance, especially when comparing models or optimizing for specific use cases. The choice depends on the evaluation goals—troubleshooting requires separation, while overall benchmarking benefits from aggregation.
Separate analysis is essential because retrieval and generation serve distinct roles. Retrieval metrics like precision@k (accuracy of top-k retrieved documents) or recall (coverage of relevant content) directly measure how well the system identifies useful information. Generation metrics like BLEU, ROUGE, or BERTScore assess the fluency, relevance, and accuracy of the generated text. For example, if a medical QA system retrieves correct studies (high precision@5) but generates answers with factual errors (low BERTScore), the issue lies in the generator. Conversely, low recall in retrieval could lead to plausible but incorrect answers due to missing context. Keeping metrics separate helps developers pinpoint where to focus improvements—whether fine-tuning the retriever, expanding the document corpus, or adjusting the generator’s prompts.
Aggregation is useful for summarizing performance or prioritizing trade-offs. One approach is weighted scoring: assign a score to retrieval (e.g., 0.7) and generation (e.g., 0.9), then combine them (e.g., 0.8 overall) based on their importance to the task. For instance, a legal research tool might weight retrieval higher (70%) because citing correct precedents is critical, while a chatbot might balance both equally. Another method is task-specific metrics like answer correctness in QA, which inherently combines retrieval and generation. Tools like RAGAS compute composite scores by aligning retrieved content with generated answers. However, aggregation risks masking component-specific issues—combine metrics only after establishing baseline performance for each. For transparency, report both aggregated and individual metrics to balance brevity with diagnostic utility.
