Which natural language generation metrics (e.g., BLEU, ROUGE, METEOR) can be used to compare a RAG system’s answers to reference answers, and what are the limitations of these metrics in this context?

To compare a RAG system’s generated answers to reference answers, common natural language generation (NLG) metrics include BLEU, ROUGE, and METEOR. These metrics measure overlap between generated and reference texts using different approaches. BLEU focuses on n-gram precision, ROUGE emphasizes recall (especially for longer sequences), and METEOR incorporates synonym matching and stemming to improve semantic alignment. For example, BLEU might check if technical terms in a RAG-generated answer match the reference, while ROUGE could highlight whether key facts from the reference are included. METEOR, by allowing synonyms, might better handle paraphrased answers that convey the same meaning with different wording.

However, these metrics have notable limitations when applied to RAG systems. First, they rely on surface-level text overlap and struggle with semantic equivalence. For instance, if a RAG answer rephrases a reference answer using different sentence structures or synonyms, BLEU and ROUGE may undervalue it despite its correctness. Similarly, METEOR’s reliance on predefined synonym lists (like WordNet) can miss domain-specific terminology. Second, these metrics assume a single “correct” reference answer, which is unrealistic for many RAG use cases where multiple valid responses exist. For example, a question like “What causes climate change?” could have multiple correct but distinct answers (e.g., emphasizing CO2 emissions vs. deforestation). Metrics like BLEU might penalize a valid RAG answer simply because it prioritizes different aspects than the reference.

Finally, these metrics fail to capture factual accuracy or coherence, which are critical for RAG systems. A generated answer might have high BLEU/ROUGE scores but include factual errors if the retrieved data is incorrect. For example, a RAG system might generate “The capital of France is Berlin,” which matches the reference’s structure but is factually wrong—yet BLEU would still reward the n-gram overlap. Additionally, these metrics don’t assess fluency or logical flow, meaning a high-scoring answer could still be confusing or poorly structured. Developers should use these metrics cautiously, supplementing them with human evaluation or task-specific checks (e.g., fact verification) to address their blind spots.