How BLEU is Calculated and Its Correlation with Factual Correctness
BLEU (Bilingual Evaluation Understudy) is a metric used to evaluate the quality of machine-generated text (e.g., machine translation or summarization) by comparing it to reference texts written by humans. It works by measuring n-gram overlap between the candidate and reference texts. Here’s a simplified breakdown:
- N-gram Precision: BLEU calculates precision for n-grams (contiguous word sequences) of varying lengths (typically 1- to 4-grams). For example, if a candidate sentence shares 3 out of 4 unique 4-grams with the reference, its 4-gram precision is 0.75.
- Modified Precision: To avoid overcounting repeated n-grams, BLEU clips the count of each n-gram in the candidate to its maximum count in any reference.
- Brevity Penalty: This penalizes overly short candidates. If the candidate length is shorter than the reference, the score is reduced exponentially. The final BLEU score is a weighted geometric mean of these n-gram precisions, multiplied by the brevity penalty, and scaled between 0 (no overlap) and 1 (perfect match).
Does a Higher BLEU Score Mean More Factually Correct Answers? No. BLEU measures lexical and structural similarity, not factual correctness. For example:
- A candidate answer could have high n-gram overlap with references but contain factual errors. Suppose the reference states, “The moon orbits Earth in 27 days,” and the candidate says, “The moon orbits Mars in 27 days.” The 4-gram overlap (“moon orbits … in 27 days”) would yield a high BLEU score despite the factual error.
- Conversely, a paraphrased answer with correct facts but different wording (e.g., “Earth’s satellite completes its orbit every 27 days”) might score lower due to lexical divergence. BLEU is blind to semantics and context. It treats all n-grams equally, whether they represent critical facts or trivial phrases.
When to Use BLEU (and When Not To) BLEU is useful for quick, automated comparisons of text similarity, especially in tasks like translation where phrasing matters. Developers often use it to benchmark model iterations. However, for applications requiring factual accuracy (e.g., medical summaries or technical documentation), BLEU should be supplemented with:
- Fact-checking tools: To validate claims.
- Semantic metrics: Like ROUGE (for content overlap) or BERTScore (for contextual embeddings).
- Human evaluation: To assess correctness and coherence. In short, BLEU is a tool for measuring “how similar,” not “how correct.”
[Reference] While the provided sources discuss general concepts like “metric” and “calculate,” they do not directly address BLEU. This explanation is based on standard NLP literature and best practices.
