How can we evaluate factual correctness of an answer when a reference answer is available? (Consider exact match or F1 as used in QA benchmarks like SQuAD.)

To evaluate factual correctness when a reference answer is available, two common metrics are exact match (EM) and F1 score, both widely used in benchmarks like SQuAD. EM checks if the predicted answer exactly matches the reference, including wording, punctuation, and capitalization. For example, if the reference answer is “Paris” and the model outputs “paris” or “Paris, France,” EM would score 0 because of case sensitivity or extra text. F1 score, on the other hand, measures token-level overlap between the predicted and reference answers. It calculates precision (percentage of predicted tokens that are correct) and recall (percentage of reference tokens captured by the prediction), then combines them into a harmonic mean. This allows partial credit for answers that are close but not perfect.

While EM is strict and easy to compute, it can be overly rigid. For instance, if the reference is “the 16th century” and the model answers “16th century,” EM would fail it, even though the core fact is correct. F1 addresses this by breaking the answers into tokens (e.g., ["the", "16th", “century”] vs. ["16th", “century”]). Here, precision is 100% (both predicted tokens are correct), recall is 66% (2 out of 3 reference tokens matched), and F1 is 80%. However, F1 can still struggle with synonyms or rephrased answers. For example, if the reference is “William Shakespeare” and the model outputs “Shakespeare,” F1 would score 67% (precision 100%, recall 50%), even though the answer is factually correct but incomplete.

These metrics are best suited for structured, short-answer scenarios where answers are unambiguous. For example, in a QA system for historical dates, EM and F1 work well because answers like “July 20, 1969” have limited valid variations. However, they fail to capture semantic equivalence in more complex cases. If the reference answer is “France” and the model says “French,” F1 and EM would both score 0, even though the error is subtle. Developers should use these metrics as a baseline but supplement them with human evaluation or semantic similarity tools (e.g., BERTScore) for nuanced tasks. Automated metrics prioritize scalability, while manual review ensures correctness in edge cases.