What are the standard evaluation metrics in IR?

Standard evaluation metrics in Information Retrieval (IR) measure how effectively a system retrieves relevant information. The most common metrics include Precision, Recall, F1 Score, Mean Average Precision (MAP), and Normalized Discounted Cumulative Gain (NDCG). These metrics address different aspects of performance: Precision and Recall focus on binary relevance (whether items are relevant or not), while MAP and NDCG evaluate ranked lists, considering the order of results. Each metric provides insights into trade-offs between returning as many relevant items as possible and minimizing irrelevant ones.

Precision measures the fraction of retrieved items that are relevant. For example, if a search returns 5 documents and 3 are relevant, Precision is 3/5 (60%). Recall quantifies the fraction of all relevant items retrieved. If there are 10 relevant documents in total and the system retrieves 3, Recall is 3/10 (30%). The F1 Score balances these two using their harmonic mean, which is useful when you need a single metric to compare systems. For instance, if Precision is 60% and Recall is 30%, F1 is 2(0.60.3)/(0.6+0.3) ≈ 40%. These metrics are straightforward but limited to binary judgments (relevant/not relevant) and don’t account for ranking order.

For ranked results, MAP and NDCG are more informative. MAP calculates the average precision across multiple queries, where precision is computed at each position a relevant item is found. For example, if the first relevant result is at position 3, precision at that point is 1/3. MAP averages these values across all queries, rewarding systems that place relevant items higher. NDCG evaluates ranking quality using graded relevance (e.g., scores like 0, 1, 2). It compares the system’s ranking to an ideal order, applying discounts for lower positions. If a search ranks documents with relevance scores [3, 2, 1] as [2, 3, 1], the DCG (summing scores divided by log position) is compared to the ideal DCG, yielding NDCG. This metric is useful when relevance isn’t binary and order matters, such as in recommendation systems. Together, these metrics help developers optimize systems for both accuracy and user experience.