Recommender systems are typically evaluated using metrics that measure accuracy, ranking quality, and diversity/coverage. These metrics help developers assess how well a system predicts user preferences, surfaces relevant items, and balances recommendations across a catalog. Below are the most common metrics grouped by their primary focus.
Accuracy Metrics Accuracy metrics evaluate how closely predicted recommendations match ground-truth user preferences. For rating-based systems (e.g., movie ratings), Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE) calculate the average deviation between predicted and actual ratings. RMSE penalizes larger errors more heavily. For binary or implicit feedback (e.g., clicks), Precision and Recall are widely used. Precision measures the fraction of recommended items that are relevant (e.g., out of 10 recommendations, 8 were clicked), while Recall measures the fraction of all relevant items that were recommended. The F1-score, the harmonic mean of Precision and Recall, balances the two. These metrics require a labeled test set and are straightforward to compute but may not reflect real-world user behavior, as they ignore item order.
Ranking Metrics Since recommendations are often ordered lists, ranking quality is critical. Normalized Discounted Cumulative Gain (NDCG) evaluates how well a system ranks items by assigning higher scores to relevant items placed at the top. For example, if a user’s most-liked movie appears first, NDCG rewards this more than if it appears fifth. Mean Average Precision (MAP) calculates the average precision at each position where a relevant item occurs, emphasizing correct rankings across multiple queries. Hit Rate (e.g., Hit@10) measures whether at least one relevant item exists in the top-N recommendations. These metrics are ideal for scenarios where item position matters, such as homepage recommendations, but they require relevance thresholds (e.g., defining what constitutes a “hit”).
Diversity and Coverage Beyond accuracy, systems must avoid over-recommending popular items. Diversity measures how dissimilar recommended items are, often using pairwise similarity scores (e.g., cosine similarity between item embeddings) or entropy-based calculations. For example, a diverse movie recommendation list might include genres like action, comedy, and documentary. Coverage quantifies the fraction of the total item catalog recommended to users, ensuring niche or long-tail items aren’t ignored. A low coverage score indicates the system favors a small subset of items, which can harm user satisfaction and business goals. These metrics are particularly important for platforms with large catalogs, such as e-commerce, where discoverability matters. However, they may trade off against accuracy, requiring developers to balance multiple objectives.
By combining these metrics, developers can holistically evaluate recommender systems, ensuring they are accurate, user-friendly, and sustainable.