Recommender systems are typically evaluated using metrics that measure accuracy, ranking quality, and business impact. The choice of metrics depends on the system’s goals, such as predicting user ratings, generating personalized item lists, or driving user engagement. Below are three categories of widely used metrics.
Accuracy Metrics: These assess how closely recommendations match user preferences. For rating prediction tasks (e.g., predicting a 1–5 star score), Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE) are common. MAE calculates the average absolute difference between predicted and actual ratings, while RMSE penalizes larger errors more heavily. For example, if a movie recommender predicts a 4-star rating for a film the user actually rates as 3 stars, the MAE contribution is 1.0. In top-N recommendation scenarios (e.g., suggesting a list of products), Precision and Recall measure relevance. Precision@10 calculates the fraction of recommended items that are relevant (e.g., 3 out of 10 items clicked by the user), while Recall@10 measures the fraction of all relevant items captured in the recommendations.
Ranking Metrics: These evaluate the order of recommended items. Normalized Discounted Cumulative Gain (NDCG) rewards placing relevant items higher in the list, with a logarithmic discount for lower positions. For instance, a search engine ranking documents would score higher if the most relevant result appears first. Mean Reciprocal Rank (MRR) focuses on the position of the first relevant item—for example, if the first correct answer in a QA system appears in position 3, the reciprocal rank is 1/3. Hit Rate (e.g., Hit@10) simply checks if at least one relevant item exists in the top-N recommendations, useful for scenarios like news feeds where surfacing any engaging content matters.
Beyond-Accuracy Metrics: These address broader goals like diversity, coverage, or fairness. Diversity measures how varied the recommended items are, often calculated using intra-list similarity (e.g., ensuring a music playlist includes multiple genres). Coverage quantifies the fraction of the catalog recommended to users, which helps avoid over-reliance on popular items. For example, a book recommender with 80% coverage suggests most titles in the inventory, reducing bias toward bestsellers. Business metrics like Click-Through Rate (CTR) or Conversion Rate are also critical for real-world systems, though they require A/B testing. Developers often balance these metrics—for instance, optimizing for NDCG might reduce coverage, requiring trade-offs based on the application’s priorities.