Evaluation
Recommendation evaluation measures whether recommended items are accurate, useful, diverse, and valuable to users.
Prediction Accuracy
For rating prediction:
- MAE measures average absolute error.
- RMSE penalizes larger errors more strongly.
Ranking Metrics
Ranking metrics evaluate the ordered list of recommendations.
- Precision@K: fraction of top K recommendations that are relevant.
- Recall@K: fraction of all relevant items found in top K.
- MAP: mean Average Precision across users.
- NDCG: ranking quality with position discounting.
- MRR: reciprocal rank of the first relevant item.
Beyond Accuracy
Accuracy is not enough. A recommender may be accurate but boring.
Other useful measures:
- coverage: percentage of items that can be recommended
- diversity: how different the recommended items are
- serendipity: useful unexpectedness
- novelty: how new or unknown items are to the user
These metrics connect directly to Recommendation System Challenges because production recommenders often optimize tradeoffs, not one metric.
Challenges
Recommendation systems face data, modeling, scale, and product-quality challenges.
Cold Start
New users have little or no interaction history. Possible solutions include asking for preferences, using demographic or contextual data, and recommending popular items.
New items have no interaction history. Content-Based Filtering can help by using metadata or item features.
Data Sparsity
Most users interact with only a small fraction of items. The user-item matrix is sparse.
Possible solutions:
- Matrix Factorization
- transfer learning
- hybrid methods
Scalability
Millions of users and items make exact similarity computation expensive.
Possible solutions:
- approximate nearest neighbors
- locality-sensitive hashing
- precomputation
- caching
- distributed computing
Diversity Vs Accuracy
Highly accurate recommendations can become repetitive.
Possible solutions:
- penalize overly similar items
- inject exploration
- optimize multiple objectives
Filter Bubble
A filter bubble happens when users only see content similar to past behavior.
Possible solutions:
- exploration vs exploitation strategies
- serendipity metrics
- diversity-promoting algorithms