How would you evaluate the performance of a RAG system over time or after updates? (Consider setting up a continuous evaluation pipeline with key metrics to catch regressions in either retrieval or generation.)

To evaluate a RAG system’s performance over time or after updates, establish a continuous evaluation pipeline that tracks key metrics for retrieval and generation. Start by defining a benchmark dataset with queries, expected retrieved documents, and reference answers. Automate testing by running this dataset through the system after each update and comparing results against baselines. Track retrieval metrics like precision@k (accuracy of top-k retrieved documents), recall@k (coverage of relevant documents), and mean reciprocal rank (MRR) to detect regressions in document relevance. For generation, measure answer quality with metrics like BLEU, ROUGE, or BERTScore against reference answers, and use human evaluation for subjective aspects like coherence or factual correctness. For example, if an update introduces a new embedding model, a drop in precision@5 could indicate retrieval issues, while a decline in BERTScore might point to degraded answer relevance.

Next, implement shadow testing and canary deployments to minimize risk. Shadow testing runs the updated system in parallel with the current version, logging differences in retrieval and generation outputs without affecting users. This helps identify edge cases where new components underperform. Canary deployments gradually roll out updates to a small user subset, monitoring real-world metrics like response latency, error rates, and user feedback. For instance, if a retriever update increases latency by 30%, you can pause the rollout and investigate. Additionally, track domain-specific metrics: a medical RAG system might measure adherence to clinical guidelines, while a customer support tool could monitor resolution rates. Automated anomaly detection (e.g., sudden spikes in “I don’t know” responses) can flag issues early.

Finally, maintain a feedback loop for iterative improvement. Log user interactions (queries, retrieved documents, answers, and thumbs-up/down ratings) to create a growing evaluation dataset. Periodically retrain the system using this data to adapt to new query patterns or knowledge gaps. For example, if users consistently downvote answers about a recent event, expand the document corpus or fine-tune the generator on newer data. Use dashboards (e.g., Grafana) to visualize trends in metrics like retrieval hit rate or answer correctness over weeks or months. If an update causes MRR to drop from 0.85 to 0.72, drill down into whether the retriever is failing on specific query types or document formats. Regularly revisit evaluation benchmarks to ensure they reflect current use cases, and automate regression alerts to maintain system reliability.