How can we incorporate user feedback or real user queries into building a dataset for RAG evaluation, and what are the challenges with using real-world queries?

To incorporate user feedback or real queries into a RAG evaluation dataset, start by collecting data from actual interactions with your application. For example, if you have a customer support chatbot, log user questions, responses provided by the system, and explicit feedback (e.g., thumbs-up/down ratings). Implicit feedback, like users rephrasing a query after an unsatisfactory answer, can also signal gaps in the model’s performance. Tools like session recording or API logs can automate this collection. Once gathered, anonymize the data to remove personally identifiable information (PII) and filter out irrelevant or low-quality entries (e.g., spam). Categorize the queries by intent or topic to ensure balanced coverage—for instance, grouping medical FAQ queries separately from technical troubleshooting requests. This raw data becomes the foundation for testing how well your RAG system handles real-world scenarios.

The main challenges with real-world queries are noise, ambiguity, and privacy. Real user inputs often contain typos, slang, or vague phrasing (e.g., “It’s not working” without context), which can confuse evaluation metrics designed for clean data. For example, a query like “fix error 404” might lack details about the specific application or environment, making it hard to assess if the RAG system’s answer is adequate. Privacy is another concern: even anonymized logs might inadvertently expose sensitive patterns, especially in domains like healthcare or finance. Additionally, user behavior shifts over time—seasonal trends or new product features can make older queries obsolete. For instance, a travel app might see a surge in “COVID travel restrictions” queries during a pandemic, but these become irrelevant once policies change, requiring constant dataset updates.

To address these challenges, balance real data with synthetic examples. For instance, augment ambiguous real queries with variations that clarify intent (e.g., expanding “fix error” to “fix login error on Android app”). Use differential privacy techniques or synthetic data generation for sensitive domains. To handle evolving user needs, periodically retest your RAG system with fresh logs and retire outdated test cases. Align evaluation metrics with user satisfaction by measuring not just answer correctness but also relevance and clarity. For example, if users often follow up with “Can you explain that differently?” after certain answers, flag those responses for improvement. By combining curated real-world data with targeted synthetic examples and adaptive evaluation practices, you can build a robust RAG testing framework that reflects actual user needs while mitigating privacy and noise issues.