What are some known metrics or scores (such as “faithfulness” scores from tools like RAGAS) that aim to quantify how well an answer sticks to the provided documents?

Several metrics and scores exist to quantify how well an answer aligns with provided documents, particularly in retrieval-augmented systems like RAG (Retrieval-Augmented Generation). These metrics focus on factual consistency, relevance, and adherence to source material. Common examples include faithfulness scores (e.g., from RAGAS), answer relevance, and contextual precision/recall. These tools help developers evaluate whether generated answers are grounded in the provided documents and avoid unsupported claims or “hallucinations.”

One widely used metric is faithfulness, which measures whether the generated answer is factually consistent with the source documents. For instance, RAGAS calculates this by comparing claims in the answer against the retrieved context. If an answer states, “The document mentions a 2023 policy change,” but the source only refers to 2022, the faithfulness score drops. Tools like RAGAS often use entailment models or cross-encoders to verify factual alignment. Developers can implement this by running automated checks that flag answers with unsupported claims, ensuring outputs stay true to the input documents.

Another key metric is answer relevance, which evaluates how directly the answer addresses the query while staying within the document’s scope. For example, if a user asks about “climate change impacts on agriculture” and the answer discusses unrelated economic policies, the relevance score would be low. Frameworks like RAGAS or custom pipelines might use semantic similarity models (e.g., SBERT) to compare the answer’s focus with the query and source context. Additionally, contextual precision and recall measure whether the retrieved documents fully cover the answer’s claims (recall) and avoid irrelevant content (precision). For developers, combining these metrics provides a robust way to assess how well answers stick to the documents while maintaining coherence and avoiding extraneous information.