What are “custom retrieval-based metrics” one could use for RAG evaluation? (Think of metrics that check if the answer contains information from the retrieved text or if all answer sentences can be found in sources.)

Custom retrieval-based metrics for RAG evaluation focus on measuring how well a generated answer aligns with the retrieved sources. These metrics ensure the answer is grounded in the provided documents and avoid unsupported claims. They typically analyze the overlap between the generated text and the retrieved content, or verify that every part of the answer can be traced back to a source. For developers, these metrics help identify issues like hallucinations (answers with fabricated information) or incomplete use of relevant context.

One common approach is sentence-level attribution. This involves checking if each sentence in the generated answer can be matched to a sentence in the retrieved documents. For example, you might split the answer into individual sentences and use exact string matching, keyword overlap, or embeddings (e.g., cosine similarity with Sentence-BERT) to compare them to the sources. Another metric is coverage, which measures the percentage of retrieved documents that were actually used in the answer. If the system retrieves 10 documents but only uses 2, coverage would be low, signaling inefficiency. Tools like ROUGE-L or BLEU can also be adapted to evaluate n-gram overlap between the answer and sources, though these may miss paraphrased or reordered content.

More advanced methods include fact verification and contextual consistency checks. For instance, a metric could extract factual claims (e.g., dates, names) from the answer and validate them against the sources using named entity recognition (NER) and rule-based matching. Alternatively, you could train a classifier to predict whether an answer sentence is supported by the retrieved context, using labeled data. For nuanced evaluation, combining multiple metrics works best: exact matches for critical facts, semantic similarity for paraphrased content, and coverage to ensure thorough use of sources. Libraries like spaCy (for sentence splitting) or Hugging Face’s evaluate can streamline implementation, letting developers test these metrics without reinventing the wheel.