What are some methods to obtain ground truth for which document or passage contains the answer to a question (e.g., using annotated datasets like SQuAD which point to evidence)?

To obtain ground truth data for identifying which document or passage contains an answer to a question, the most common approach is leveraging annotated datasets. These datasets explicitly map questions to specific text segments, enabling models to learn patterns for answer retrieval. For example, the SQuAD dataset contains over 100,000 question-answer pairs, each linked to a passage in Wikipedia articles. Annotators manually highlighted the exact text span answering each question, creating a reliable reference. Similar datasets like TriviaQA and Natural Questions use slightly different annotation methods—TriviaQA includes distant supervision by aligning questions with web pages containing answers, while Natural Questions uses real Google search queries paired with human-annotated answers from Wikipedia. These datasets provide a standardized way to train and evaluate models by offering clear, verified examples of question-to-passage relationships.

Another method involves manual annotation for custom use cases. When existing datasets don’t align with a project’s domain (e.g., legal documents or medical records), teams often create their own ground truth. This process typically involves domain experts or trained annotators labeling documents or passages that answer predefined questions. Tools like Prodigy, Label Studio, or even custom scripts can streamline this workflow. For instance, annotators might review a set of technical support tickets and mark which sections address a user’s issue. To ensure quality, teams use metrics like inter-annotator agreement (measuring consistency between annotators) and iterative refinement of annotation guidelines. While time-consuming, this approach ensures the ground truth aligns with specific requirements, such as industry jargon or document formats not covered in public datasets.

A third approach combines automated heuristics with human validation. For example, keyword matching or retrieval models like BM25 can pre-select candidate passages, which are then reviewed by humans for accuracy. In the legal domain, a tool might search for statute names or case references in a corpus and flag relevant paragraphs for verification. Similarly, embedding-based methods (e.g., using sentence transformers) can rank passages by semantic similarity to a question, and humans confirm the top results. This hybrid method reduces manual effort while maintaining reliability. For instance, the MS MARCO dataset uses Bing search results as candidate answers, which are then refined by human annotators. These strategies balance scalability and accuracy, making them practical for projects where fully manual annotation is infeasible.