How does fine-tuning a model on a domain (so it “knows” a lot of the answers) compare to using an external retrieval system for that domain? What evaluation would highlight the differences (like evaluating on questions outside the fine-tuned knowledge)?

Fine-tuning a model for a specific domain and using an external retrieval system serve distinct purposes, each with trade-offs. Fine-tuning trains a model on domain-specific data, embedding knowledge directly into its parameters, enabling it to generate answers without external lookups. In contrast, a retrieval system dynamically fetches relevant information from a structured database or document corpus, allowing the model to incorporate up-to-date or broader knowledge. The key difference lies in where knowledge is stored: fine-tuning internalizes it, while retrieval relies on external access. For example, a medical chatbot fine-tuned on clinical notes might answer common diagnoses fluently, whereas a retrieval-augmented system could pull the latest drug guidelines from a maintained database.

To evaluate the differences, focus on scenarios where knowledge scope, accuracy, and adaptability matter. For instance, test both approaches on (1) in-domain questions the model was fine-tuned on, (2) out-of-domain or newer questions beyond its training data, and (3) dynamic content requiring real-time updates. A fine-tuned model may excel on in-domain queries (e.g., diagnosing common illnesses from historical data) but fail on newer topics (e.g., post-2023 treatment protocols) or highly specific edge cases not in its training set. A retrieval system, however, could handle newer or niche queries if its external data source is updated, but might struggle with synthesizing complex answers from retrieved snippets. Metrics like accuracy, latency, and response consistency across these categories would highlight trade-offs. For example, fine-tuning might yield faster inference but degrade on time-sensitive tasks, while retrieval adds latency but maintains accuracy for evolving knowledge.

Specific evaluation examples could include testing a legal advice system. A fine-tuned model trained on past case law might answer general questions effectively but fail on recent court rulings or jurisdiction-specific nuances. A retrieval system paired with a legal database could cite newer precedents but might produce less coherent answers if retrieval results are fragmented. Measuring precision (correctness of answers) and recall (ability to address diverse queries) would reveal gaps. Additionally, stress-testing with ambiguous or multi-hop questions (e.g., “How does X law apply to scenario Y in 2024?”) could show whether the fine-tuned model hallucinates due to knowledge gaps, while the retrieval system might miss context without precise document matching. These tests underscore that fine-tuning prioritizes speed and coherence within known data, while retrieval offers flexibility at the cost of complexity and dependency on external data quality.