How could you use the LLM itself to improve retrieval — for example, by generating a better search query or re-ranking the retrieved results? How would you measure the impact of such techniques?

Large language models (LLMs) can improve retrieval systems in two key ways: by refining search queries and re-ranking retrieved results. First, LLMs can generate more effective search queries by interpreting a user’s intent and expanding or rephrasing the original query. For example, if a user searches for “best laptops for coding,” an LLM might generate a revised query like “top-rated laptops with high-performance CPUs, 16GB RAM, and long battery life for software development.” This expanded query includes specific technical terms and context that align better with the user’s needs, increasing the likelihood of retrieving relevant documents. Unlike keyword-based approaches, LLMs can infer unstated requirements (e.g., battery life for portability) and adjust the query accordingly.

Second, LLMs can re-rank retrieved results to prioritize the most relevant items. After an initial set of documents is fetched using a traditional retrieval system (like BM25 or embedding-based search), the LLM can analyze each document’s content relative to the query. For instance, if the query is “Python tutorials for data science,” the LLM might score documents higher if they mention libraries like Pandas or NumPy, even if those terms weren’t in the original query. This re-ranking step leverages the LLM’s ability to understand context and semantic relationships. Some implementations use a “cross-encoder” architecture, where the query and document text are processed together to compute a relevance score, enabling finer-grained ranking than simpler similarity metrics.

To measure the impact of these techniques, developers can use offline metrics like precision@k (the proportion of relevant documents in the top k results) or normalized discounted cumulative gain (NDCG), which accounts for the position of relevant items in the ranked list. For example, if re-ranking improves the average precision@10 from 0.4 to 0.6, it indicates a tangible improvement. A/B testing in production can also track user engagement metrics like click-through rates or time spent on pages. Additionally, human evaluators can rate the relevance of results before and after applying LLM-based improvements. Combining these methods provides a comprehensive view of performance gains while ensuring the changes align with real-world user needs.