How do I set up and train a retriever in Haystack?

To set up and train a retriever in Haystack, start by installing the library and configuring a document store. Haystack supports retrievers like BM25 (keyword-based) and dense neural models (e.g., Dense Passage Retriever). First, prepare your data by converting documents into Haystack’s Document format, which includes text and metadata. For example, load a CSV or JSON file using Document objects and write them to a document store like Elasticsearch or FAISS. Initialize the retriever by selecting a model—BM25 requires minimal setup, while dense retrievers need a pre-trained transformer model (e.g., facebook/dpr-question_encoder). Configure parameters such as embedding_dim or max_seq_len to match your model.

Training a custom retriever involves fine-tuning a dense model on your dataset. Use Haystack’s DensePassageRetriever and provide a dataset with query-document pairs. For example, if your data includes questions and relevant paragraphs, structure it as a list of dictionaries with "question" and "positive_context" keys. Load the dataset using Dataset and DataLoader, then train the model with retriever.train(). Specify hyperparameters like learning rate (e.g., 1e-5), batch size, and epochs. During training, the model learns to map queries and documents into a shared embedding space where relevant pairs are closer. Monitor metrics like recall@k to evaluate retrieval accuracy.

After training, save the model and integrate it into a Haystack pipeline. For example, combine the retriever with a reader model to build a question-answering system. Test the retriever by running pipeline.run(query="your question") and inspect the returned documents. If performance is lacking, adjust the training data or fine-tune further. For BM25, tweak parameters like top_k to control how many documents are retrieved. Always validate with a holdout dataset to ensure generalization. Haystack’s modular design lets you swap retrievers or combine them (e.g., using an EnsembleRetriever) for improved results.