How do I handle document metadata in Haystack?

Handling document metadata in Haystack involves managing additional information attached to documents, such as titles, authors, dates, or custom tags. In Haystack, metadata is stored as key-value pairs within the meta attribute of a Document object. This allows developers to associate structured data with unstructured text content, which can be leveraged for filtering, enhancing search relevance, or organizing results. For example, a document could include metadata like {"author": "Jane Doe", "category": "research", "publish_date": "2023-05-01"} to enable queries filtered by author or category.

When working with metadata, the first step is attaching it to documents during ingestion. You can create a Document object with content and metadata, then store it in a Haystack-compatible document store like Elasticsearch or Weaviate. For instance, Document(content="A study on AI...", meta={"source": "arxiv.org", "year": 2022}) creates a document with text content and source/year metadata. Document stores automatically index metadata fields, making them queryable. However, some stores require explicit field mappings. For example, Elasticsearch needs predefined mappings for metadata fields to ensure correct data types (e.g., dates vs. text), while Weaviate handles dynamic types more flexibly. Developers should verify their document store’s requirements to avoid indexing issues.

Metadata becomes powerful when used in retrieval pipelines. For example, you can use a FilterRetriever to fetch documents where meta["year"] >= 2020, or combine keyword search with metadata filters using Elasticsearch’s query DSL. In pipelines, metadata can also influence ranking—for instance, boosting results from specific sources. Here’s a code snippet using metadata filtering with an ElasticsearchRetriever:

retriever = ElasticsearchRetriever(document_store)
results = retriever.retrieve(
 query="machine learning",
 filters={"year": 2023, "source": "arxiv.org"}
)

This retrieves documents containing “machine learning” published in 2023 on arXiv. Metadata can also be passed to subsequent pipeline components, like using author information in a prompt for a generative QA step. Always validate metadata consistency during ingestion to ensure reliable filtering and avoid unexpected gaps in query results.