How does LlamaIndex manage document metadata?

LlamaIndex manages document metadata by allowing developers to attach custom information to documents and their individual chunks, then leveraging this data for organized storage and targeted retrieval. Metadata can include details like document titles, authors, timestamps, or application-specific tags (e.g., “legal” or “technical”). This metadata is stored alongside the document content and embeddings during indexing, enabling queries to filter or prioritize results based on these attributes. For example, a document loaded via SimpleDirectoryReader might have metadata like source="contracts/2023" or category="agreement" added programmatically before indexing.

During indexing, LlamaIndex processes metadata in two primary ways. First, metadata is preserved at both the document level and the chunk level (when documents are split into smaller sections). For instance, a PDF document split into paragraphs might retain the original file’s author metadata while adding chunk-specific details like section_number=2. Second, LlamaIndex structures metadata in a way that integrates with its storage systems, such as vector databases or in-memory indices. This allows metadata to be efficiently queried alongside semantic content. Developers can customize how metadata is handled—for example, by excluding certain fields from embedding generation or defining which fields to index for fast filtering.

When querying, metadata acts as a filter to narrow search results. Using QueryEngine tools, developers can specify conditions like metadata_filter={"source": "research_papers"} to retrieve only documents from a specific source. Advanced use cases might combine metadata with hybrid search—for example, finding text semantically related to “data privacy” in documents tagged department="legal" and year>=2022. LlamaIndex’s API supports operators (e.g., equality, ranges) and logical combinations (AND/OR) for granular filtering. This approach reduces irrelevant results and improves performance by limiting the search space to contextually relevant data.