How does LlamaIndex perform full-text search?

LlamaIndex performs full-text search by leveraging a combination of keyword-based indexing and retrieval mechanisms, designed to efficiently locate relevant text data. At its core, it uses a keyword table index structure, which maps keywords extracted from documents to the nodes (chunks of text) where those keywords appear. When a user submits a query, LlamaIndex breaks the query into keywords, retrieves matching nodes from the index, and ranks results based on keyword relevance. This approach balances speed and accuracy, making it suitable for applications like document search or question-answering systems.

The process starts with index creation. During indexing, LlamaIndex parses input documents into smaller nodes (e.g., sentences or paragraphs) and extracts keywords from each node. For example, a node containing the text “Python supports object-oriented programming” might generate keywords like “Python,” “object-oriented,” and “programming.” These keywords are stored in a lookup table, associating each keyword with the nodes that contain it. Developers can customize keyword extraction rules, such as ignoring common stopwords (e.g., “the,” “and”) or using NLP libraries to identify domain-specific terms. This preprocessing ensures the index is optimized for fast retrieval.

During query execution, LlamaIndex splits the search input (e.g., “How does Python handle OOP?”) into keywords like “Python” and “OOP.” The system then fetches all nodes linked to these keywords from the keyword table. To improve relevance, LlamaIndex may apply additional filters, such as checking for exact phrase matches or using TF-IDF scoring to prioritize nodes where keywords appear frequently. Optionally, it can integrate with vector stores (e.g., FAISS) to combine keyword search with semantic similarity. For instance, a hybrid approach might first retrieve nodes via keywords, then rerank them using vector embeddings to surface results that contextually align with the query. This flexibility allows developers to tailor the search pipeline to their needs, balancing precision and computational efficiency.