SPARSE_INVERTED_INDEX

The SPARSE_INVERTED_INDEX index is an index type used by Milvus to efficiently store and search sparse vectors. It builds an inverted structure from the non-zero dimensions in sparse vectors. You can use this index for BM25 full text search and for sparse embedding search based on inner product.

For more information about sparse vector fields, metric types, and full text search, refer to Sparse Vector, Metric Types, and Full Text Search.

Build index

To build a SPARSE_INVERTED_INDEX index on a sparse vector field in Milvus, use the add_index() method and specify index_type, metric_type, and index parameters.

For BM25 full text search, build the index on the sparse vector field generated by a BM25 function. Set metric_type to BM25.

from pymilvus import MilvusClient

client = MilvusClient(uri="http://localhost:19530")

# Prepare index building params
index_params = client.prepare_index_params()
index_params.add_index(
    field_name="sparse", # Name of the sparse vector field to index
    index_type="SPARSE_INVERTED_INDEX", # Type of the index to create
    index_name="sparse_bm25_index", # Name of the index to create
    metric_type="BM25", # Metric type used for full text search
    params={"inverted_index_algo": "DAAT_MAXSCORE"},
)

client.create_index(
    collection_name="your_collection_name",
    index_params=index_params,
)

For sparse embedding search, build the index on a sparse vector field that stores externally generated sparse vectors. Set metric_type to IP.

# Prepare index building params
index_params = client.prepare_index_params()
index_params.add_index(
    field_name="sparse_vector", # Name of the sparse vector field to index
    index_type="SPARSE_INVERTED_INDEX", # Type of the index to create
    index_name="sparse_ip_index", # Name of the index to create
    metric_type="IP", # Metric type used to measure similarity
    params={"inverted_index_algo": "SINDI"},
)

client.create_index(
    collection_name="your_collection_name",
    index_params=index_params,
)

In the preceding configurations:

• index_type: The type of index to build. Set this value to SPARSE_INVERTED_INDEX.

• metric_type: The metric used to calculate similarity between sparse vectors. Valid Values:

  • BM25: Uses BM25 relevance scoring for full text search.

  • IP (Inner Product): Measures sparse vector similarity using dot product.

  For details, refer to Metric Types and Full Text Search.

• params.inverted_index_algo: The algorithm used for building and querying the index. Valid values:

  • "DAAT_MAXSCORE": Document-at-a-Time MaxScore query processing. This is the default for BM25. For background, refer to Query Evaluation: Strategies and Optimizations.

  • "DAAT_WAND": Document-at-a-Time WAND query processing. This algorithm is suitable for smaller topK values or shorter queries. For background, refer to Efficient Query Evaluation using a Two-Level Retrieval Process.

  • "TAAT_NAIVE": Basic Term-at-a-Time query processing. Use this option as a baseline or when you need scoring to adapt dynamically to global collection statistics such as average document length.

  • "BLOCK_MAX_MAXSCORE": MaxScore query processing with block-level max-score metadata. For background, refer to Faster Top-k Document Retrieval Using Block-Max Indexes.

  • "BLOCK_MAX_WAND": WAND query processing with block-level max-score metadata. For background, refer to Faster Top-k Document Retrieval Using Block-Max Indexes.

  • "SINDI": A sparse inverted index based on fixed document-id windows, with SIMD acceleration for search. This is the default for IP. For details, refer to the SINDI paper.

  If you do not specify inverted_index_algo, Milvus selects the default algorithm based on metric_type: DAAT_MAXSCORE for BM25, and SINDI for IP.

  To learn more building parameters available for the SPARSE_INVERTED_INDEX index, refer to Index building params.

Once the index parameters are configured, you can create the index by using the create_index() method directly or passing the index params in the create_collection method. For details, refer to Create Collection.

Search on index

Once the index is built and entities are inserted, you can perform similarity searches on the index.

For BM25 full text search, use raw text as the query. Milvus converts the query text into a sparse vector through the BM25 function.

res = client.search(
    collection_name="your_collection_name",
    data=["what is information retrieval?"],
    anns_field="sparse",
    output_fields=["text"],
    limit=3,
)

For sparse embedding search, use a sparse-vector dictionary as the query vector.

# Prepare the query vector
query_vector = [{1: 0.2, 50: 0.4, 1000: 0.7}]

res = client.search(
    collection_name="your_collection_name",
    anns_field="sparse_vector",
    data=query_vector,
    limit=3,
    search_params={"metric_type": "IP"},
)

By default, Milvus uses the search algorithm configured for the index.

To learn more search parameters available for the SPARSE_INVERTED_INDEX index, refer to Index-specific search params.

Index params

This section provides an overview of the parameters used for building an index and performing searches on the index.

Index building params

The following table lists the parameters that can be configured in params when building an index.

Parameter	Description	Value Range	Tuning Suggestion
inverted_index_algo	The algorithm used for building and querying the index. It determines how the index processes queries.	"DAAT_MAXSCORE", "DAAT_WAND", "TAAT_NAIVE", "BLOCK_MAX_MAXSCORE", "BLOCK_MAX_WAND", "SINDI" Default value: "DAAT_MAXSCORE" for BM25; "SINDI" for IP.	Use "DAAT_MAXSCORE" for BM25 full text search workloads with high k values or queries with many terms. Use "DAAT_WAND" for BM25 workloads with small k values or short queries. Use "TAAT_NAIVE" as a baseline, or when you need scoring to adapt dynamically to global collection statistics such as average document length. Use "BLOCK_MAX_MAXSCORE" or "BLOCK_MAX_WAND" to use block-level max-score metadata for query pruning. Use "SINDI" for sparse embedding search with IP.
bm25_k1	Controls term frequency saturation for BM25 scoring. This parameter applies only when metric_type is BM25.	Recommended range: [1.2, 2.0] Default value: 1.2	Increase this value to give term frequency more weight in document ranking.
bm25_b	Controls the strength of document length normalization for BM25 scoring. This parameter applies only when metric_type is BM25.	Range: [0, 1] Default value: 0.75	Use a higher value to apply stronger length normalization. Use a lower value to reduce the effect of document length on ranking.

Index-specific search params

The following table lists the parameters that can be configured in search_params.params when searching on the index.

Parameter	Description	Value Range	Tuning Suggestion
drop_ratio_search	The proportion of the smallest values to ignore during search, helping to reduce noise.	Range: [0.0, 1.0) (for example, 0.2 ignores the smallest 20% of values)	Tune this parameter based on the sparsity and noise level of your query vectors. This parameter controls the proportion of low-magnitude values dropped during search. Increasing this value (for example, to 0.2) can reduce noise and focus the search on more significant components, which may improve precision and efficiency. However, dropping more values can also reduce recall by excluding potentially relevant signals. Choose a value that balances recall and accuracy for your workload.