Full Text Search​

Full text search is a feature that retrieves documents containing specific terms or phrases in text datasets, then ranking the results based on relevance. This feature overcomes semantic search limitations, which might overlook precise terms, ensuring you receive the most accurate and contextually relevant results. Additionally, it simplifies vector searches by accepting raw text input, automatically converting your text data into sparse embeddings without the need to manually generate vector embeddings.​

Using the BM25 algorithm for relevance scoring, this feature is particularly valuable in retrieval-augmented generation (RAG) scenarios, where it prioritizes documents that closely match specific search terms.​

Overview​

Full text search simplifies the process of text-based searching by eliminating the need for manual embedding. This feature operates through the following workflow:​

1. Text input: You insert raw text documents or provide query text without any need for manual embedding.​

2. Text analysis: Milvus uses an analyzer to tokenize input text into individual, searchable terms.​

3. Function processing: The built-in function receives tokenized terms and converts them into sparse vector representations.​

4. Collection store: Milvus stores these sparse embeddings in a collection for efficient retrieval.​

5. BM25 scoring: During a search, Milvus applies the BM25 algorithm to calculate scores for the stored documents and ranks matched results based on relevance to the query text.​

Full text search

To use full text search, follow these main steps:​

1. Create a collection: Set up a collection with necessary fields and define a function to convert raw text into sparse embeddings.​

2. Insert data: Ingest your raw text documents to the collection.​

3. Perform searches: Use query texts to search through your collection and retrieve relevant results.​

Create a collection for full text search​

To enable full text search, create a collection with a specific schema. This schema must include three necessary fields:​

• The primary field that uniquely identifies each entity in a collection.​

• A VARCHAR field that stores raw text documents, with the enable_analyzer attribute set to True. This allows Milvus to tokenize text into specific terms for function processing.​

• A SPARSE_FLOAT_VECTOR field reserved to store sparse embeddings that Milvus will automatically generate for the VARCHAR field.​

Define the collection schema

First, create the schema and add the necessary fields:​

from pymilvus import MilvusClient, DataType, Function, FunctionType​
​
schema = MilvusClient.create_schema()​
​
schema.add_field(field_name="id", datatype=DataType.INT64, is_primary=True, auto_id=True)​
schema.add_field(field_name="text", datatype=DataType.VARCHAR, max_length=1000, enable_analyzer=True)​
schema.add_field(field_name="sparse", datatype=DataType.SPARSE_FLOAT_VECTOR)​

In this configuration,​

• id: serves as the primary key and is automatically generated with auto_id=True.​

• text: stores your raw text data for full text search operations. The data type must be VARCHAR, as VARCHAR is Milvus’ string data type for text storage. Set enable_analyzer=True to allow Milvus to tokenize the text. By default, Milvus uses the standard analyzer for text analysis. To configure a different analyzer, refer to ​Overview.​

• sparse: a vector field reserved to store internally generated sparse embeddings for full text search operations. The data type must be SPARSE_FLOAT_VECTOR.​

Now, define a function that will convert your text into sparse vector representations and then add it to the schema:​

bm25_function = Function(​
    name="text_bm25_emb", # Function name​
    input_field_names=["text"], # Name of the VARCHAR field containing raw text data​
    output_field_names=["sparse"], # Name of the SPARSE_FLOAT_VECTOR field reserved to store generated embeddings​
    function_type=FunctionType.BM25,​
)​
​
schema.add_function(bm25_function)​

Parameter​	Description​
name​ ​	The name of the function. This function converts your raw text from the text field into searchable vectors that will be stored in the sparse field.​
input_field_names​	The name of the VARCHAR field requiring text-to-sparse-vector conversion. For FunctionType.BM25, this parameter accepts only one field name.​
output_field_names​	The name of the field where the internally generated sparse vectors will be stored. For FunctionType.BM25, this parameter accepts only one field name.​
function_type​	The type of the function to use. Set the value to FunctionType.BM25.​

Configure the index

After defining the schema with necessary fields and the built-in function, set up the index for your collection. To simplify this process, use AUTOINDEX as the index_type, an option that allows Milvus to choose and configure the most suitable index type based on the structure of your data.​

index_params = MilvusClient.prepare_index_params()​
​
index_params.add_index(​
    field_name="sparse",​
    index_type="AUTOINDEX", ​
    metric_type="BM25"​
)​

Parameter​	Description​
field_name​	The name of the vector field to index. For full text search, this should be the field that stores the generated sparse vectors. In this example, set the value to sparse.​
index_type​	The type of the index to create. AUTOINDEX allows Milvus to automatically optimize index settings. If you need more control over your index settings, you can choose from various index types available for sparse vectors in Milvus. For more information, refer to Indexes supported in Milvus.​
metric_type​	The value for this parameter must be set to BM25 specifically for full text search functionality.​

Create the collection​

Now create the collection using the schema and index parameters defined.​

MilvusClient.create_collection(​
    collection_name='demo', ​
    schema=schema, ​
    index_params=index_params​
)​

Insert text data

After setting up your collection and index, you’re ready to insert text data. In this process, you need only to provide the raw text. The built-in function we defined earlier automatically generates the corresponding sparse vector for each text entry.​

MilvusClient.insert('demo', [​
    {'text': 'Artificial intelligence was founded as an academic discipline in 1956.'},​
    {'text': 'Alan Turing was the first person to conduct substantial research in AI.'},​
    {'text': 'Born in Maida Vale, London, Turing was raised in southern England.'},​
])​

Perform full text search

Once you’ve inserted data into your collection, you can perform full text searches using raw text queries. Milvus automatically converts your query into a sparse vector and ranks the matched search results using the BM25 algorithm, and then returns the topK (limit) results.​

search_params = {​
    'params': {'drop_ratio_search': 0.6},​
}​
​
MilvusClient.search(​
    collection_name='demo', ​
    data=['Who started AI research?'],​
    anns_field='sparse',​
    limit=3,​
    search_params=search_params​
)​

Parameter​	Description​
search_params​	A dictionary containing search parameters.​
params.drop_ratio_search​	Proportion of low-frequency terms to ignore during search. For details, refer to Sparse Vector.​
data​	The raw query text.​
anns_field​	The name of the field that contains internally generated sparse vectors.​
limit​	Maximum number of top matches to return.​

​