Conduct a Hybrid Search

This topic describes how to conduct a hybrid search.

A hybrid search is essentially a vector search with attribute filtering. By specifying boolean expressions that filter the scalar fields or the primary key field, you can limit your search with certain conditions.

The following example shows how to perform a hybrid search on the basis of a regular vector search. Suppose you want to search for certain books based on their vectorized introductions, but you only want those within a specific range of word count. You can then specify the boolean expression to filter the word_count field in the search parameters. Milvus will search for similar vectors only among those entities that match the expression.

Load collection

All search and query operations within Milvus are executed in memory. Load the collection to memory before conducting a vector search.

Python Java GO Node.js C#

from pymilvus import Collection
collection = Collection("book")      # Get an existing collection.
collection.load()

await milvusClient.loadCollection({
  collection_name: "book",
});

err := milvusClient.LoadCollection(
  context.Background(),   // ctx
  "book",                 // CollectionName
  false                   // async
)
if err != nil {
  log.Fatal("failed to load collection:", err.Error())
}

milvusClient.loadCollection(
  LoadCollectionParam.newBuilder()
    .withCollectionName("book")
    .build()
);

var collection = milvusClient.GetCollection("book").LoadAsync();

Conduct a hybrid vector search

By specifying the boolean expression, you can filter the scalar field of the entities during the vector search. The following example limits the scale of search to the vectors within a specified word_count value range.

You can also use dynamic fields in the filter expression and output fields in the search requests. For example, refer to Dynamic Schema.

Python Java GO Node.js C#

search_param = {
  "data": [[0.1, 0.2]],
  "anns_field": "book_intro",
  "param": {"metric_type": "L2", "params": {"nprobe": 10}, "offset": 0},
  "limit": 10,
  "expr": "word_count <= 11000",
}
res = collection.search(**search_param)

const results = await milvusClient.search({
    collection_name: "book",
    vector: [0.1, 0.2],
    filter: null,
    // the sum of `limit` and `offset` should be less than 16384.
    limit: 10,
    offset: 2,
    metric_type: MetricType.L2,
    param: {
      params: { nprobe: 1024 } 
    },
    consistency_level: ConsistencyLevelEnum.Strong,
});

sp, _ := entity.NewIndexFlatSearchParam(   // NewIndex*SearchParam func
  10,                                      // searchParam
)

opt := client.SearchQueryOptionFunc(func(option *client.SearchQueryOption) {
    option.Limit = 3
    option.Offset = 0
    option.ConsistencyLevel = entity.ClStrong
    option.IgnoreGrowing = false
})

searchResult, err := milvusClient.Search(
  context.Background(),                    // ctx
  "book",                                  // CollectionName
  []string{},                              // partitionNames
  "word_count <= 11000",                   // expr
  []string{"book_id"},                     // outputFields
  []entity.Vector{entity.FloatVector([]float32{0.1, 0.2})}, // vectors
  "book_intro",                            // vectorField
  entity.L2,                               // metricType
  2,                                       // topK
  sp,                                      // sp
  opt,                                     // search options
)

if err != nil {
  log.Fatal("fail to search collection:", err.Error())
}

final Integer SEARCH_K = 2;
final String SEARCH_PARAM = "{\"nprobe\":10, \”offset\”:5}";
List<String> search_output_fields = Arrays.asList("book_id");
List<List<Float>> search_vectors = Arrays.asList(Arrays.asList(0.1f, 0.2f));

SearchParam searchParam = SearchParam.newBuilder()
  .withCollectionName("book")
  .withMetricType(MetricType.L2)
  .withOutFields(search_output_fields)
  .withTopK(SEARCH_K)
  .withVectors(search_vectors)
  .withVectorFieldName("book_intro")
  .withExpr("word_count <= 11000")
  .withParams(SEARCH_PARAM)
  .build();
R<SearchResults> respSearch = milvusClient.search(searchParam);

var results = await milvusClient.GetCollection("book").SearchAsync(
    vectorFieldName: "book_intro",
    vectors: new ReadOnlyMemory<float>[] { new[] { 0.1f, 0.2f } },
    SimilarityMetricType.L2,
    limit: 10,
    new SearchParameters
    {
        OutputFields = { "title" },
        ConsistencyLevel = ConsistencyLevel.Strong,
        Offset = 5,
        Expression = "word_count <= 11000",
        ExtraParameters = { ["nprobe"] = "1024" }
    });

Parameter	Description
`data`	Vectors to search with.
`anns_field`	Name of the field to search on.
`param`	Search parameters. Possible options are as follows: `metric_type` Method used to measure the distance between vectors during search. It should be the same as the one specified for the index-building process. See Simlarity Metrics for more information. `offset` Number of entities to skip during the search. The sum of this parameter and `limit` of the `search` method should be less than `16384`. `ignore_growing` Whether to ignore growing segments during similarity searches. The value defaults to `False`, indicating that searches involve growing segments. `params` Search parameter(s) specific to the specified index type. See Vector Index for more information. Possible options are as follows: `nprobe` Indicates the number of cluster units to search. This parameter is available only when `index_type` is set to `IVF_FLAT`, `IVF_SQ8`, or `IVF_PQ`. The value should be less than `nlist` specified for the index-building process. `ef` Indicates the search scope. This parameter is available only when `index_type` is set to `HNSW`. The value should be within the range from `top_k` to `32768`. `search_k` Indicates the search scope. This parameter is available only when `index_type` is set to `ANNOY`. The value should be greater than or equal to the top K.
`limit`	Number of the most similar results to return. The sum of this value and `offset` should be less than 16384.
`expr`	Boolean expression used to filter attribute. See Boolean Expression Rules for more information.
`output_fields` (optional)	Name of the field to return. Vector field is not supported in current release.

Parameter	Description
`collection_name`	Name of the collection to search in.
`search_params`	Parameters (as an object) used for search.
`vectors`	Vectors to search with.
`vector_type`	Pre-check of binary or float vectors. `100` for binary vectors and `101` for float vectors.
`expr` (optional)	Boolean expression used to filter attribute. See Boolean Expression Rules for more information.
`output_fields` (optional)	Name of the field to return. The vector field does not support in the current release.
`limit` (optional)	Number of entities to return. The sum of this parameter and `offset` should be less than 16384.
`offset` (optional)	Number of entities to skip. This parameter applies only when `limit` is specified, and the sum of this parameter and `limit` should be less than 16384.

Parameter	Description	Options
`ctx`	Context to control API invocation process.	N/A
`CollectionName`	Name of the collection to load.	N/A
`partitionNames`	List of names of the partitions to load. All partitions will be searched if it is left empty.	N/A
`expr`	Boolean expression used to filter attribute.	See Boolean Expression Rules for more information.
`output_fields`	Name of the field to return.	Vector field is not supported in current release.
`vectors`	Vectors to search with.	N/A
`vectorField`	Name of the field to search on.	N/A
`metricType`	Metric type used for search.	This parameter must be set identical to the metric type used for index building.
`topK`	Number of the most similar results to return.	N/A
`sp`	Search parameter(s) specific to the index.	See Vector Index for more information. Possible options are as follows: For floating point vectors: `NewIndexFlatSearchParam()` (FLAT) `NewIndexIvfFlatSearchParam(nprobe int)` (IVF_FLAT) `NewIndexIvfSQ8SearchParam(nprobe int)` (IVF_SQ8) `NewIndexIvfPQSearchParam(nprobe int)` (RNSG) `NewIndexHNSWSearchParam(ef int)` (HNSW) `NewIndexANNOYSearchParam(search_k int)` (ANNOY) For binary vectors: `NewIndexBinFlatSearchParam(nprobe int)` (BIN_FLAT) `NewIndexBinIvfFlatSearchParam(nprobe int)` (BIN_IVF_FLAT)
`opts`	Search options in the form of `entity.SearchQueryOptionFunc`.	`Limit` Indicates the number of entities to return. `Offset` Indicates the number of entities to skip during the search. The sum of this parameter and `Limit` should be less than `16384`. `ConsistencyLevel` Indicates the consistency level applied during the search. `Ignore Growing` Indicates whether to ignore growing segments during similarity searches. The value defaults to `False`, indicating that searches involve growing segments.

Parameter	Description	Options
`CollectionName`	Name of the collection to load.	N/A
`MetricType`	Metric type used for search.	This parameter must be set identical to the metric type used for index building.
`OutFields`	Name of the field to return.	Vector field is not supported in current release.
`TopK`	Number of the most similar results to return.	N/A
`Vectors`	Vectors to search with.	N/A
`VectorFieldName`	Name of the field to search on.	N/A
`Expr`	Boolean expression used to filter attribute.	See Boolean Expression Rules for more information.
`Params`	Search parameter(s) specific to the index.	See Vector Index for more information. Possible options are as follows: `nprobe` Indicates the number of cluster units to search. This parameter is available only when `index_type` is set to `IVF_FLAT`, `IVF_SQ8`, or `IVF_PQ`. The value should be less than `nlist` specified for the index-building process. `ef` Indicates the search scope. This parameter is available only when `index_type` is set to `HNSW`. The value should be within the range from `top_k` to `32768`. `search_k` Indicates the search scope. This parameter is available only when `index_type` is set to `ANNOY`. The value should be greater than or equal to the top K. `metric_type` Indicates the metric type used in the search. It should be the same as the one specified when you index the collection. `limit` Indicates the number of entities to return starting from the last skippped entity. `offset` Indicates the number of entities to skip during the search. The sum of this parameter and `topK` of the `withTopK()` method should be less than `16384`.

Parameter	Description
OutputFields	A dictionary of the fields in the search results.
ConsistencyLevel	Consistency level for the search. Possible values are: ConsistencyLevel.Strong ConsistencyLevel.Session ConsistencyLevel.BoundedStaleness ConsistencyLevel.Eventually ConsistencyLevel.Customized
Offset	Number of records to skip before return. The sum of this value and `limit` in the search request should be less than `16384`.
Expression	Boolean expression used to filter attribute. See Boolean Expression Rules for more information.
ExtraParameters	Other applicable parameters. See Vector Index for more information. Possible options are as follows: `nprobe` Indicates the number of cluster units to search. This parameter is available only when `index_type` is set to `IVF_FLAT`, `IVF_SQ8`, or `IVF_PQ`. The value should be less than `nlist` specified for the index-building process. `ef` Indicates the search scope. This parameter is available only when `index_type` is set to `HNSW`. The value should be within the range from `top_k` to `32768`. `search_k` Indicates the search scope. This parameter is available only when `index_type` is set to `ANNOY`. The value should be greater than or equal to the top K.

Check the returned results.

Python Java GO Node.js C#

assert len(res) == 1
hits = res[0]
assert len(hits) == 2
print(f"- Total hits: {len(hits)}, hits ids: {hits.ids} ")
print(f"- Top1 hit id: {hits[0].id}, distance: {hits[0].distance}, score: {hits[0].score} ")

console.log(results.results)

fmt.Printf("%#v\n", searchResult)
for _, sr := range searchResult {
  fmt.Println(sr.IDs)
  fmt.Println(sr.Scores)
}

SearchResultsWrapper wrapperSearch = new SearchResultsWrapper(respSearch.getData().getResults());
System.out.println(wrapperSearch.getIDScore(0));
System.out.println(wrapperSearch.getFieldData("book_id", 0));

// # get the IDs of all returned hits
Console.WriteLine(results.Ids.LongIds)
// alternative Console.WriteLine(results.Ids.StringIds)

// get the scores to the query vector from all returned hits
foreach (var score in results.Scores.ToList()) {
    Console.WriteLine(score);
};

What's next

Try Search with Time Travel
Explore API references for Milvus SDKs: