milvus-logo
LFAI
Home
  • User Guide

Conduct a Hybrid Search

This topic describes how to conduct a hybrid search.

A hybrid search is essentially a vector search with attribute filtering. By specifying boolean expressions that filter the scalar fields or the primary key field, you can limit your search with certain conditions.

The following example shows how to perform a hybrid search on the basis of a regular vector search. Suppose you want to search for certain books based on their vectorized introductions, but you only want those within a specific range of word count. You can then specify the boolean expression to filter the word_count field in the search parameters. Milvus will search for similar vectors only among those entities that match the expression.

Load collection

All search and query operations within Milvus are executed in memory. Load the collection to memory before conducting a vector search.

from pymilvus import Collection
collection = Collection("book")      # Get an existing collection.
collection.load()
await milvusClient.loadCollection({
  collection_name: "book",
});
err := milvusClient.LoadCollection(
  context.Background(),   // ctx
  "book",                 // CollectionName
  false                   // async
)
if err != nil {
  log.Fatal("failed to load collection:", err.Error())
}
milvusClient.loadCollection(
  LoadCollectionParam.newBuilder()
    .withCollectionName("book")
    .build()
);
```shell load -c book ```
# See the following step.

By specifying the boolean expression, you can filter the scalar field of the entities during the vector search. The following example limits the scale of search to the vectors within a specified word_count value range.

You can also use dynamic fields in the filter expression and output fields in the search requests. For example, refer to Dynamic Schema.

search_param = {
  "data": [[0.1, 0.2]],
  "anns_field": "book_intro",
  "param": {"metric_type": "L2", "params": {"nprobe": 10}, "offset": 0},
  "limit": 10,
  "expr": "word_count <= 11000",
}
res = collection.search(**search_param)
const results = await milvusClient.search({
    collection_name: "book",
    vector: [0.1, 0.2],
    filter: null,
    // the sum of `limit` and `offset` should be less than 16384.
    limit: 10,
    offset: 2,
    metric_type: MetricType.L2,
    param: {
      params: { nprobe: 1024 } 
    },
    consistency_level: ConsistencyLevelEnum.Strong,
});
sp, _ := entity.NewIndexFlatSearchParam(   // NewIndex*SearchParam func
  10,                                      // searchParam
)

opt := client.SearchQueryOptionFunc(func(option *client.SearchQueryOption) {
    option.Limit = 3
    option.Offset = 0
    option.ConsistencyLevel = entity.ClStrong
    option.IgnoreGrowing = false
})

searchResult, err := milvusClient.Search(
  context.Background(),                    // ctx
  "book",                                  // CollectionName
  []string{},                              // partitionNames
  "word_count <= 11000",                   // expr
  []string{"book_id"},                     // outputFields
  []entity.Vector{entity.FloatVector([]float32{0.1, 0.2})}, // vectors
  "book_intro",                            // vectorField
  entity.L2,                               // metricType
  2,                                       // topK
  sp,                                      // sp
  opt,                                     // search options
)

if err != nil {
  log.Fatal("fail to search collection:", err.Error())
}
final Integer SEARCH_K = 2;
final String SEARCH_PARAM = "{\"nprobe\":10, \”offset\”:5}";
List<String> search_output_fields = Arrays.asList("book_id");
List<List<Float>> search_vectors = Arrays.asList(Arrays.asList(0.1f, 0.2f));

SearchParam searchParam = SearchParam.newBuilder()
  .withCollectionName("book")
  .withMetricType(MetricType.L2)
  .withOutFields(search_output_fields)
  .withTopK(SEARCH_K)
  .withVectors(search_vectors)
  .withVectorFieldName("book_intro")
  .withExpr("word_count <= 11000")
  .withParams(SEARCH_PARAM)
  .build();
R<SearchResults> respSearch = milvusClient.search(searchParam);
```shell search

Collection name (book): book

The vectors of search data(the length of data is number of query (nq), the dim of every vector in data must be equal to vector field’s of collection. You can also import a csv file without headers): [[0.1, 0.2]]

The vector field used to search of collection (book_intro): book_intro

Metric type: L2

Search parameter nprobe's value: 10

The max number of returned record, also known as topk: 2

The boolean expression used to filter attribute []: word_count <= 11000

The names of partitions to search (split by "," if multiple) ['_default'] []:

timeout []:

Guarantee Timestamp(It instructs Milvus to see all operations performed before a provided timestamp. If no such timestamp is provided, then Milvus will search all operations performed to date) [0]:

Travel Timestamp(Specify a timestamp in a search to get results based on a data view) [0]:


```curl
curl -X 'POST' \
  'http://localhost:9091/api/v1/search' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "collection_name": "book",
    "output_fields": ["book_id"],
    "search_params": [
      {"key": "anns_field", "value": "book_intro"},
      {"key": "topk", "value": "2"},
      {"key": "params", "value": "{\"nprobe\": 10}"},
      {"key": "metric_type", "value": "L2"},
      {"key": "round_decimal", "value": "-1"}
    ],
    "vectors": [ [0.1,0.2] ],
    "dsl": "word_count >= 11000",
    "dsl_type": 1
  }'
Output:
{
  "status":{},
  "results":{
    "num_queries":1,
    "top_k":2,
    "fields_data":[
      {
        "type":5,
        "field_name":"book_id",
        "Field":{"Scalars":{"Data":{"LongData":{"data":[11,12]}}}},
        "field_id":100
      }
    ],
    "scores":[119.44999,142.24998],
    "ids":{"IdField":{"IntId":{"data":[11,12]}}},"topks":[2]
  },
  "collection_name":"book"
}
Parameter Description
data Vectors to search with.
anns_field Name of the field to search on.
param Search parameters. Possible options are as follows:
  • metric_type Method used to measure the distance between vectors during search. It should be the same as the one specified for the index-building process. See Simlarity Metrics for more information.
  • offset Number of entities to skip during the search. The sum of this parameter and limit of the search method should be less than 16384.
  • ignore_growing Whether to ignore growing segments during similarity searches. The value defaults to False, indicating that searches involve growing segments.
  • params Search parameter(s) specific to the specified index type. See Vector Index for more information. Possible options are as follows:
    • nprobe Indicates the number of cluster units to search. This parameter is available only when index_type is set to IVF_FLAT, IVF_SQ8, or IVF_PQ. The value should be less than nlist specified for the index-building process.
    • ef Indicates the search scope. This parameter is available only when index_type is set to HNSW. The value should be within the range from top_k to 32768.
limit Number of the most similar results to return. The sum of this value and offset should be less than 16384.
expr Boolean expression used to filter attribute. See Boolean Expression Rules for more information.
output_fields (optional) Name of the field to return. Milvus supports returning the vector field.
Parameter Description
collection_name Name of the collection to search in.
search_params Parameters (as an object) used for search.
vectors Vectors to search with.
vector_type Pre-check of binary or float vectors. 100 for binary vectors and 101 for float vectors.
expr (optional) Boolean expression used to filter attribute. See Boolean Expression Rules for more information.
output_fields (optional) Name of the field to return. Milvus supports returning the vector field.
limit (optional) Number of entities to return. The sum of this parameter and offset should be less than 16384.
offset (optional) Number of entities to skip. This parameter applies only when limit is specified, and the sum of this parameter and limit should be less than 16384.
Parameter Description Options
ctx Context to control API invocation process. N/A
CollectionName Name of the collection to load. N/A
partitionNames List of names of the partitions to load. All partitions will be searched if it is left empty. N/A
expr Boolean expression used to filter attribute. See Boolean Expression Rules for more information.
output_fields Name of the field to return. Milvus supports returning the vector field.
vectors Vectors to search with. N/A
vectorField Name of the field to search on. N/A
metricType Metric type used for search. This parameter must be set identical to the metric type used for index building.
topK Number of the most similar results to return. N/A
sp Search parameter(s) specific to the index. See Vector Index for more information. Possible options are as follows:
  • For floating point vectors:
    • NewIndexFlatSearchParam() (FLAT)
    • NewIndexIvfFlatSearchParam(nprobe int) (IVF_FLAT)
    • NewIndexIvfSQ8SearchParam(nprobe int) (IVF_SQ8)
    • NewIndexIvfPQSearchParam(nprobe int) (RNSG)
    • NewIndexHNSWSearchParam(ef int) (HNSW)
  • For binary vectors:
    • NewIndexBinFlatSearchParam(nprobe int) (BIN_FLAT)
    • NewIndexBinIvfFlatSearchParam(nprobe int) (BIN_IVF_FLAT)
opts Search options in the form of entity.SearchQueryOptionFunc.
  • Limit Indicates the number of entities to return.
  • Offset Indicates the number of entities to skip during the search. The sum of this parameter and Limit should be less than 16384.
  • ConsistencyLevel Indicates the consistency level applied during the search.
  • Ignore Growing Indicates whether to ignore growing segments during similarity searches. The value defaults to False, indicating that searches involve growing segments.
Parameter Description Options
CollectionName Name of the collection to load. N/A
MetricType Metric type used for search. This parameter must be set identical to the metric type used for index building.
OutFields Name of the field to return. Milvus supports returning the vector field.
TopK Number of the most similar results to return. N/A
Vectors Vectors to search with. N/A
VectorFieldName Name of the field to search on. N/A
Expr Boolean expression used to filter attribute. See Boolean Expression Rules for more information.
Params Search parameter(s) specific to the index. See Vector Index for more information. Possible options are as follows:
  • nprobe Indicates the number of cluster units to search. This parameter is available only when index_type is set to IVF_FLAT, IVF_SQ8, or IVF_PQ. The value should be less than nlist specified for the index-building process.
  • ef Indicates the search scope. This parameter is available only when index_type is set to HNSW. The value should be within the range from top_k to 32768.
  • metric_type Indicates the metric type used in the search. It should be the same as the one specified when you index the collection.
  • limit Indicates the number of entities to return starting from the last skippped entity.
  • offset Indicates the number of entities to skip during the search. The sum of this parameter and topK of the withTopK() method should be less than 16384.

Check the returned results.

assert len(res) == 1
hits = res[0]
assert len(hits) == 2
print(f"- Total hits: {len(hits)}, hits ids: {hits.ids} ")
print(f"- Top1 hit id: {hits[0].id}, distance: {hits[0].distance}, score: {hits[0].score} ")
console.log(results.results)
fmt.Printf("%#v\n", searchResult)
for _, sr := range searchResult {
  fmt.Println(sr.IDs)
  fmt.Println(sr.Scores)
}
SearchResultsWrapper wrapperSearch = new SearchResultsWrapper(respSearch.getData().getResults());
System.out.println(wrapperSearch.getIDScore(0));
System.out.println(wrapperSearch.getFieldData("book_id", 0));
# Milvus CLI automatically returns the primary key values of the most similar vectors and their distances.
# See the output of the previous step.

What's next