Basic Operators
Milvus provides a rich set of basic operators to help you filter and query data efficiently. These operators allow you to refine your search conditions based on scalar fields, numeric calculations, logical conditions, and more. Understanding how to use these operators is crucial for building precise queries and maximizing the efficiency of your searches.
Comparison operators
Comparison operators are used to filter data based on equality, inequality, or size. They are applicable to numeric, text, and date fields.
Supported Comparison Operators:
==
(Equal to)!=
(Not equal to)>
(Greater than)<
(Less than)>=
(Greater than or equal to)<=
(Less than or equal to)
Example 1: Filtering with Greater Than or Equal To (>=
)
If you want to find all entities with rating
greater than or equal to 4:
filter = 'rating >= 4'
Example 2: Filtering with Less Than or Equal To (<=
)
To find entities with discount
less than or equal to 10%:
filter = 'discount <= 10'
Range operators
Range operators help filter data based on specific sets or ranges of values.
Supported Range Operators:
IN
: Used to match values within a specific set or range.LIKE
: Used to match a pattern (mostly for text fields).
Example 1: Using IN
to Match Multiple Values
If you want to find all entities where the color
is either "red", "green", or "blue":
filter = 'color in ["red", "green", "blue"]'
This is useful when you want to check for membership in a list of values.
Example 2: Using LIKE
for Pattern Matching
The LIKE
operator is used for pattern matching in string fields. It can match substrings in different positions within the text: as a prefix, infix, or suffix. The LIKE
operator uses the %
symbol as a wildcard, which can match any number of characters (including zero).
Prefix Match (Starts With)
To perform a prefix match, where the string starts with a given pattern, you can place the pattern at the beginning and use %
to match any characters following it. For example, to find all products whose name
starts with "Prod":
filter = 'name LIKE "Prod%"'
This will match any product whose name starts with "Prod", such as "Product A", "Product B", etc.
Suffix Match (Ends With)
For a suffix match, where the string ends with a given pattern, place the %
symbol at the beginning of the pattern. For example, to find all products whose name
ends with "XYZ":
filter = 'name LIKE "%XYZ"'
This will match any product whose name ends with "XYZ", such as "ProductXYZ", "SampleXYZ", etc.
Infix Match (Contains)
To perform an infix match, where the pattern can appear anywhere in the string, you can place the %
symbol at both the beginning and the end of the pattern. For example, to find all products whose name
contains the word "Pro":
filter = 'name LIKE "%Pro%"'
This will match any product whose name contains the substring "Pro", such as "Product", "ProLine", or "SuperPro".
Arithmetic Operators
Arithmetic operators allow you to create conditions based on calculations involving numeric fields.
Supported Arithmetic Operators:
+
(Addition)-
(Subtraction)*
(Multiplication)/
(Division)%
(Modulus)**
(Exponentiation)
Example 1: Using Addition (+
)
To find entities where the total
price is the sum of base_price
and tax
:
filter = 'total == base_price + tax'
Example 2: Using Subtraction (-
)
To find entities where quantity
is greater than 50 and quantity_sold
is less than 30:
filter = 'quantity - quantity_sold > 50'
Example 3: Using Multiplication (*
)
To find entities where price
is greater than 100 and quantity
is greater than 10, multiplied:
filter = 'price * quantity > 1000'
Example 4: Using Division (/
)
To find products where total_price
divided by quantity
is less than 50:
filter = 'total_price / quantity < 50'
Example 5: Using Modulus (%
)
To find entities where the id
is an even number (i.e., divisible by 2):
filter = 'id % 2 == 0'
Example 6: Using Exponentiation (**
)
To find entities where price
raised to the power of 2 is greater than 1000:
filter = 'price ** 2 > 1000'
Logical Operators
Logical operators are used to combine multiple conditions into a more complex filter expression. These include AND
, OR
, and NOT
.
Supported Logical Operators:
AND
: Combines multiple conditions that must all be true.OR
: Combines conditions where at least one must be true.NOT
: Negates a condition.
Example 1: Using AND
to Combine Conditions
To find all products where price
is greater than 100 and stock
is greater than 50:
filter = 'price > 100 AND stock > 50'
Example 2: Using OR
to Combine Conditions
To find all products where color
is either “red” or "blue":
filter = 'color == "red" OR color == "blue"'
Example 3: Using NOT
to Exclude a Condition
To find all products where color
is not "green":
filter = 'NOT color == "green"'
IS NULL and IS NOT NULL Operators
The IS NULL
and IS NOT NULL
operators are used to filter fields based on whether they contain a null value (absence of data).
IS NULL
: Identifies entities where a specific field contains a null value, i.e., the value is absent or undefined.IS NOT NULL
: Identifies entities where a specific field contains any value other than null, meaning the field has a valid, defined value.
The operators are case-insensitive, so you can use IS NULL
or is null
, and IS NOT NULL
or is not null
.
Regular Scalar Fields with Null Values
Milvus allows filtering on regular scalar fields, such as strings or numbers, with null values.
An empty string ""
is not treated as a null value for a VARCHAR field.
To retrieve entities where the description
field is null:
filter = 'description IS NULL'
To retrieve entities where the description
field is not null:
filter = 'description IS NOT NULL'
To retrieve entities where the description
field is not null and the price
field is higher than 10:
filter = 'description IS NOT NULL AND price > 10'
JSON Fields with Null Values
Milvus allows filtering on JSON fields that contain null values. A JSON field is treated as null in the following ways:
- The entire JSON object is explicitly set to None (null), for example,
{"metadata": None}
. - The JSON field itself is completely missing from the entity.
If some elements within a JSON object are null (e.g. individual keys), the field is still considered non-null. For example, {"metadata": {"category": None, "price": 99.99}}
is not treated as null, even though the category
key is null.
To further illustrate how Milvus handles JSON fields with null values, consider the following sample data with a JSON field metadata
:
data = [
{
"metadata": {"category": "electronics", "price": 99.99, "brand": "BrandA"},
"pk": 1,
"embedding": [0.12, 0.34, 0.56]
},
{
"metadata": None, # Entire JSON object is null
"pk": 2,
"embedding": [0.56, 0.78, 0.90]
},
{ # JSON field `metadata` is completely missing
"pk": 3,
"embedding": [0.91, 0.18, 0.23]
},
{
"metadata": {"category": None, "price": 99.99, "brand": "BrandA"}, # Individual key value is null
"pk": 4,
"embedding": [0.56, 0.38, 0.21]
}
]
Example 1: Retrieve entities where metadata
is null
To find entities where the metadata
field is either missing or explicitly set to None:
filter = 'metadata IS NULL'
# Example output:
# data: [
# "{'metadata': None, 'pk': 2}",
# "{'metadata': None, 'pk': 3}"
# ]
Example 2: Retrieve entities where metadata
is not null
To find entities where the metadata
field is not null:
filter = 'metadata IS NOT NULL'
# Example output:
# data: [
# "{'metadata': {'category': 'electronics', 'price': 99.99, 'brand': 'BrandA'}, 'pk': 1}",
# "{'metadata': {'category': None, 'price': 99.99, 'brand': 'BrandA'}, 'pk': 4}"
# ]
ARRAY Fields with Null Values
Milvus allows filtering on ARRAY fields that contain null values. An ARRAY field is treated as null in the following ways:
- The entire ARRAY field is explicitly set to None (null), for example,
"tags": None
. - The ARRAY field is completely missing from the entity.
An ARRAY field cannot contain partial null values as all elements in an ARRAY field must have the same data type. For details, refer to Array Field.
To further illustrate how Milvus handles ARRAY fields with null values, consider the following sample data with an ARRAY field tags
:
data = [
{
"tags": ["pop", "rock", "classic"],
"ratings": [5, 4, 3],
"pk": 1,
"embedding": [0.12, 0.34, 0.56]
},
{
"tags": None, # Entire ARRAY is null
"ratings": [4, 5],
"pk": 2,
"embedding": [0.78, 0.91, 0.23]
},
{ # The tags field is completely missing
"ratings": [9, 5],
"pk": 3,
"embedding": [0.18, 0.11, 0.23]
}
]
Example 1: Retrieve entities where tags
is null
To retrieve entities where the tags
field is either missing or explicitly set to None:
filter = 'tags IS NULL'
# Example output:
# data: [
# "{'tags': None, 'ratings': [4, 5], 'embedding': [0.78, 0.91, 0.23], 'pk': 2}",
# "{'tags': None, 'ratings': [9, 5], 'embedding': [0.18, 0.11, 0.23], 'pk': 3}"
# ]
Example 2: Retrieve entities where tags
is not null
To retrieve entities where the tags
field is not null:
filter = 'tags IS NOT NULL'
# Example output:
# data: [
# "{'metadata': {'category': 'electronics', 'price': 99.99, 'brand': 'BrandA'}, 'pk': 1}",
# "{'metadata': {'category': None, 'price': 99.99, 'brand': 'BrandA'}, 'pk': 4}"
# ]
Tips on Using Basic Operators with JSON and ARRAY Fields
While the basic operators in Milvus are versatile and can be applied to scalar fields, they can also be effectively used with the keys and indexes in the JSON and ARRAY fields.
For example, if you have a product
field that contains multiple keys like price
, model
, and tags
, always reference the key directly:
filter = 'product["price"] > 1000'
To find records where the first temperature in an array of recorded temperatures exceeds a certain value, use:
filter = 'history_temperatures[0] > 30'
Conclusion
Milvus offers a range of basic operators that give you flexibility in filtering and querying your data. By combining comparison, range, arithmetic, and logical operators, you can create powerful filter expressions to narrow down your search results and retrieve the data you need efficiently.