Length

The length filter removes tokens that do not meet specified length requirements, allowing you to control the length of tokens retained during text processing.

Configuration

The length filter is a custom filter in Milvus, specified by setting "type": "length" in the filter configuration. You can configure it as a dictionary within the analyzer_params to define length limits.

analyzer_params = {
    "tokenizer": "standard",
    "filter":[{
        "type": "length", # Specifies the filter type as length
        "max": 10, # Sets the maximum token length to 10 characters
    }],
}

The length filter accepts the following configurable parameters.

The length filter operates on the terms generated by the tokenizer, so it must be used in combination with a tokenizer.

After defining analyzer_params, you can apply them to a VARCHAR field when defining a collection schema. This allows Milvus to process the text in that field using the specified analyzer for efficient tokenization and filtering. For details, refer to Example use.

Example output

Here’s an example of how the length filter processes text:

Example text:

"The length filter allows control over token length requirements for text processing."

Expected output (with max: 10):

["length", "filter", "allows", "control", "over", "token", "length", "for", "text"]