Decimal DigitCompatible with Milvus 3.0.0+
The decimaldigit filter is a built-in token filter that converts Unicode decimal digits from supported scripts into ASCII digits. This makes numeric tokens consistent across languages and writing systems.
Configuration
For Arabic text, the built-in arabic analyzer already includes the decimaldigit filter. Use decimaldigit directly when you need digit normalization in a custom analyzer pipeline.
To use the decimaldigit filter in a custom analyzer, add it to the filter section in analyzer_params:
analyzer_params = {
"tokenizer": "standard",
"filter": ["decimaldigit"],
}
The decimaldigit filter has no configurable parameters.
The filter converts Unicode decimal digits, including Arabic-Indic, Thai, Devanagari, Bengali, and fullwidth digits, to ASCII digits. It operates on tokens generated by the tokenizer. The configuration above is intentionally a custom analyzer example and does not include the complete Arabic processing pipeline.
Examples
Before applying the analyzer configuration to your collection schema, verify its behavior using the run_analyzer method.
Analyzer configuration
analyzer_params = {
"tokenizer": "standard",
"filter": ["decimaldigit"],
}
Verification using run_analyzer
from pymilvus import MilvusClient
client = MilvusClient(uri="http://localhost:19530")
sample_text = "Arabic ١٢٣ Thai ๑๒๓ Devanagari १२३ Fullwidth １２３"
result = client.run_analyzer(sample_text, analyzer_params)
print(result)
Expected output
['Arabic', '123', 'Thai', '123', 'Devanagari', '123', 'Fullwidth', '123']