Arabic NormalizationCompatible with Milvus 3.0.0+
The arabic_normalization filter is a built-in token filter for Arabic text. It normalizes Arabic-specific letter variants and removes optional marks that can make equivalent Arabic terms appear different during text analysis.
Configuration
For Arabic text, use the built-in arabic analyzer in most cases. The built-in analyzer includes this filter together with standard tokenization, lowercasing, decimal digit normalization, Arabic stemming, and Arabic stop-word removal. Use arabic_normalization directly only when you need to build a custom analyzer pipeline.
To use the arabic_normalization filter in a custom analyzer, add it to the filter section in analyzer_params:
analyzer_params = {
"tokenizer": "standard",
"filter": ["arabic_normalization"],
}
The arabic_normalization filter has no configurable parameters.
The filter applies the following transformations:
Transformation |
From |
To |
|---|---|---|
Hamza + Alef variants |
|
|
Teh Marbuta |
|
|
Alef Maksura |
|
|
Harakat |
|
Removed |
Tatweel / Kashida |
|
Removed |
The filter operates on tokens generated by the tokenizer. The configuration above is intentionally a custom analyzer example and does not include the complete Arabic processing pipeline.
Examples
Before applying the analyzer configuration to your collection schema, verify its behavior using the run_analyzer method.
Analyzer configuration
analyzer_params = {
"tokenizer": "standard",
"filter": ["arabic_normalization"],
}
Verification using run_analyzer
from pymilvus import MilvusClient
client = MilvusClient(uri="http://localhost:19530")
sample_text = "آدم أحمد إسلام مدرسة كبرى كِتَابٌ عـــربي"
result = client.run_analyzer(sample_text, analyzer_params)
print(result)
Expected output
['ادم', 'احمد', 'اسلام', 'مدرسه', 'كبري', 'كتاب', 'عربي']