ThaiCompatible with Milvus 3.0.0+

The thai analyzer is a built-in analyzer for Thai text. Use this analyzer when you need Milvus to segment Thai text into words, normalize Thai digits, lowercase mixed Latin text, and remove Thai stop words.

Configuration

Built-in analyzers are Milvus-provided analyzer templates. To use a built-in analyzer, set type to a predefined analyzer name in analyzer_params.

To use the built-in Thai analyzer, set type to thai:

analyzer_params = {
    "type": "thai",
}

The thai analyzer accepts the following optional parameter:

Parameter	Type	Default	Description
stop_words	list[str]	_thai_	A list of additional stop words to remove from tokenization. By default, the thai analyzer uses the built-in _thai_ dictionary. To inspect the default dictionary, refer to the Milvus Thai stop-word list. The list is sourced from the Apache Lucene Thai stopwords file.

To add custom stop words, include stop_words:

analyzer_params = {
    "type": "thai",
    "stop_words": ["มิลวัส"],
}

Milvus applies custom stop words in addition to the built-in _thai_ dictionary.

The built-in thai analyzer is equivalent to the following custom analyzer configuration:

analyzer_params = {
    "tokenizer": "thai",
    "filter": [
        "lowercase",
        "decimaldigit",
        {
            "type": "stop",
            "stop_words": ["_thai_"],
        },
    ],
}

This analyzer applies the following processing steps:

• Tokenization: Uses the thai tokenizer to segment Thai text into word tokens without relying on whitespace. The tokenizer filters out whitespace and punctuation-only segments.
• Case normalization: Uses the lowercase filter, which affects Latin letters in mixed Thai/English text.
• Digit normalization: Uses the decimaldigit filter to convert Thai digits and other Unicode decimal digits to ASCII digits.
• Stop-word removal: Uses the stop filter with the built-in _thai_ dictionary.
• No stemming: The built-in thai analyzer does not apply a stemmer filter.

After defining analyzer_params, you can apply the analyzer to a VARCHAR field when defining a collection schema. For details, refer to Example use.

Examples

Before applying the analyzer configuration to your collection schema, verify its behavior using the run_analyzer method.

Analyzer configuration

analyzer_params = {
    "type": "thai",
}

Verification using run_analyzer

from pymilvus import MilvusClient

client = MilvusClient(uri="http://localhost:19530")

sample_text = "ฉันรักการค้นหาข้อความใน Milvus ๑๒๓"

result = client.run_analyzer(sample_text, analyzer_params)
print(result)

Expected output

['ฉัน', 'รัก', 'ค้นหา', 'ข้อความ', 'milvus', '123']