Jieba

The jieba tokenizer processes Chinese text by breaking it down into its component words.

Configuration

To configure an analyzer using the jieba tokenizer, set tokenizer to jieba in analyzer_params.

analyzer_params = {
    "tokenizer": "jieba",
}

After defining analyzer_params, you can apply them to a VARCHAR field when defining a collection schema. This allows Milvus to process the text in that field using the specified analyzer for efficient tokenization and filtering. For details, refer to Example use.

Example output

Here’s an example of how the jieba tokenizer processes text:

Original text:

"Milvus 是一个高性能、可扩展的向量数据库！"

Expected output:

["Milvus", " ", "是", "一个", "高性", "性能", "高性能", "、", "可", "扩展", "的", "向量", "数据", "据库", "数据库", "！"]