🚀 Try Zilliz Cloud, the fully managed Milvus, for free—experience 10x faster performance! Try Now>>

Milvus
Zilliz
Home
  • User Guide
  • Home
  • Docs
  • User Guide

  • Schema & Data Fields

  • Analyzer

  • Tokenizers

  • Lindera

Lindera

The lindera tokenizer performs dictionary-based morphological analysis. It is a good choice for languages—such as Japanese, Korean, and Chinese—whose words are not separated by spaces.

Configuration

To configure an analyzer using the lindera tokenizer, set tokenizer.type to lindera and choose a dictionary with dict_kind.

analyzer_params = {
    "tokenizer": {
      "type": "lindera""dict_kind": "ipadic"
    }
}

Parameter

Description

type

The type of tokenizer. This is fixed to "lindera".

dict

A list of dictionaries used to define vocabulary. Possible values:

  • ipadic: Japanese

  • ko-dic: Korean

  • cc-cedict: Mandarin Chinese (traditional/simpl.)

After defining analyzer_params, you can apply them to a VARCHAR field when defining a collection schema. This allows Milvus to process the text in that field using the specified analyzer for efficient tokenization and filtering. For details, refer to Example use.

Examples

Before applying the analyzer configuration to your collection schema, verify its behavior using the run_analyzer method.

Analyzer configuration

analyzer_params = {
    "tokenizer": {
      "type": "lindera",
      "dict_kind": "ipadic"
    }
}

Verification using run_analyzer

# Sample text to analyze
sample_text = "東京スカイツリーの最寄り駅はとうきょうスカイツリー駅で"

# Run the standard analyzer with the defined configuration
result = MilvusClient.run_analyzer(sample_text, analyzer_params)
print(result)

Expected output

{tokens: ['東京', 'スカイ', 'ツリー', 'の', '最寄り駅', 'は', 'とう', 'きょう', 'スカイ', 'ツリー', '駅', 'で']} 

Try Managed Milvus for Free

Zilliz Cloud is hassle-free, powered by Milvus and 10x faster.

Get Started
Feedback

Was this page helpful?