Decompounder

The decompounder filter splits compound words into individual components based on a specified dictionary, making it easier to search for parts of compound terms. This filter is particularly useful for languages that frequently use compound words, such as German.

The component dictionary can be supplied inline via the word_list parameter or loaded from a registered file resource via the word_list_file parameter.

Configuration

The decompounder filter accepts its component dictionary either inline via the word_list parameter or from a registered file resource via the word_list_file parameter.

Inline word list

The decompounder filter is a custom filter in Milvus. To use it, specify "type": "decompounder" in the filter configuration, along with a word_list parameter that provides the dictionary of word components to recognize.

analyzer_params = {
    "tokenizer": "standard",
    "filter":[{
        "type": "decompounder", # Specifies the filter type as decompounder
        "word_list": ["dampf", "schiff", "fahrt", "brot", "backen", "automat"],
    }],
}

import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.Map;

Map<String, Object> analyzerParams = new HashMap<>();
analyzerParams.put("tokenizer", "standard");
analyzerParams.put("filter",
        Collections.singletonList(
                new HashMap<String, Object>() {{
                    put("type", "decompounder");
                    put("word_list", Arrays.asList("dampf", "schiff", "fahrt", "brot", "backen", "automat"));
                }}
        )
);

analyzerParams := map[string]any{"tokenizer": "standard",
    "filter": []any{map[string]any{
        "type":       "decompounder",
        "word_list": []string{"dampf", "schiff", "fahrt", "brot", "backen", "automat"},
    }}}

let analyzer_params = serde_json::json!({
    "tokenizer": "standard",
    "filter": [{
        "type": "decompounder",
        "word_list": ["dampf", "schiff", "fahrt", "brot", "backen", "automat"]
    }]
});

nlohmann::json analyzer_params = {
    {"tokenizer", "standard"},
    {"filter", {
        {
            {"type", "decompounder"},
            {"word_list", {"dampf", "schiff", "fahrt", "brot", "backen", "automat"}}
        }
    }}
};

const analyzer_params = {
    "tokenizer": "standard",
    "filter":[{
        "type": "decompounder", // Specifies the filter type as decompounder
        "word_list": ["dampf", "schiff", "fahrt", "brot", "backen", "automat"],
    }],
};

using System.Collections.Generic;

var analyzerParams = new Dictionary<string, object>
{
    ["tokenizer"] = "standard",
    ["filter"] = new object[]
    {
        new Dictionary<string, object>
        {
            ["type"] = "decompounder", // Specifies the filter type as decompounder
            ["word_list"] = new[] { "dampf", "schiff", "fahrt", "brot", "backen", "automat" }
        }
    }
};

# restful
analyzerParams='{
  "tokenizer": "standard",
  "filter": [
    {
      "type": "decompounder",
      "word_list": [
        "dampf",
        "schiff",
        "fahrt",
        "brot",
        "backen",
        "automat"
      ]
    }
  ]
}'

The decompounder filter accepts the following configurable parameters.

Parameter	Description
word_list	A list of word components used to split compound terms. This dictionary determines how compound words are decomposed into individual terms.

The decompounder filter operates on the terms generated by the tokenizer, so it must be used in combination with a tokenizer. For a list of tokenizers available in Milvus, refer to Standard Tokenizer and its sibling pages.

After defining analyzer_params, you can apply them to a VARCHAR field when defining a collection schema. This allows Milvus to process the text in that field using the specified analyzer for efficient tokenization and filtering. For details, refer to Example use.

Load word components from a file resourceCompatible with Milvus 3.0.x

For large component dictionaries — especially full-language word lists — store the components in a file and register the file as a remote file resource, then reference it from the filter via the word_list_file parameter. You can use word_list_file on its own or alongside inline word_list; when both are set, the filter merges the two sources into a single component list.

The file is plain UTF‑8 text with one component word per line. For example:

dampf
schiff
fahrt
brot
backen
automat

Upload the file to the object store that your Milvus cluster is configured to use, then register it:

from pymilvus import MilvusClient

client = MilvusClient(uri="http://localhost:19530")

# Register the uploaded file under a name you'll reference from analyzer configs.
client.add_file_resource(
    name="de_components",
    path="file/decompounder.txt",    # full S3 object key, including the rootPath prefix
)

Reference the registered resource in the filter via word_list_file:

analyzer_params = {
    "tokenizer": "standard",
    "filter": [{
        "type": "decompounder",
        "word_list_file": {
            "type": "remote",
            "resource_name": "de_components",
            "file_name": "decompounder.txt",
        },
    }],
}

The word_list_file parameter accepts an object with the following fields:

Field	Description
type	The resource type. Use "remote" for a file registered via add_file_resource. For the "local" variant used in self-hosted deployments, refer to Manage File Resources.
resource_name	The name used when the file was registered with add_file_resource.
file_name	The filename portion of the registered resource's object-store path (for example, "decompounder.txt" if the resource was registered with path="file/decompounder.txt").

Examples

Before applying the analyzer configuration to your collection schema, verify its behavior using the run_analyzer method.

Analyzer configuration

analyzer_params = {
    "tokenizer": "standard",
    "filter":[{
        "type": "decompounder", # Specifies the filter type as decompounder
        "word_list": ["dampf", "schiff", "fahrt", "brot", "backen", "automat"],
    }],
}

import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.Map;

Map<String, Object> analyzerParams = new HashMap<>();
analyzerParams.put("tokenizer", "standard");
analyzerParams.put("filter",
        Collections.singletonList(
                new HashMap<String, Object>() {{
                    put("type", "decompounder");
                    put("word_list", Arrays.asList("dampf", "schiff", "fahrt", "brot", "backen", "automat"));
                }}
        )
);

analyzerParams := map[string]any{"tokenizer": "standard",
    "filter": []any{map[string]any{
        "type":       "decompounder",
        "word_list": []string{"dampf", "schiff", "fahrt", "brot", "backen", "automat"},
    }}}

let analyzer_params = serde_json::json!({
    "tokenizer": "standard",
    "filter": [{
        "type": "decompounder",
        "word_list": ["dampf", "schiff", "fahrt", "brot", "backen", "automat"]
    }]
});

nlohmann::json analyzer_params = {
    {"tokenizer", "standard"},
    {"filter", {
        {
            {"type", "decompounder"},
            {"word_list", {"dampf", "schiff", "fahrt", "brot", "backen", "automat"}}
        }
    }}
};

const analyzer_params = {
    "tokenizer": "standard",
    "filter":[{
        "type": "decompounder", // Specifies the filter type as decompounder
        "word_list": ["dampf", "schiff", "fahrt", "brot", "backen", "automat"],
    }],
};

using System.Collections.Generic;

var analyzerParams = new Dictionary<string, object>
{
    ["tokenizer"] = "standard",
    ["filter"] = new object[]
    {
        new Dictionary<string, object>
        {
            ["type"] = "decompounder", // Specifies the filter type as decompounder
            ["word_list"] = new[] { "dampf", "schiff", "fahrt", "brot", "backen", "automat" }
        }
    }
};

# restful
analyzerParams='{
  "tokenizer": "standard",
  "filter": [
    {
      "type": "decompounder",
      "word_list": [
        "dampf",
        "schiff",
        "fahrt",
        "brot",
        "backen",
        "automat"
      ]
    }
  ]
}'

Verification using run_analyzer

from pymilvus import (
    MilvusClient,
)

client = MilvusClient(uri="http://localhost:19530", token="root:Milvus")

# Sample text to analyze
sample_text = "dampfschifffahrt brotbackautomat"

# Run the analyzer with the defined configuration
result = client.run_analyzer(sample_text, analyzer_params)
print("Analyzer output:", result)

import io.milvus.v2.client.ConnectConfig;
import io.milvus.v2.client.MilvusClientV2;
import io.milvus.v2.service.vector.request.RunAnalyzerReq;
import io.milvus.v2.service.vector.response.RunAnalyzerResp;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

ConnectConfig config = ConnectConfig.builder()
        .uri("http://localhost:19530")
        .token("root:Milvus")
        .build();
MilvusClientV2 client = new MilvusClientV2(config);

List<String> texts = new ArrayList<>();
texts.add("dampfschifffahrt brotbackautomat");

RunAnalyzerResp resp = client.runAnalyzer(RunAnalyzerReq.builder()
        .texts(texts)
        .analyzerParams(analyzerParams)
        .build());
List<RunAnalyzerResp.AnalyzerResult> results = resp.getResults();

import (
    "context"
    "encoding/json"
    "fmt"

    "github.com/milvus-io/milvus/client/v3/milvusclient"
)

ctx := context.Background()

client, err := milvusclient.New(ctx, &milvusclient.ClientConfig{
    Address: "http://localhost:19530",
    APIKey:  "root:Milvus",
})
if err != nil {
    fmt.Println(err.Error())
    // handle error
}

bs, _ := json.Marshal(analyzerParams)
texts := []string{"dampfschifffahrt brotbackautomat"}
option := milvusclient.NewRunAnalyzerOption(texts...).
    WithAnalyzerParamsStr(string(bs))

result, err := client.RunAnalyzer(ctx, option)
if err != nil {
    fmt.Println(err.Error())
    // handle error
}
fmt.Println(result)

use milvus::v2::error::Result;
use milvus::v2::prelude::*;
use serde_json::json;

let config = ConnectConfig::new().uri("http://localhost:19530").token("root:Milvus");
let client = ClientV2::new(&config).await?;

let analyzer_params = json!({
    "tokenizer": "standard",
    "filter": [{
        "type": "decompounder",
        "word_list": ["dampf", "schiff", "fahrt", "brot", "backen", "automat"]
    }]
});

let result = client.run_analyzer(
    RunAnalyzerRequest::builder()
        .texts(["dampfschifffahrt brotbackautomat".to_string()])
        .analyzer_params(analyzer_params)
        .build()?,
).await?;
println!("{:?}", result.results());

#include "milvus/MilvusClientV2.h"
#include <iostream>
#include <string>

auto client = milvus::MilvusClientV2::Create();

milvus::ConnectParam connect_param{"http://localhost:19530", "root:Milvus"};
auto status = client->Connect(connect_param);
if (!status.IsOk()) {
    std::cout << status.Message() << std::endl;
}

std::string text = "dampfschifffahrt brotbackautomat";
auto request = milvus::RunAnalyzerRequest()
                       .AddText(text)
                       .WithAnalyzerParams(analyzer_params);

milvus::RunAnalyzerResponse response;
status = client->RunAnalyzer(request, response);
if (!status.IsOk()) {
    std::cout << status.Message() << std::endl;
}

for (const auto& result : response.Results()) {
    for (const auto& token : result.Tokens()) {
        std::cout << token.token_ << " ";
    }
    std::cout << std::endl;
}

import { MilvusClient } from '@zilliz/milvus2-sdk-node';

const client = new MilvusClient({
  address: 'http://localhost:19530',
  token: 'root:Milvus'
});

const res = await client.runAnalyzer({
  text: 'dampfschifffahrt brotbackautomat',
  analyzer_params: analyzer_params
});

console.log(res);

using Milvus.Client.V2;
using Milvus.Client.V2.Requests.Utility;
using Milvus.Client.V2.Types;

using var client = new MilvusClientV2(new ConnectConfig { Uri = "http://localhost:19530", ApiKey = "root:Milvus" });
await client.ConnectAsync();

// Sample text to analyze
var sampleText = "dampfschifffahrt brotbackautomat";

var result = await client.RunAnalyzerAsync(new RunAnalyzerReq
{
    AnalyzerParams = analyzerParams,
    Texts = new[] { sampleText }
});
foreach (var r in result.Results)
{
    Console.WriteLine("Analyzer output: " + string.Join(", ", r.Tokens.Select(t => t.Token)));
}

# restful
export CLUSTER_ENDPOINT="http://localhost:19530"
export TOKEN="root:Milvus"

curl --request POST \
  --url "${CLUSTER_ENDPOINT}/v2/vectordb/common/run_analyzer" \
  --header "Authorization: Bearer ${TOKEN}" \
  --header "Content-Type: application/json" \
  --data '{
    "analyzerParams": "{\"tokenizer\":\"standard\",\"filter\":[{\"type\":\"decompounder\",\"word_list\":[\"dampf\",\"schiff\",\"fahrt\",\"brot\",\"backen\",\"automat\"]}]}",
    "text": ["dampfschifffahrt brotbackautomat"]
  }'

Expected output

['dampf', 'schiff', 'fahrt', 'brotbackautomat']