跳到主要内容

Stemmer

stemmer filter 将单词减少到其基本或词根形式（称为词干提取），使得更容易匹配不同屈折变化中具有相似含义的单词。stemmer filter 支持多种语言，允许在各种语言环境中进行有效的搜索和索引。

配置​

stemmer filter 是 Milvus 中的自定义 filter。要使用它，请在 filter 配置中指定 "type": "stemmer"，并使用 language 参数选择所需的词干提取语言。

PythonJavaNodeJSGocURL
analyzer_params = {
    "tokenizer": "standard",
    "filter":[{
        "type": "stemmer", # Specifies the filter type as stemmer
        "language": "english", # Sets the language for stemming to English
    }],
}
Map<String, Object> analyzerParams = new HashMap<>();
analyzerParams.put("tokenizer", "standard");
analyzerParams.put("filter",
        Collections.singletonList(
                new HashMap<String, Object>() {{
                    put("type", "stemmer");
                    put("language", "english");
                }}
        )
);
const analyzer_params = {
    "tokenizer": "standard",
    "filter":[{
        "type": "stemmer", // Specifies the filter type as stop
        "language": "english", 
    }],
};
analyzerParams = map[string]any{"tokenizer": "standard",
    "filter": []any{map[string]any{
        "type":     "stemmer",
        "language": "english",
    }}}
# restful
analyzerParams='{
  "tokenizer": "standard",
  "filter": [
    {
      "type": "stemmer",
      "language": "english"
    }
  ]
}'

stemmer filter 接受以下可配置参数：

参数

描述

language

指定词干提取过程的语言。支持的语言包括："arabic"、"danish"、"dutch"、"english"、"finnish"、"french"、"german"、"greek"、"hungarian"、"italian"、"norwegian"、"portuguese"、"romanian"、"russian"、"spanish"、"swedish"、"tamil"、"turkish"

stemmer filter 对 tokenizer 生成的 term 进行操作，因此必须与 tokenizer 结合使用。

定义 analyzer_params 后，您可以在定义 collection schema 时将其应用于 VARCHAR field。这允许 Milvus 使用指定的 analyzer 处理该 field 中的文本，以实现高效的分词和过滤。有关详细信息，请参阅 示例使用。

示例​

在将 analyzer 配置应用到您的 collection schema 之前，使用 run_analyzer 方法验证其行为。

Analyzer 配置​

PythonJavaNodeJSGocURL
analyzer_params = {
    "tokenizer": "standard",
    "filter":[{
        "type": "stemmer", # Specifies the filter type as stemmer
        "language": "english", # Sets the language for stemming to English
    }],
}
Map<String, Object> analyzerParams = new HashMap<>();
analyzerParams.put("tokenizer", "standard");
analyzerParams.put("filter",
        Collections.singletonList(
                new HashMap<String, Object>() {{
                    put("type", "stemmer");
                    put("language", "english");
                }}
        )
);
// javascript
analyzerParams = map[string]any{"tokenizer": "standard",
    "filter": []any{map[string]any{
        "type":     "stemmer",
        "language": "english",
    }}}
# restful
analyzerParams='{
  "tokenizer": "standard",
  "filter": [
    {
      "type": "stemmer",
      "language": "english"
    }
  ]
}'

使用 run_analyzer 验证​

PythonJavaNodeJSGo
# Sample text to analyze
sample_text = "running runs looked ran runner"

# Run the standard analyzer with the defined configuration
result = MilvusClient.run_analyzer(sample_text, analyzer_params)
print(result)
// java
// javascript
// go
# restful
not support yet

预期输出​

['run', 'run', 'look', 'ran', 'runner']