如何在索引时停止在内容中存储特殊字符

问题描述

这是一个示例文档，具有以下几点： 制药业 营销学 建筑物– 责任。 马萨诸塞州-2020年8月13日-

如何在索引时从内容中删除特殊字符或非ASCII Unicode字符？我正在使用ES 7.x和Storm搜寻器1.17

解决方法

看起来像是对字符集的错误检测。您可以在写入索引之前通过写custom parse filter并删除那里不需要的字符来规范内容。

,

如果编写自定义的解析过滤器和规范化对您来说似乎很困难。您只需在分析器定义中添加asciifolding token filter，即可将非ascii字符转换为其ascii字符，如下所示

POST http：// {{hostname}}：{{port}} / _analyze

{
    "tokenizer": "standard","filter": [
        "asciifolding"
    ],"text": "Pharmaceutical Marketing Building â responsibilities.  Mass. â Aug. 13,2020 âÂ"
}

并为您的文本生成令牌。

{
    "tokens": [
        {
            "token": "Pharmaceutical","start_offset": 0,"end_offset": 14,"type": "<ALPHANUM>","position": 0
        },{
            "token": "Marketing","start_offset": 15,"end_offset": 24,"position": 1
        },{
            "token": "Building","start_offset": 25,"end_offset": 33,"position": 2
        },{
            "token": "a","start_offset": 34,"end_offset": 35,"position": 3
        },{
            "token": "responsibilities.A","start_offset": 36,"end_offset": 54,"position": 4
        },{
            "token": "A","start_offset": 55,"end_offset": 56,"position": 5
        },{
            "token": "Mass","start_offset": 57,"end_offset": 61,"position": 6
        },"start_offset": 63,"end_offset": 64,"position": 7
        },{
            "token": "Aug","start_offset": 65,"end_offset": 68,"position": 8
        },{
            "token": "13","start_offset": 70,"end_offset": 72,"type": "<NUM>","position": 9
        },{
            "token": "2020","start_offset": 74,"end_offset": 78,"position": 10
        },{
            "token": "aA","start_offset": 79,"end_offset": 81,"position": 11
        }
    ]
}