Logstash过滤器中的Grok和mutate对输出不起作用

问题描述

Logstash将通过远程主机上的Filebeat收集信息。我最终的预期结果是创建一个图表，用于每天显示来自HTTP日志的响应时间。因此，我将重点放在日志消息中的响应时间上。

现在从Kibana获得的json消息看起来像这样

{
  "_index": "filebeat-7.9.2-2020.10.06-000001","_type": "_doc","_id": "NYaZB3UB-JR2hYtrwawX","_version": 1,"_score": null,"_source": {
    "@timestamp": "2020-10-08T09:45:17.358Z","agent": {
      "id": "acbc8cf5-7bc1-46ab-a9ad-8ee1fef26183","name": "42dcb5cf4d4a","type": "filebeat","version": "7.9.2","hostname": "42dcb5cf4d4a","ephemeral_id": "e45f89ee-f355-4490-b79b-394a2226ffe7"
    },"log": {
      "offset": 5157015,"file": {
        "path": "/usr/logs/tomcat/test.com.access.2020-10-08.log"
      }
    },"message": "203.149.37.226 - - [08/Oct/2020:11:45:10 +0200] \"POST /TestserviceWarning/get_warning/render?method=get_warnings_onsite_status_excel&subDomain=testClick&webId=1196&scanId=4482 HTTP/1.1\" 200 1151 0.010","tags": [
      "tomcat"
    ],"input": {
      "type": "log"
    },"ecs": {
      "version": "1.5.0"
    },"host": {
      "name": "42dcb5cf4d4a"
    }
  },"fields": {
    "@timestamp": [
      "2020-10-08T09:45:17.358Z"
    ]
  },"sort": [
    1602150317358
  ]
}

和我的管道配置

input {
        beats {
                port => 5000
        }
}

filter {
        if "tomcat" in [tags] {
                grok {
                        match => { "message" => "%{IPORHOST:client} %{DATA} %{DATA:user} \[%{DATA:logtimestamp} %{ISO8601_TIMEZONE:timezone}\] \"%{WORD:method} %{URIPATH:uri_path}(%{URIPARAM:params}|) %{DATA:protocol}\" %{NUMBER:code} (%{NUMBER:bytes}|%{DATA}) %{NUMBER:response_time_sec}"}
                        overwrite => [ "message" ]
                }
                mutate { add_field => {"respones_time" => "%{response_time_sec}"}
                }
        }
}

一旦管道被触发，我的json消息中就看不到任何其他字段。结果保持不变。

于2020年9月10日编辑

花了整整一天的时间之后，我仍然无法正常工作。

所以我打开了DEBUG日志级别，似乎Logstash能够访问过滤器，但是它某种程度上无法将过滤器应用于输出。

 org.logstash.config.ir.compiler.ComputeStepSyntaxElement@8d7da797
 P[filter-grok{"match"=>{"message"=>"%{IPORHOST:client} %{DATA} %{DATA:user} \\[%{DATA:logtimestamp} %{ISO8601_TIMEZONE:timezone}\\] \\\"%{WORD:method} %{URIPATH:uri_path}(%{URIPARAM:params}|) %{DATA:protocol}\\\" %{NUMBER:code} (%{NUMBER:bytes}|%{DATA}) %{NUMBER:response_time_sec}"}}|[file]/usr/share/logstash/pipeline/logstash.conf:9:17:```
grok {
                        match => { "message" => "%{IPORHOST:client} %{DATA} %{DATA:user} \[%{DATA:logtimestamp} %{ISO8601_TIMEZONE:timezone}\] \"%{WORD:method} %{URIPATH:uri_path}(%{URIPARAM:params}|) %{DATA:protocol}\" %{NUMBER:code} (%{NUMBER:bytes}|%{DATA}) %{NUMBER:response_time_sec}"}
                     }
```] 
 into 
 org.logstash.config.ir.compiler.ComputeStepSyntaxElement@8d7da797

然后我看到了新的异常消息

[INFO ] 2020-10-09 06:24:55.568 [Agent thread] agent - Pipelines running {:count=>1,:running_pipelines=>[:main],:non_running_pipelines=>[]}
[INFO ] 2020-10-09 06:24:55.649 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
java.lang.UnsupportedOperationException: Reflective setAccessible(true) disabled

和

java.lang.IllegalAccessException: class io.netty.util.internal.PlatformDependent0$6 cannot access class jdk.internal.misc.Unsafe (in module java.base) because module java.base does not export jdk.internal.misc to unnamed module @72da5f0e

解决方法

暂无找到可以解决该程序问题的有效方法，小编努力寻找整理中！

如果你已经找到好的解决方法，欢迎将解决方案带上本链接一起发送给小编。

小编邮箱:dio#foxmail.com (将#修改为@）

logstash logstash-configuration logstash-grok