EFKElasticsearch + Fluentd-td-agent+ Kibana：Kibana未显示正确的日志

问题描述

我已经在redhat linux系统（版本7.6）上安装了EFK。流利的稳定分布，即使用td-agent代替流利。日志文件（/mnt/Log/Startup.log）的路径在td-agent.conf文件中配置。但是在kibana仪表板上，它显示的是td-agent.log中的内容，而不是日志文件（Startup.log）。

td-agent.log：

2020-09-04 16:02:16 +0530 [info]: parsing config file is succeeded path="/etc/td-agent/td-agent.conf"
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-elasticsearch' version '4.0.9'
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-kafka' version '0.13.0'
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-prometheus' version '1.8.0'
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-prometheus_pushgateway' version '0.0.2'
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0'
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.3.0'
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-s3' version '1.3.2'
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-systemd' version '1.0.2'
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-td' version '1.1.0'
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.4'
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-webhdfs' version '1.2.5'
2020-09-04 16:02:16 +0530 [info]: gem 'fluentd' version '1.11.1'
2020-09-04 16:02:16 +0530 [info]: 'flush_interval' is configured at out side of <buffer>. 'flush_mode' is set to 'interval' to keep existing behaviour
2020-09-04 16:02:16 +0530 [debug]: 'host localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: localhost' doesn't have tag placeholder
2020-09-04 16:02:16 +0530 [debug]: 'index_name fluentd' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: fluentd' doesn't have tag placeholder
2020-09-04 16:02:16 +0530 [debug]: 'template_name ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: ' doesn't have tag placeholder
2020-09-04 16:02:16 +0530 [debug]: 'logstash_prefix logstash' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: logstash' doesn't have tag placeholder
2020-09-04 16:02:16 +0530 [debug]: 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' has timestamp placeholders,but chunk key 'time' is not configured
2020-09-04 16:02:16 +0530 [debug]: 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have tag placeholder
2020-09-04 16:02:16 +0530 [debug]: 'deflector_alias ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: ' doesn't have tag placeholder
2020-09-04 16:02:16 +0530 [debug]: 'application_name default' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: default' doesn't have tag placeholder
2020-09-04 16:02:16 +0530 [debug]: 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have tag placeholder
2020-09-04 16:02:16 +0530 [debug]: Need substitution: false
2020-09-04 16:02:16 +0530 [debug]: 'host_placeholder localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: localhost' doesn't have tag placeholder
2020-09-04 16:02:16 +0530 [warn]: define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2020-09-04 16:02:16 +0530 [info]: using configuration file: <ROOT>
  <system>
    log_level debug
  </system>
  <source>
    @type tail
    path "/mnt/Log/Startup.log"
    pos_file "/mnt/Log/Startup.log.pos"
    format multiline
    format_firstline /\d{4}-\d{1,2}-\d{1,2}/
    format1 /^(?<time>\d{4}-\d{1,2} \d{1,2}:\d{1,2}) \[(?<thread>.*)\] (?<level>[^\s]+)(?<message>.*)/
    tag "log"
    <parse>
      format_firstline /\d{4}-\d{1,2}/
      @type multiline
      format1 /^(?<time>\d{4}-\d{1,2}) \[(?<thread>.*)\] (?<level>[^\s]+)(?<message>.*)/
      unmatched_lines
    </parse>
  </source>
  <match *.**>
    @type elasticsearch
    host "localhost"
    port 9200
    include_tag_key true
    tag_key "@log_name"
    logstash_format true
    flush_interval 10s
    <buffer>
      flush_interval 10s
    </buffer>
  </match>
</ROOT>
2020-09-04 16:02:16 +0530 [info]: starting fluentd-1.11.1 pid=67918 ruby="2.4.10"
2020-09-04 16:02:16 +0530 [info]: spawn command to main:  cmdline=["/opt/td-agent/embedded/bin/ruby","-Eascii-8bit:ascii-8bit","/opt/td-agent/embedded/bin/fluentd","--log","/var/log/td-agent/td-agent.log","--daemon","/var/run/td-agent/td-agent.pid","--under-supervisor"]
2020-09-04 16:02:17 +0530 [info]: adding match pattern="*.**" type="elasticsearch"
2020-09-04 16:02:17 +0530 [info]: #0 'flush_interval' is configured at out side of <buffer>. 'flush_mode' is set to 'interval' to keep existing behaviour
2020-09-04 16:02:17 +0530 [debug]: #0 'host localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: localhost' doesn't have tag placeholder
2020-09-04 16:02:17 +0530 [debug]: #0 'index_name fluentd' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: fluentd' doesn't have tag placeholder
2020-09-04 16:02:17 +0530 [debug]: #0 'template_name ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: ' doesn't have tag placeholder
2020-09-04 16:02:17 +0530 [debug]: #0 'logstash_prefix logstash' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: logstash' doesn't have tag placeholder
2020-09-04 16:02:17 +0530 [debug]: #0 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' has timestamp placeholders,but chunk key 'time' is not configured
2020-09-04 16:02:17 +0530 [debug]: #0 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have tag placeholder
2020-09-04 16:02:17 +0530 [debug]: #0 'deflector_alias ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: ' doesn't have tag placeholder
2020-09-04 16:02:17 +0530 [debug]: #0 'application_name default' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: default' doesn't have tag placeholder
2020-09-04 16:02:17 +0530 [debug]: #0 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have tag placeholder
2020-09-04 16:02:17 +0530 [debug]: #0 Need substitution: false
2020-09-04 16:02:17 +0530 [debug]: #0 'host_placeholder localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: localhost' doesn't have tag placeholder
2020-09-04 16:02:17 +0530 [warn]: #0 Detected ES 7.x: `_doc` will be used as the document `_type`.
2020-09-04 16:02:17 +0530 [info]: adding source type="tail"
2020-09-04 16:02:17 +0530 [warn]: #0 define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2020-09-04 16:02:17 +0530 [info]: #0 starting fluentd worker pid=67935 ppid=67930 worker=0
2020-09-04 16:02:17 +0530 [debug]: #0 buffer started instance=70139276565080 stage_size=0 queue_size=0
2020-09-04 16:02:17 +0530 [debug]: #0 enqueue_thread actually running
2020-09-04 16:02:17 +0530 [debug]: #0 tailing paths: target = /mnt/Log/Startup.log | existing =
2020-09-04 16:02:17 +0530 [info]: #0 following tail of /mnt/Log/Startup.log
2020-09-04 16:02:17 +0530 [info]: #0 fluentd worker is now running worker=0
2020-09-04 16:02:17 +0530 [debug]: #0 flush_thread actually running

尽管上面的日志显示它正在拖累Startup.log。它仍然显示kibana仪表板上td-agent.log中的内容，而不显示已配置日志文件中的内容。我也可以在kibana上找到默认索引

td-agent.conf：

<system>
log_level debug
</system>

<source>
@type tail
path /mnt/Log/Startup.log
pos_file /mnt/Log/Startup.log.pos
format multiline
format_firstline /\d{4}-\d{1,2}/
format1 /^(?<time>\d{4}-\d{1,2}) \[(?<thread>.*)\] (?<level>[^\s]+)(?<message>.*)/
tag log
</source>

<match *.**>
   @type elasticsearch
    host localhost
    port 9200
    include_tag_key true
    tag_key @log_name
    logstash_format true
    flush_interval 10s
</match>

版本详细信息：

Elasticsearch版本7.8.0 Fluentd（TD代理）版本3.8.0 Kibana版本7.8.0

我已经使用rpm安装了以上所有工具：

elasticsearch-7.8.1-x86_64.rpm td-agent-3.8.0-0.el7.x86_64.rpm kibana-7.8.1-x86_64.rpm

curl -X GET“ localhost：9200 /？pretty”给出：

{
  "name" : "ncnsidapp2","cluster_name" : "elasticsearch","cluster_uuid" : "MnfFXTyZT0ahp9u4eLSW2A","version" : {
    "number" : "7.8.1","build_flavor" : "default","build_type" : "rpm","build_hash" : "b5ca9c58fb664ca8bf9e4057fc229b3396bf3a89","build_date" : "2020-07-21T16:40:44.668009Z","build_snapshot" : false,"lucene_version" : "8.5.1","minimum_wire_compatibility_version" : "6.8.0","minimum_index_compatibility_version" : "6.0.0-beta1"
  },"tagline" : "You Know,for Search"
}

我能够在kibana上找到默认索引。

这里可能是什么问题？请帮忙。

解决方法

尝试在您的部分中设置logstash_prefix。将logstash_format设置为true时，它将覆盖您设置为索引名称的内容。如果您设置logstash_prefix，则默认情况下它将以{logstash_prefix}-％y。％m。％d的格式命名索引。