使用filebeat发送logstash时，如何使用nginx的主机覆盖logstash的主机信息？

编程问答 2022-05-29

问题描述

现在使用filebeat和logstash在k8s上发送Nginx的json日志。

Nginx的配置喜欢

Nginx.conf

http {
    log_format bucket escape=json
    '{'
        '"request_id": "$request_id",'
        '"method": "$request_method",'
        '"status": "$status",'
        '"forwarded_for": "$http_x_forwarded_for",'
        '"host": "$host",'
        '"url": "$request_uri",'
        '"referer": "$http_referer",'
        '"remote_ip": "$remote_addr",'
        '"server_ip": "$server_addr",'
        '"user_agent": "$http_user_agent",'
    '}';
}

server {
    access_log  /var/log/Nginx/access.json  bucket;
}

Filebeat 的配置：

filebeat.yml

filebeat.shutdown_timeout: 5s

filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/Nginx/access.json*
    exclude_files: ['\.gz$']
    tags: ["access"]

processors:
  - decode_json_fields:
      fields: ["message"]
      process_array: true
      max_depth: 1
      target: ""
      overwrite_keys: true
      add_error_key: false

output.logstash:
  hosts: ["logstash.default.svc.cluster.local:5044"]

此处 overwirte_keys 为真，因此它应该覆盖元数据，对吗？

Logstash 的配置：

logstash.conf

input {
  beats {
    port => 5044
  }
}

filter {
  if "access" in [tags] {
    mutate {
      add_field => { "[@Metadata][tags]" => "%{tags}" }
      remove_field => [
        "agent","event","service","log","input","fileset","ecs","container","kubernetes","@timestamp","@version","message","tags"
      ]
    }
  }
}

output {
  if "access" in [@Metadata][tags] {
    google_cloud_storage {
      bucket => "Nginx_logs"
      json_key_file => "/secrets/service_account/credentials.json"
      temp_directory => "/tmp/Nginx_logs"
      log_file_prefix => "logstash_Nginx_logs"
      max_file_size_kbytes => 1024
      output_format => "json"
      date_pattern => "%Y-%m-%dT%H:00"
      flush_interval_secs => 2
      gzip => false
      gzip_content_encoding => false
      uploader_interval_secs => 60
      include_uuid => true
      include_hostname => true
    }
  }
}

一开始效果很好。日志数据已经生成到json文件中：

{"user_agent":"Mozilla/5.0 (iPhone; cpu iPhone OS 14_2_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML,like Gecko) Mobile/15E148 [FBAN/FBIOS;FBDV/iPhone13,1;FBMD/iPhone;FBSN/iOS;FBSV/14.2.1;FBSS/3;FBID/phone;FBLC/ja_JP;FBOP/5]","forwarded_for":"1.2.3.4","host":"api.mysite.com","method":"OPTIONS","request_id":"0127054b954fe4973852e1886130a6ca","referer":"https://www.world.com/","remote_ip":"2.3.4.5","server_ip":"3.4.5.6","status":"204","url":"/api/v1/post"}

但最近，出现了这样的数据：

{"user_agent":"Mozilla/5.0 (iPhone; cpu iPhone OS 14_2_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML,"url":"/api/v1/post"}
{"host":{"name":"filebeat-adio3"}}
{"host":{"name":"filebeat-adio3"}}
{"host":{"name":"filebeat-adio3"}}

这不是常规数据。看起来 filebeat 服务器的 host 元数据已发送。但为什么？是filebeat 的错误还是logstash 的错误？ 是否有另一种好方法来过滤此主机数据以确保发送时不会与 fb/logstash 的元数据发生冲突？

解决方法

暂无找到可以解决该程序问题的有效方法，小编努力寻找整理中！

如果你已经找到好的解决方法，欢迎将解决方案带上本链接一起发送给小编。

小编邮箱:dio#foxmail.com (将#修改为@）

filebeathosthostkuberneteslogstashnginx