使用Kinesis Agent从单行收集的自定义日志写入Kinesis Firehose

问题描述

所以我有一行日志一起生成。

我想在我的EC2实例上使用Kinesis Agent将这些数据写入Kinesis Firehose。

我已经将自定义matchPattern与自定义fieldNames一起使用，但似乎代理不能从一行中获取多个记录。

我在python脚本生成的伪数据上对此进行了测试。此处的示例：

77.253.46.255 - - [03/Sep/2020:09:46:12 +0000] "PUT /explore HTTP/1.0" 200 4957 "http://www.moore.biz/post.PHP" "Mozilla/5.0 (Android 2.3.5; Mobile; rv:48.0) Gecko/48.0 Firefox/48.0"  128.206.196.208 - - [03/Sep/2020:09:49:14 +0000] "DELETE /search/tag/list HTTP/1.0" 200 4998 "http://hernandez.com/explore/faq.PHP" "Mozilla/5.0 (Windows CE) AppleWebKit/533.1 (KHTML,like Gecko) Chrome/29.0.806.0 Safari/533.1" 75.217.144.114 - - [03/Sep/2020:09:51:04 +0000] "GET /wp-content HTTP/1.0" 200 5047 "http://www.mason.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_8) AppleWebKit/536.2 (KHTML,like Gecko) Chrome/33.0.891.0 Safari/536.2"  17.39.118.208 - - [03/Sep/2020:09:52:27 +0000] "PUT /explore HTTP/1.0" 404 5094 "https://ellison-saunders.com/faq/" "Mozilla/5.0 (Android 4.4.2; Mobile; rv:42.0) Gecko/42.0 Firefox/42.0"  9.101.23.95 - - [03/Sep/2020:09:57:04 +0000] "PUT /wp-admin HTTP/1.0" 200 5034 "http://mcmahon-cooper.com/app/main/posts/login.PHP" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10_6_9 rv:3.0; af-ZA) AppleWebKit/535.43.1 (KHTML,like Gecko) Version/5.1 Safari/535.43.1"    43.182.24.1 - - [03/Sep/2020:10:01:55 +0000] "GET /posts/posts/explore HTTP/1.0" 200 4981 "https://pacheco.com/home.html" "Mozilla/5.0 (iPhone; cpu iPhone OS 9_3_6 like Mac OS X) AppleWebKit/534.2 (KHTML,like Gecko) FxiOS/12.5b2439.0 Mobile/84P693 Safari/534.2"

我尝试了一些不同的代理配置。这是一个示例：

{
  "cloudwatch.emiTMetrics": true,"kinesis.endpoint": "","firehose.endpoint": "https://firehose.eu-west-1.amazonaws.com","flows": [
    {
      "filePattern": "/tmp/access_log*","deliveryStream": "web-log-ingestion-stream","initialPosition": "START_OF_FILE","dataProcessingOptions": [
        {
       "optionName": "LOGTOJSON","logFormat": "COMMONAPACHELOG","matchPattern": "([\\d.]+) (\\S+) (\\S+) \\[([\\w:/]+\\s[+\\-]\\d{4})\\] \"(.+?)\" (\\d{3})","customFieldNames": ["host","ident","authuser","datetime","request","response"]
        }
      ]
    }
  ]
}

任何建议如何解决此问题？

解决方法

暂无找到可以解决该程序问题的有效方法，小编努力寻找整理中！

如果你已经找到好的解决方法，欢迎将解决方案带上本链接一起发送给小编。

小编邮箱:dio#foxmail.com (将#修改为@）

amazon-kinesis regex