Logstash多行无法正确解析Oracle日志

编程问答 2022-06-10

问题描述

我一直在使用Logstash管道进行测试，以处理具有以下格式的Oracle多行审核日志：

Audit file /u01/app/oracle/admin/DEVINST/adump/DEVINST_ora_15460_20201001230100743853143795.aud
Oracle Database 12c Standard Edition Release 12.2.0.1.0 - 64bit DEV
Build label:    RDBMS_12.2.0.1.0_LINUX.X64_170125
ORACLE_HOME:    /u01/app/oracle/product/12.2.0/dbhome_1
System name:    Linux
Node name:      testdevserver
Release:        3.10.0-862.14.4.el7.x86_64
Version:        #1 SMP Fri Sep 21 09:07:21 UTC 2018
Machine:        x86_64
Instance name: DEVINST
Redo thread mounted by this instance: 1
Oracle process number: 57
Unix process pid: 15460,image: oracle@testdevserver (TNS V1-V3)

Thu Oct  1 23:01:00 2020 +00:00
LENGTH : '275'
ACTION :[7] 'CONNECT'
DATABASE USER:[1] '/'
PRIVILEGE :[6] 'SYSDBA'
CLIENT USER:[9] 'test_user'
CLIENT TERMINAL:[5] 'pts/0'
STATUS:[1] '0'
DBID:[10] '1762369616'
SESSIONID:[10] '4294967295'
USERHOST:[21] 'testdevserver'
CLIENT ADDRESS:[0] ''
ACTION NUMBER:[3] '100'

Thu Oct  1 23:01:00 2020 +00:00
LENGTH : '296'
ACTION :[29] 'SELECT STATUS FROM V$INSTANCE'
DATABASE USER:[1] '/'
PRIVILEGE :[6] 'SYSDBA'
CLIENT USER:[9] 'test_user'
CLIENT TERMINAL:[5] 'pts/0'
STATUS:[1] '0'
DBID:[10] '1762369616'
SESSIONID:[10] '4294967295'
USERHOST:[21] 'testdevserver'
CLIENT ADDRESS:[0] ''
ACTION NUMBER:[1] '3'

我的/etc/logstash/conf.d/25-filter.conf：

filter {
    grok {
      match => { "message" => "(?<day>^[A-Za-z]{3}) (?<month>[A-Za-z]{3}) (?<monthday>[0-9]{2}) (?<hour>[0-9]{2}):(?<min>[0-9]{2}):(?<sec>[0-9]{2}) (?<year>[0-9]{4}) %{GREEDYDATA:message}" }
      overwrite => [ "message" ]
      add_tag => [ "oracle_audit" ]
    }
    grok {
    match => { "ACTION :\[[0-9]*\] '(?<ora_audit_action>.*)'.*DATABASE USER:\[[0-9]*\] '(?<ora_audit_dbuser>.*)'.*PRIVILEGE :\[[0-9]*\] '(?<ora_audit_priv>.*)'.*CLIENT USER:\[[0-9]*\] '(?<ora_audit_osuser>.*)'.*CLIENT TERMINAL:\[[0-9]*\] '(?<ora_audit_term>.*)'.*STATUS:\[[0-9]*\] '(?<ora_audit_status>.*)'.*DBID:\[[0-9]*\] '(?<ora_audit_dbid>.*)'.*SESSIONID:\[[0-9]*\] '(?<ora_audit_sessionid>.*)'.*USERHOST:\[[0-9]*\] '(?<ora_audit_dbhost>.*)'.*CLIENT ADDRESS:\[[0-9]*\] '(?<ora_audit_clientaddr>.*)'.*ACTION NUMBER:\[[0-9]*\] '(?<ora_audit_actionnum>.*)'" }
    }
    grok {
      match => { "source" => [ ".*/[a-zA-Z0-9_#$]*_[a-z0-9]*_(?<ora_audit_derived_pid>[0-9]*)_[0-9]*\.aud" ] }
    }
    mutate {
      add_field => { "timestamp" => "%{year}-%{day}-%{monthday} %{hour}:%{min}:%{sec}" }
    }
    date {
      locale => "en"
      match => [ "timestamp","YYYY-MMM-dd HH:mm:ss" ]
    }

    mutate {
      remove_field => [ "timestamp","year","day","monthday","hour","min","sec" ]
    }
}

我的/etc/logstash/conf.d/000-file-in.conf文件：

input {
    file {
        path => [ "/tmp/testora" ]
        start_position => "beginning"
        codec => multiline
        {
                pattern => "^[A-Za-z]{3} [A-Za-z]{3} [0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2} [0-9]{4}"
                negate => "true"
                what => "previous"
         }
    }
}

然后我通过运行进行测试：

/usr/share/logstash/bin/logstash -r -f /etc/logstash/conf.d/：

....

[INFO ] 2020-10-05 11:16:30.656 [Converge PipelineAction::Reload<main>] reload - Reloading pipeline {"pipeline.id"=>:main}
[INFO ] 2020-10-05 11:16:30.662 [Converge PipelineAction::Reload<main>] observingtail - QUIT - closing all files and shutting down.
{
         "ora_audit_dbid" => "1762369616","ora_audit_actionnum" => "3","ora_audit_sessionid" => "4294967295","@version" => "1","tags" => [
        [0] "multiline",[1] "_grokparsefailure",[2] "_dateparsefailure"
    ],"path" => "/tmp/testora","ora_audit_action" => [
        [0] "275'\nACTION :[7] 'CONNECT'\nDATABASE USER:[1] '/'\nPRIVILEGE :[6] 'SYSDBA'\nCLIENT USER:[9] 'test_user'\nCLIENT TERMINAL:[5] 'pts/0'\nSTATUS:[1] '0'\nDBID:[10] '1762369616'\nSESSIONID:[10] '4294967295'\nUSERHOST:[21] 'testdevserver'\nCLIENT ADDRESS:[0] ''\nACTION NUMBER:[3] '100'\nThu Oct  1 23:01:00 2020 +00:00\nLENGTH : '296",[1] "SELECT STATUS FROM V$INSTANCE"
    ],"@timestamp" => 2020-10-05T01:16:30.889Z,"ora_audit_priv" => "SYSDBA","message" => "Audit file /u01/app/oracle/admin/DEVINST/adump/DEVINST_ora_15460_20201001230100743853143795.aud\nOracle Database 12c Standard Edition Release 12.2.0.1.0 - 64bit DEV\nBuild label:    RDBMS_12.2.0.1.0_LINUX.X64_170125\nORACLE_HOME:    /u01/app/oracle/product/12.2.0/dbhome_1\nSystem name:    Linux\nNode name:      testdevserver\nRelease:        3.10.0-862.14.4.el7.x86_64\nVersion:        #1 SMP Fri Sep 21 09:07:21 UTC 2018\nMachine:        x86_64\nInstance name: DEVINST\nRedo thread mounted by this instance: 1\nOracle process number: 57\nUnix process pid: 15460,image: oracle@testdevserver (TNS V1-V3)\nThu Oct  1 23:01:00 2020 +00:00\nLENGTH : '275'\nACTION :[7] 'CONNECT'\nDATABASE USER:[1] '/'\nPRIVILEGE :[6] 'SYSDBA'\nCLIENT USER:[9] 'test_user'\nCLIENT TERMINAL:[5] 'pts/0'\nSTATUS:[1] '0'\nDBID:[10] '1762369616'\nSESSIONID:[10] '4294967295'\nUSERHOST:[21] 'testdevserver'\nCLIENT ADDRESS:[0] ''\nACTION NUMBER:[3] '100'\nThu Oct  1 23:01:00 2020 +00:00\nLENGTH : '296'\nACTION :[29] 'SELECT STATUS FROM V$INSTANCE'\nDATABASE USER:[1] '/'\nPRIVILEGE :[6] 'SYSDBA'\nCLIENT USER:[9] 'test_user'\nCLIENT TERMINAL:[5] 'pts/0'\nSTATUS:[1] '0'\nDBID:[10] '1762369616'\nSESSIONID:[10] '4294967295'\nUSERHOST:[21] 'testdevserver'\nCLIENT ADDRESS:[0] ''\nACTION NUMBER:[1] '3'","ora_audit_dbhost" => "testdevserver","host" => "myhost","ora_audit_dbuser" => "/","ora_audit_osuser" => "test_user","ora_audit_term" => "pts/0","ora_audit_status" => "0"
}

不幸的是，这不是我所期望的。不知何故，它没有对消息进行分块和正确地解析消息。我期待的是类似的东西：

         "ora_audit_action" => "CONNECT","ora_audit_dbid" => "1762369616","ora_audit_status" => "0","ora_audit_clientaddr" => "","message" => "Thu Oct  1 23:01:00 2020 +00:00\nLENGTH : '275'\nACTION :[7] 'CONNECT'\nDATABASE USER:[1] '/'\nPRIVILEGE :[6] 'SYSDBA'\nCLIENT USER:[9] 'test_user'\nCLIENT TERMINAL:[5] 'pts/0'\nSTATUS:[1] '0'\nDBID:[10] '1762369616'\nSESSIONID:[10] '4294967295'\nUSERHOST:[21] 'testdevserver'\nCLIENT ADDRESS:[0] ''\nACTION NUMBER:[3] '100'\nThu Oct  1 23:01:00 2020 +00:00\nLENGTH : '296'\nACTION :[29] 'SELECT STATUS FROM V$INSTANCE'\nDATABASE USER:[1] '/'\nPRIVILEGE :[6] 'SYSDBA'\nCLIENT USER:[9] 'test_user'\nCLIENT TERMINAL:[5] 'pts/0'\nSTATUS:[1] '0'\nDBID:[10] '1762369616'\nSESSIONID:[10] '4294967295'\nUSERHOST:[21] 'testdevserver'\nCLIENT ADDRESS:[0] ''\nACTION NUMBER:[1] '3'"

我看到了其中一些信息，但是我也希望它能吐出2条“大块”消息（与模式^[A-Za-z]{3} [A-Za-z]{3} [0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2} [0-9]{4}相匹配，作为行的开头，例如Thu Oct 1 23:01:00 2020 +00:00） -相反，它看起来只看到一个？我认为我的模式匹配可能是这里的问题，如果有人可以提供任何提示，谢谢。

此外，我不确定是什么导致了这些错误[1] "_grokparsefailure",[2] "_dateparsefailure"-显然，它无法正确解析内容，但我只是不知道该怎么做。

帮助：（

感谢J

解决方法

暂无找到可以解决该程序问题的有效方法，小编努力寻找整理中！

如果你已经找到好的解决方法，欢迎将解决方案带上本链接一起发送给小编。

小编邮箱:dio#foxmail.com (将#修改为@）

elasticsearchlogstashlogstash-grok