我有一些catchall日志文件,格式如下：
timestamp event summary foo details account name: userA bar more details timestamp event summary baz details account name: userB qux more details timestamp etc.
我想在日志文件中搜索userB,如果找到,则从前面的时间戳回显到(但不包括)以下时间戳.可能会有几个与我的搜索匹配的事件.能够在每场比赛中回应某种—开始—和—结束 – 这将是一件好事.
这对pcregrep -M来说是完美的,对吧？问题是,GnuWin32的pcregrep在多行regexps搜索大文件时崩溃,这些全能日志可能是100兆或更多.
我试过的
到目前为止,我的hackish解决方法是使用grep -B15 -A30找到匹配的行并打印周围的内容,然后将现在更易管理的块管道输入pcregrep进行抛光.问题是某些事件少于十行,而其他事件少于30行;我遇到了一些意外的结果,遇到了较短的事件.
:parselog <username> <logfile> set silent=1 set count=0 set deez=20\d\d-\d\d-\d\d \d\d:\d\d:\d\d echo Searching %~2 for records containing %~1... for /f "delims=" %%I in ( 'grep -P -i -B15 -A30 ":\s+\b%~1\b(@mydomain\.ext)?$" "%~2" ^| pcregrep -M -i "^%deez%(.|\n)+?\b%~1\b(@mydomain\.ext|\r?\n)(.|\n)+?\n%deez%" 2^>NUL' ) do ( echo(%%I| findstr "^20[0-9][0-9]-[0-9][0-9]-[0-9][0-9].[0-9][0-9]:[0-9][0-9]:[0-9][0-9]" >NUL && ( if defined silent ( set silent= set found=1 set /a "count+=1" echo; echo ---------------start of record !count!------------- ) else ( set silent=1 echo ----------------end of record !count!-------------- echo; ) ) if not defined silent echo(%%I ) goto :EOF
有一个更好的方法吗？我遇到了一个看起来很有趣的awk命令,例如：
awk "/start pattern/,/end pattern/" logfile
……但它也需要匹配中间模式.不幸的是,我对awk语法并不熟悉.有什么建议么？
Ed Morton建议我提供一些示例记录和预期输出.
示例全能
2013-03-25 08:02:32 Auth.Critical 169.254.8.110 Mar 25 08:02:32 dc3 MSWinEventLog 2 Security 11730158 Mon Mar 25 08:02:28 2013 529 Security NT AUTHORITY\SYstem N/A Audit Failure dc3 2 logon Failure: Reason: UnkNown user name or bad password User Name: user5f Domain: MYDOMAIN logon Type: 3 logon Process: Advapi Authentication Package: Negotiate Workstation Name: dc3 Caller User Name: dc3$ Caller Domain: MYDOMAIN Caller logon ID: (0x0,0x3E7) Caller Process ID: 400 Transited Services: - Source Network Address: 169.254.7.86 Source Port: 40838 2013-03-25 08:02:32 Auth.Critical 169.254.8.110 Mar 25 08:02:32 dc3 MSWinEventLog 2 Security 11730159 Mon Mar 25 08:02:29 2013 680 Security NT AUTHORITY\SYstem N/A Audit Failure dc3 9 logon attempt by: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0 logon account: USER6Q Source Workstation: dc3 Error Code: 0xC0000234 2013-03-25 08:02:32 Auth.Critical 169.254.8.110 Mar 25 08:02:32 dc3 MSWinEventLog 2 Security 11730160 Mon Mar 25 08:02:29 2013 539 Security NT AUTHORITY\SYstem N/A Audit Failure dc3 2 logon Failure: Reason: Account locked out User Name: USER6Q@MYDOMAIN.TLD Domain: MYDOMAIN logon Type: 3 logon Process: Advapi Authentication Package: Negotiate Workstation Name: dc3 Caller User Name: dc3$ Caller Domain: MYDOMAIN Caller logon ID: (0x0,0x3E7) Caller Process ID: 400 Transited Services: - Source Network Address: 169.254.7.89 Source Port: 55314 2013-03-25 08:02:32 Auth.Notice 169.254.5.62 Mar 25 08:36:38 DC4.mydomain.tld MSWinEventLog 5 Security 201326798 Mon Mar 25 08:36:37 2013 4624 Microsoft-Windows-Security-Auditing N/A Audit Success DC4.mydomain.tld 12544 An account was successfully logged on. Subject: Security ID: S-1-0-0 Account Name: - Account Domain: - logon ID: 0x0 logon Type: 3 New logon: Security ID: S-1-5-21-606747145-1409082233-725345543-160838 Account Name: DEPTACCT16$ Account Domain: MYDOMAIN logon ID: 0x1158e6012c logon GUID: {BCC72986-82A0-4EE9-3729-847BA6FA3A98} Process information: Process ID: 0x0 Process Name: - Network information: Workstation Name: Source Network Address: 169.254.114.62 Source Port: 42183 Detailed Authentication information: logon Process: Kerberos Authentication Package: Kerberos Transited Services: - Package Name (NTLM only): - Key Length: 0 This event is generated when a logon session is created. It is generated on the computer that was accessed. The subject fields indicate... 2013-03-25 08:02:32 Auth.Critical 169.254.8.110 Mar 25 08:02:32 dc3 MSWinEventLog 2 Security 11730162 Mon Mar 25 08:02:30 2013 675 Security NT AUTHORITY\SYstem N/A Audit Failure dc3 9 Pre-authentication Failed: User Name: USER8Y User ID: %{S-1-5-21-606747145-1409082233-725345543-3904} Service Name: krbtgt/MYDOMAIN Pre-Authentication Type: 0x0 Failure Code: 0x19 Client Address: 169.254.87.158 2013-03-25 08:02:32 Auth.Critical etc.
示例命令
call :parselog user6q \\path\to\catch-all.log
预期结果
---------------start of record 1------------- 2013-03-25 08:02:32 Auth.Critical 169.254.8.110 Mar 25 08:02:32 dc3 MSWinEventLog 2 Security 11730159 Mon Mar 25 08:02:29 2013 680 Security NT AUTHORITY\SYstem N/A Audit Failure dc3 9 logon attempt by: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0 logon account: USER6Q Source Workstation: dc3 Error Code: 0xC0000234 ---------------end of record 1------------- ---------------start of record 2------------- 2013-03-25 08:02:32 Auth.Critical 169.254.8.110 Mar 25 08:02:32 dc3 MSWinEventLog 2 Security 11730160 Mon Mar 25 08:02:29 2013 539 Security NT AUTHORITY\SYstem N/A Audit Failure dc3 2 logon Failure: Reason: Account locked out User Name: USER6Q@MYDOMAIN.TLD Domain: MYDOMAIN logon Type: 3 logon Process: Advapi Authentication Package: Negotiate Workstation Name: dc3 Caller User Name: dc3$ Caller Domain: MYDOMAIN Caller logon ID: (0x0,0x3E7) Caller Process ID: 400 Transited Services: - Source Network Address: 169.254.7.89 Source Port: 55314 ---------------end of record 2-------------
这就是GNU awk所需要的(对于IGnorECASE)：
$cat tst.awk
function prtRecord() {
if (record ~ regexp) {
printf "-------- start of record %d --------%s",++numRecords,ORS
printf "%s",record
printf "--------- end of record %d ---------%s%s",numRecords,ORS,ORS
}
record = ""
}
BEGIN{ IGnorECASE=1 }
/^[[:digit:]]+-[[:digit:]]+-[[:digit:]]+/ { prtRecord() }
{ record = record $0 ORS }
END { prtRecord() }
或任何awk：
$cat tst.awk
function prtRecord() {
if (tolower(record) ~ tolower(regexp)) {
printf "-------- start of record %d --------%s",ORS
}
record = ""
}
/^[[:digit:]]+-[[:digit:]]+-[[:digit:]]+/ { prtRecord() }
{ record = record $0 ORS }
END { prtRecord() }
无论哪种方式,您都可以在UNIX上运行它：
$awk -v regexp=user6q -f tst.awk file
我不知道Windows语法,但我希望它非常相似,如果不相同的话.
请注意在脚本中使用tolower()使比较小写的两边都匹配,因此匹配不区分大小写.如果你可以传入一个正确的搜索正则表达式,那么你不需要在比较的任何一侧调用tolower(). nbd,它可能会略微加快脚本速度.
$awk -v regexp=user6q -f tst.awk file -------- start of record 1 -------- 2013-03-25 08:02:32 Auth.Critical 169.254.8.110 Mar 25 08:02:32 dc3 MSWinEventLog 2 Security 11730159 Mon Mar 25 08:02:29 2013 680 Security NT AUTHORITY\SYstem N/A Audit Failure dc3 9 logon attempt by: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0 logon account: USER6Q Source Workstation: dc3 Error Code: 0xC0000234 --------- end of record 1 --------- -------- start of record 2 -------- 2013-03-25 08:02:32 Auth.Critical 169.254.8.110 Mar 25 08:02:32 dc3 MSWinEventLog 2 Security 11730160 Mon Mar 25 08:02:29 2013 539 Security NT AUTHORITY\SYstem N/A Audit Failure dc3 2 logon Failure: Reason: Account locked out User Name: USER6Q@MYDOMAIN.TLD Domain: MYDOMAIN logon Type: 3 logon Process: Advapi Authentication Package: Negotiate Workstation Name: dc3 Caller User Name: dc3$ Caller Domain: MYDOMAIN Caller logon ID: (0x0,0x3E7) Caller Process ID: 400 Transited Services: - Source Network Address: 169.254.7.89 Source Port: 55314 --------- end of record 2 ---------
