log4j.xml 以不同的方式将相同的日志发送到文件和 td-agent

问题描述

如何以两种方式处理相同的日志？在文件中保存为 json 的日志被 td-agent 停止，说模式（即 json）不匹配这是我的 log4j.xml 日志配置，它将日志保存（参见下面的第一个 BOLD 注释）到 /opt/tpa/logs/SPSServer.log 并且将它们转发到 8514 端口的 fluentd（参见下面的第二个 BOLD 注释）。 ```

<!-- ================================= -->
<!-- **Preserve messages in a local file** -->
<!-- ================================= -->

<appender name="SPSServerFileAppender"
          class="com.alcatel.tpapps.common.server.logging.log4jappenders.RollingZipFileAppender">
    <errorHandler class="org.apache.log4j.helpers.OnlyOnceErrorHandler"/>
    <param name="File" value="/opt/tpa/logs/SPSServer.log"/>
    <param name="Folding" value="true"/>
    <param name="Debug" value="true"/>
    <param name="MaxFileSize" value="30MB"/>
    <param name="CompressLogs" value="true"/>
    <param name="MaxBackupIndex" value="15"/>
    <param name="MaxBackupDir" value="-1"/>
    <!-- retain backup folders for 3 months (90 days)
         (units of the following parameter are ms)-->
    <param name="MaxBackupTime" value="7776000000"/>
    <param name="SavePrevIoUsLogs" value="true"/>
    <param name="MaxArchiveFileNumber" value="500"/>
    <param name="MaxArchiveTime" value="-1"/>
    <param name="BackupDirTag" value="SPSServer"/>
    <param name="ConsoleLog" value="server_console.log"/>
    <layout class="com.alcatel.tpapps.common.server.logging.log4jappenders.UnifiedLoggingFormat">
    </layout>
    <filter class="com.alcatel.tpapps.common.utils.AddMDCFilter"/>
</appender>


 <!--**Use to forward logs to Fluentd**-->
     <appender name="SocketAppender" class="com.alcatel.tpapps.common.server.logging.log4jappenders.UnifiedFormatSocketAppender">
     <param name="ReconnectionDelay" value="10000"/>
     <param name="Application" value="SPS"/>
     <param name="Port" value="8514"/>
     <param name="RemoteHost" value="localhost"/>
    </appender>

所以 SPSServer.log 中的日志是： {"host":"sps-me1-diameterapp-0","level":"WARN","log":{"classname":"com.nokia.as.mux.reactor.Agent:76","message":"ReactorProvider Mux Agent started!","stacktrace":"","threadname":"Main-reactor"},"process":"init","service":"","time":"2021-01-08T01:21:33.470Z","timezone":"Asia/Kolkata","type":"log","system":"SPS_ME_AI_Punit","systemid":"CBAM-11ee306d636b4177b3d78c65cc15d06e"}

{"host":"sps-me1-comsvc-0","level":"INFO","log":{"classname":"com.nokia.csf.cdr.common.util.Env:17","message":"Env is not initialized.","threadname":"FelixdispatchQueue"},"process":"isDockerEnv","time":"2021-01-08T01:19:54.815Z","systemid":"CBAM-11ee306d636b4177b3d78c65cc15d06e"}

{"host":"sps-me1-oame-0","log":{"classname":"com.nokia.tpapps.spscommon.heartbeat.common.client.MemberStatusOAMClient:245","message":"HB-sps-me1-R8292-sps-me2-O> Unable to contact service for 'sps-me2'.","stacktrace":"[com.sun.xml.internal.ws.client.ClientTransportException: HTTP transport error: java.net.ConnectException: Connection refused (Connection refused),at com.sun.xml.internal.ws.transport.http.client.HttpClientTransport.getoutput(HttpClientTransport.java:117),at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:208),at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.processRequest(HttpTransportPipe.java:130),at com.sun.xml.internal.ws.transport.DeferredTransportPipe.processRequest(DeferredTransportPipe.java:124),at com.sun.xml.internal.ws.api.pipe.Fiber.__doRun(Fiber.java:1121),at com.sun.xml.internal.ws.api.pipe.Fiber._doRun(Fiber.java:1035),at com.sun.xml.internal.ws.api.pipe.Fiber.doRun(Fiber.java:1004),at com.sun.xml.internal.ws.api.pipe.Fiber.runSync(Fiber.java:862),at com.sun.xml.internal.ws.client.Stub.process(Stub.java:448),at com.sun.xml.internal.ws.client.sei.SEIStub.doProcess(SEIStub.java:178),at com.sun.xml.internal.ws.client.sei.SyncmethodHandler.invoke(SyncmethodHandler.java:93),at com.sun.xml.internal.ws.client.sei.SyncmethodHandler.invoke(SyncmethodHandler.java:77),at com.sun.xml.internal.ws.client.sei.SEIStub.invoke(SEIStub.java:147),at com.sun.proxy.$Proxy267.getHeartBeatStatus(UnkNown Source),at com.nokia.tpapps.spscommon.heartbeat.common.client.MemberStatusOAMClient.getMemberStatusesInternal(MemberStatusOAMClient.java:160),at com.nokia.tpapps.spscommon.heartbeat.common.client.MemberStatusOAMClient.getMemberStatuses(MemberStatusOAMClient.java:130),at com.nokia.tpapps.spscommon.heartbeat.common.HeartBeatMonitorSystemIntegrator.poll(HeartBeatMonitorSystemIntegrator.java:226),at com.nokia.tpapps.spscommon.heartbeat.common.HeartBeatMonitorSystemIntegrator.pollOAM(HeartBeatMonitorSystemIntegrator.java:194),at com.nokia.tpapps.spscommon.heartbeat.common.HeartBeatMonitor.lambda$poll$0(HeartBeatMonitor.java:219),at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511),at java.util.concurrent.FutureTask.run(FutureTask.java:266),at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149),at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624),at java.lang.Thread.run(Thread.java:748),Caused by: java.net.ConnectException: Connection refused (Connection refused),at java.net.PlainSocketImpl.socketConnect(Native Method),at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350),at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206),at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188),at java.net.socksSocketImpl.connect(SocksSocketImpl.java:392),at java.net.socket.connect(Socket.java:607),at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:660),at sun.net.NetworkClient.doConnect(NetworkClient.java:175),at sun.net.www.http.HttpClient.openServer(HttpClient.java:463),at sun.net.www.http.HttpClient.openServer(HttpClient.java:558),at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264),at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367),at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191),at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1205),at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056),at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177),at sun.net.www.protocol.http.HttpURLConnection.getoutputStream0(HttpURLConnection.java:1340),at sun.net.www.protocol.http.HttpURLConnection.getoutputStream(HttpURLConnection.java:1315),at sun.net.www.protocol.https.HttpsURLConnectionImpl.getoutputStream(HttpsURLConnectionImpl.java:264),at com.sun.xml.internal.ws.transport.http.client.HttpClientTransport.getoutput(HttpClientTransport.java:104),... 23 more]","threadname":"Heartbeat-4"},"process":"getMemberStatusesInternal","time":"2021-01-08T09:42:24.081Z","systemid":"CBAM-11ee306d636b4177b3d78c65cc15d06e"}

现在，我在 8514 上处理这些日志的 td-agent 配置是

  <source>
    @type tcp
    port 8514
    bind 0.0.0.0
    format /(^(?<header>[^\{]+)?(?<message>\{.+type.+\})$)|(^(?<log>[^\{].+))/
    tag system
  </source>
</worker>

<match system syslog.**>
  @type rewrite_tag_filter
  <rule>
    key message
    pattern .+
    tag nokia.logging.json
  </rule>
  <rule>
    key log
    pattern .+
    tag nokia.logging.clear
  </rule>
</match>

<filter nokia.logging.json>
  @type parser
  key_name message
  <parse>
    @type json
    time_key time
    time_type string
    keep_time_key true
  </parse>
</filter>

但是td-agent在解析log4j.xml保存在SPSServer.log中的相同日志时出现如下错误（同上）

2021-01-08 06:51:33 +0530 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not match with data '{{\"host\":\"sps-me1-diameterapp-0\",\"level\":\"WARN\",\"log\":{\"classname\":\"com.nokia.as.mux.reactor.Agent:76\",\"message\":\"ReactorProvider Mux Agent started!\",\"stacktrace\":\"\",\"threadname\":\"Main-reactor\"},\"process\":\"init\",\"service\":\"SPS\",\"time\":\"2021-01-08T01:21:33.470Z\",\"timezone\":\"Asia/Kolkata\",\"type\":\"log\",\"system\":\"SPS_ME_AI_Punit\",\"systemid\":\"CBAM-11ee306d636b4177b3d78c65cc15d06e\"}'" location=nil tag="nokia.logging.json" time=2021-01-08 06:51:33.471639265 +0530 record={"header"=>"yt\x01","message"=>"{{\"host\":\"sps-me1-diameterapp-0\",\"systemid\":\"CBAM-11ee306d636b4177b3d78c65cc15d06e\"}"}

2021-01-08 09:41:33 +0530 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not match with data '{{\"host\":\"sps-me1-comsvc-0\",\"level\":\"INFO\",\"log\":{\"classname\":\"com.nokia.csf.cdr.common.util.Env:17\",\"message\":\"Env is not initialized.\",\"threadname\":\"FelixdispatchQueue\"},\"process\":\"isDockerEnv\",\"time\":\"2021-01-08T04:11:33.780Z\",\"systemid\":\"CBAM-11ee306d636b4177b3d78c65cc15d06e\"}'" location=nil tag="nokia.logging.json" time=2021-01-08 09:41:33.781026491 +0530 record={"header"=>"yt\x01","message"=>"{{\"host\":\"sps-me1-comsvc-0\",\"systemid\":\"CBAM-11ee306d636b4177b3d78c65cc15d06e\"}"}

2021-01-08 07:08:44 +0530 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not match with data '{{\"host\":\"sps-me1-oame-0\",\"log\":{\"classname\":\"com.nokia.tpapps.spscommon.heartbeat.common.client.MemberStatusOAMClient:245\",\"message\":\"HB-sps-me1-R30-sps-me2-O> Unable to contact service for 'sps-me2'.\",\"stacktrace\":\"[com.sun.xml.internal.ws.client.ClientTransportException: HTTP transport error: java.net.ConnectException: Connection refused (Connection refused),... 23 more]\",\"threadname\":\"Heartbeat-1\"},\"process\":\"getMemberStatusesInternal\",\"time\":\"2021-01-08T01:38:44.064Z\",\"systemid\":\"CBAM-11ee306d636b4177b3d78c65cc15d06e\"}'" location=nil tag="nokia.logging.json" time=2021-01-08 07:08:44.067375936 +0530 record={"header"=>"yt\x11","message"=>"{{\"host\":\"sps-me1-oame-0\",\"systemid\":\"CBAM-11ee306d636b4177b3d78c65cc15d06e\"}"}
2021-01-08 07:09:44 +0530 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not match with data '{{\"host\":\"sps-me1-oame-0\",\"message\":\"HB-sps-me1-R60-sps-me2-O> Unable to contact service for 'sps-me2'.\",\"threadname\":\"Heartbeat-3\"},\"time\":\"2021-01-08T01:39:44.083Z\",\"systemid\":\"CBAM-11ee306d636b4177b3d78c65cc15d06e\"}'" location=nil tag="nokia.logging.json" time=2021-01-08 07:09:44.091157349 +0530 record={"header"=>"yt\x11",\"systemid\":\"CBAM-11ee306d636b4177b3d78c65cc15d06e\"}"}

解决方法

暂无找到可以解决该程序问题的有效方法，小编努力寻找整理中！

如果你已经找到好的解决方法，欢迎将解决方案带上本链接一起发送给小编。

小编邮箱:dio#foxmail.com (将#修改为@）

fluentd java java log4j td-agent xml xml xml xml xml xml