使用 Gelf logback appender 在 Logstash 中创建动态索引

问题描述

我正在使用 logback 与 biz.paluch.logging.gelf.logback.GelfLogbackAppender 一起登录。我目前正在运行 3 个服务，我希望其中 2 个在 logstash 中的输出为 logstash-ingest，另一个为 logstash-digest。示例：

我希望这两个有索引 logstash-ingest

服务 1

<!DOCTYPE configuration>

<configuration>
    <contextName>test</contextName>
    <jmxConfigurator/>
    <appender name="gelf" class="biz.paluch.logging.gelf.logback.GelfLogbackAppender">
        <host>udp:localhost</host>
        <port>12201</port>
        <version>1.1</version>
        <extractStackTrace>true</extractStackTrace>
        <filterStackTrace>true</filterStackTrace>
        <timestampPattern>yyyy-MM-dd HH:mm:ss,SSS</timestampPattern>
        <maximumMessageSize>8192</maximumMessageSize>

        <param name="AdditionalFields" value="tag=example1-api,INDEX_PREFIX=ingest" />
        <param name="AdditionalFieldTypes" value="tag=String,INDEX_PREFIX=String" />

        <dynamicMdcFields>(field1|field2)</dynamicMdcFields>
    </appender>

    <logger name="com.example1" level="INFO" />

    <root level="INFO">
        <appender-ref ref="gelf" />
    </root>
</configuration>

服务 2

<!DOCTYPE configuration>

<configuration>
    <contextName>test</contextName>
    <jmxConfigurator/>
    <appender name="gelf" class="biz.paluch.logging.gelf.logback.GelfLogbackAppender">
        <host>udp:localhost</host>
        <port>12201</port>
        <version>1.1</version>
        <extractStackTrace>true</extractStackTrace>
        <filterStackTrace>true</filterStackTrace>
        <timestampPattern>yyyy-MM-dd HH:mm:ss,SSS</timestampPattern>
        <maximumMessageSize>8192</maximumMessageSize>

        <param name="AdditionalFields" value="tag=example2-api,INDEX_PREFIX=String" />

        <dynamicMdcFields>(field1|field2)</dynamicMdcFields>
    </appender>

    <logger name="com.example2" level="INFO" />

    <root level="INFO">
        <appender-ref ref="gelf" />
    </root>
</configuration>

和第三个 logstash-digest

服务 3

<!DOCTYPE configuration>

<configuration>
    <contextName>test</contextName>
    <jmxConfigurator/>
    <appender name="gelf" class="biz.paluch.logging.gelf.logback.GelfLogbackAppender">
        <host>udp:localhost</host>
        <port>12201</port>
        <version>1.1</version>
        <extractStackTrace>true</extractStackTrace>
        <filterStackTrace>true</filterStackTrace>
        <timestampPattern>yyyy-MM-dd HH:mm:ss,SSS</timestampPattern>
        <maximumMessageSize>8192</maximumMessageSize>

        <param name="AdditionalFields" value="tag=example3-api,INDEX_PREFIX=digest" />
        <param name="AdditionalFieldTypes" value="tag=String,INDEX_PREFIX=String" />

        <dynamicMdcFields>(field1|field2)</dynamicMdcFields>
    </appender>

    <logger name="com.example3" level="INFO" />

    <root level="INFO">
        <appender-ref ref="gelf" />
    </root>
</configuration>

这是我的 logstash.conf，但我不确定如何配置它以使用 INDEX_PREFIX 来区分这三个服务的输出。

input {

  gelf {
    id => "gelf"
    use_udp => true
    use_tcp => false
  }
}

## filters???

output {
    elasticsearch {
        hosts => ["es01:9200"]
        user => "elastic"
        password => "changeme"
        index => "logstash-{%}" ## what am I missing here?
    }
}

解决方法

我一定是对配置进行了太长时间的调整并更改了太多内容，最后我不知道什么才是真正有效的。第二天，带着清醒的头脑，再次尝试，显然我的问题的答案是index => "logstash-%{INDEX_PREFIX}" 我很确定我尝试过这个，但可能与其他失败的东西结合使用。

gelf logback logback logging