问题描述
我创建了一个 AWS MSK 集群，使用文档中提到的 SASL/SCRAM Authentication。
我正在尝试使用 fluentd-plugin-kafka
将日志从 fluentd 发送到此 AWS MSK<match **>
@type kafka2
# list of seed brokers
brokers "b-1.devclient3msk.r6v5cv.c3.kafka.us-east-2.amazonaws.com:9096,b-2.devclient3msk.r6v5cv.c3.kafka.us-east-2.amazonaws.com:9096"
sasl_over_ssl false
username "<myusername>"
password "********"
scram_mechanism "sha256"
# topic settings
default_topic messages
topic_key kubernetes
get_kafka_client_log true
<format>
@type json
</format>
# Optional. See https://docs.fluentd.org/v/1.0/configuration/inject-section
<inject>
tag_key tag
time_key time
</inject>
# See fluentd document for buffer related parameters: https://docs.fluentd.org/v/1.0/configuration/buffer-section
# Buffer chunk key should be same with topic_key. If value is not found in the record,default_topic is used.
<buffer topic>
flush_interval 10s
</buffer>
# ruby-kafka producer options
max_send_retries 5
</match>
fluentd 插件无法与种子代理通信并遇到超时错误，如下所示
2021-05-26 15:59:00 +0000 [warn]: #0 Failed to flush the buffer. retry_time=0 next_retry_seconds=2021-05-26 15:59:01 +0000 chunk="5c33db6ccf6627f81dd216edd689b9ca" error_class=Kafka::ConnectionError error="Could not connect to any of the seed brokers:\n- kafka://b-2.devclient3msk.r6v5cv.c3.kafka.us-east-2.amazonaws.com:9096: Connection timed out\n- kafka://b-1.devclient3msk.r6v5cv.c3.kafka.us-east-2.amazonaws.com:9096: Connection timed out"
2021-05-26 15:59:00 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:448:in `fetch_cluster_info'
2021-05-26 15:59:00 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:402:in `cluster_info'
2021-05-26 15:59:00 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:102:in `refresh_Metadata!'
2021-05-26 15:59:00 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:56:in `add_target_topics'
2021-05-26 15:59:00 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/fluent-plugin-kafka-0.16.3/lib/fluent/plugin/kafka_producer_ext.rb:93:in `initialize'
2021-05-26 15:59:00 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/fluent-plugin-kafka-0.16.3/lib/fluent/plugin/kafka_producer_ext.rb:60:in `new'
2021-05-26 15:59:00 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/fluent-plugin-kafka-0.16.3/lib/fluent/plugin/kafka_producer_ext.rb:60:in `topic_producer'
2021-05-26 15:59:00 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/fluent-plugin-kafka-0.16.3/lib/fluent/plugin/out_kafka2.rb:233:in `write'
2021-05-26 15:59:00 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/fluentd-1.12.4/lib/fluent/plugin/output.rb:1138:in `try_flush'
2021-05-26 15:59:00 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/fluentd-1.12.4/lib/fluent/plugin/output.rb:1450:in `flush_thread_run'
2021-05-26 15:59:00 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/fluentd-1.12.4/lib/fluent/plugin/output.rb:462:in `block (2 levels) in start'
2021-05-26 15:59:00 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/fluentd-1.12.4/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2021-05-26 15:59:01 +0000 [info]: #0 New topics added to target list: messages
2021-05-26 15:59:01 +0000 [info]: #0 Fetching cluster Metadata from kafka://b-1.devclient3msk.r6v5cv.c3.kafka.us-east-2.amazonaws.com:9096
我不确定我是否遗漏了什么，或者这不是一个正确的配置。
解决方法
我遇到了同样的问题，发现无法通过公共终端节点从 Amazon MSK 集群生成或使用它，因为 Amazon MSK 不支持公共终端节点。它始终是专用连接。
可能的解决方案：有多种方法可以连接到 VPC 之外的 AWS MSK 集群。
- VPN：https://docs.aws.amazon.com/vpc/latest/userguide/vpn-connections.html
- VPC 对等互连：https://docs.aws.amazon.com/vpc/latest/peering/what-is-vpc-peering.html
- VPC 中转网关：https://docs.aws.amazon.com/vpc/latest/tgw/what-is-transit-gateway.html
- AWS Direct Connect：https://aws.amazon.com/directconnect/
- REST 代理：REST 代理可以安装在您的 VPC 内运行的实例上。 REST 代理允许您的生产者和消费者通过 HTTP API 请求与集群通信。
资源： [常见问题]：https://aws.amazon.com/msk/faqs/