尝试通过 pykafka 连接到 kafka 时，在所有 docker 映像中出现导入 _rd_kafka 错误

编程问答 2022-05-26

问题描述

您好，我尝试使用多个 docker 映像，例如 Ubuntu 和 python:3.8-alpine 等。在尝试通过 pykafka 库连接到我的 kafka 集群 (2.7) 时，到处都出现如下错误。

环境信息：

kafka server : 2.7 (installed via strimzi kafka in EKS)
kafka clinet: pykafka (2.8.0)
python version: 3.8,3.7 in all version i get same error

注意：只有当我在外部运行时，即直接从我的机器运行代码在容器内运行时，才会发生此错误，然后它工作正常

INFO:pykafka.topic:Could not load pykafka.rdkafka extension.
DEBUG:pykafka.topic:Traceback:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/pykafka/topic.py",line 43,in <module>
    from . import rdkafka
  File "/usr/local/lib/python3.8/site-packages/pykafka/rdkafka/__init__.py",line 1,in <module>
    from .producer import RdKafkaProducer
  File "/usr/local/lib/python3.8/site-packages/pykafka/rdkafka/producer.py",line 7,in <module>
    from . import _rd_kafka
ImportError: cannot import name '_rd_kafka' from partially initialized module 'pykafka.rdkafka' (most likely due to a circular import) (/usr/local/lib/python3.8/site-pac
INFO:pykafka.cluster:broker version is too old to use automatic API version discovery. Falling back to hardcoded versions list.
DEBUG:pykafka.cluster:Updating cluster,attempt 1/3
DEBUG:pykafka.connection:Connecting to kafka-kafka-bootstrap.kafka:9093
INFO:pykafka.connection:Attempt 0: Failed to connect to kafka-kafka-bootstrap.kafka:9093
INFO:pykafka.connection:[Errno 2] No such file or directory
INFO:pykafka.connection:retrying in 300ms.
INFO:pykafka.connection:Attempt 1: Failed to connect to kafka-kafka-bootstrap.kafka:9093
INFO:pykafka.connection:[Errno 2] No such file or directory
INFO:pykafka.connection:retrying in 300ms.
INFO:pykafka.connection:Attempt 2: Failed to connect to kafka-kafka-bootstrap.kafka:9093
INFO:pykafka.connection:[Errno 2] No such file or directory
INFO:pykafka.connection:retrying in 300ms.
WARNING:pykafka.broker:Failed to connect to broker at kafka-kafka-bootstrap.kafka:9093. Check the `listeners` property in server.config.

解决方法

这适用于 centos docker 映像，但不确定为什么不适用于 ubunut python3.8-alpine 等其他映像。

pykafkapython-3.xstrimzi