问题描述
我正在尝试从本地计算机向Ubuntu vm生成kafka消息。
Telnet commad表示已成功连接到主机。
PS C:\Users\harshal> Telnet harshal-VirtualBox 9092
Successfully connected to Host: "harshal-VirtualBox" on Port: "9092"
broker.properties:
request.timeout.ms=1200000
max.block.ms=1200000
server.properties:
broker.id=0
listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://harshal-VirtualBox:9092,PLAINTEXT://192.168.43.171:9092
我正在使用以下代码，该代码在Ubuntu vm上运行良好，但是当我尝试从主机上执行该代码时却出现错误。
from kafka import KafkaProducer,KafkaConsumer
from time import sleep
producer =KafkaProducer(bootstrap_servers=['harshal-VirtualBox:9092'],api_version=(0,10,1))
for e in range(10):
producer.send('TestTopicNew',b'sending from host ')
sleep(3)
获取 KafkaTimeoutError：KafkaTimeoutError：60.0秒后无法更新元数据。
以下是调试日志。
DEBUG:kafka.producer.kafka:Starting the Kafka producer
DEBUG:kafka.metrics.metrics:Added sensor with name connections-closed
DEBUG:kafka.metrics.metrics:Added sensor with name connections-created
DEBUG:kafka.metrics.metrics:Added sensor with name select-time
DEBUG:kafka.metrics.metrics:Added sensor with name io-time
DEBUG:kafka.metrics.metrics:Added sensor with name bufferpool-wait-time
DEBUG:kafka.metrics.metrics:Added sensor with name batch-size
DEBUG:kafka.metrics.metrics:Added sensor with name compression-rate
DEBUG:kafka.metrics.metrics:Added sensor with name queue-time
DEBUG:kafka.metrics.metrics:Added sensor with name produce-throttle-time
DEBUG:kafka.metrics.metrics:Added sensor with name records-per-request
DEBUG:kafka.metrics.metrics:Added sensor with name bytes
---------------------------------------------------------------------------
KafkaTimeoutError Traceback (most recent call last)
<ipython-input-69-157f243dc84c> in <module>
10
11 producer =KafkaProducer(bootstrap_servers=['harshal-VirtualBox:9092'],1))
---> 12 producer.send('TestTopicNew',b'sending from host')
13
C:\anaconda\lib\site-packages\kafka\producer\kafka.py in send(self,topic,value,key,headers,partition,timestamp_ms)
562 key_bytes = value_bytes = None
563 try:
--> 564 self._wait_on_Metadata(topic,self.config['max_block_ms'] / 1000.0)
565
566 key_bytes = self._serialize(
C:\anaconda\lib\site-packages\kafka\producer\kafka.py in _wait_on_Metadata(self,max_wait)
689 if not Metadata_event.is_set():
690 raise Errors.KafkaTimeoutError(
--> 691 "Failed to update Metadata after %.1f secs." % (max_wait,))
692 elif topic in self._Metadata.unauthorized_topics:
693 raise Errors.TopicAuthorizationFailedError(topic)
KafkaTimeoutError: KafkaTimeoutError: Failed to update Metadata after 60.0 secs.
DEBUG:kafka.metrics.metrics:Added sensor with name record-retries
INFO:kafka.producer.kafka:Closing the Kafka producer with 4294967.0 secs timeout.
WARNING:kafka.client:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connected> [IPv4 ('127.0.0.1',9092)]> timed out after 30000 ms. Closing connection.
WARNING:kafka.client:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connected> [IPv4 ('127.0.0.1',9092)]> timed out after 30000 ms. Closing connection.
DEBUG:kafka.metrics.metrics:Added sensor with name errors
WARNING:kafka.client:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connected> [IPv4 ('127.0.0.1',9092)]> timed out after 30000 ms. Closing connection.
INFO:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connected> [IPv4 ('127.0.0.1',9092)]>: Closing connection. [Error 7] RequestTimedOutError: Request timed out after 30000 ms
INFO:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connected> [IPv4 ('127.0.0.1',9092)]>: Closing connection. [Error 7] RequestTimedOutError: Request timed out after 30000 ms
WARNING:kafka.client:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connected> [IPv4 ('127.0.0.1',9092)]> timed out after 30000 ms. Closing connection.
DEBUG:kafka.metrics.metrics:Added sensor with name record-size-max
INFO:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connected> [IPv4 ('127.0.0.1',9092)]>: Closing connection. [Error 7] RequestTimedOutError: Request timed out after 30000 ms
DEBUG:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connected> [IPv4 ('127.0.0.1',9092)]>: reconnect backoff 0.053733828642659076 after 1 failures
DEBUG:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connected> [IPv4 ('127.0.0.1',9092)]>: reconnect backoff 0.0555060173039363 after 1 failures
INFO:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connected> [IPv4 ('127.0.0.1',9092)]>: Closing connection. [Error 7] RequestTimedOutError: Request timed out after 30000 ms
DEBUG:kafka.producer.sender:Starting Kafka producer I/O thread.
DEBUG:kafka.producer.kafka:Kafka producer started
DEBUG:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connected> [IPv4 ('127.0.0.1',9092)]>: reconnect backoff 0.04056800940753716 after 1 failures
DEBUG:kafka.client:Initializing connection to node bootstrap-0 for Metadata request
DEBUG:kafka.client:Initializing connection to node bootstrap-0 for Metadata request
DEBUG:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connected> [IPv4 ('127.0.0.1',9092)]>: reconnect backoff 0.05091141786347146 after 1 failures
DEBUG:kafka.client:Initializing connection to node bootstrap-0 for Metadata request
DEBUG:kafka.producer.kafka:Requesting Metadata update for topic TestTopicNew
DEBUG:kafka.producer.sender:Beginning shutdown of Kafka producer I/O thread,sending remaining records.
DEBUG:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <disconnected> [IPv4 ('127.0.0.1',9092)]>: creating new socket
DEBUG:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <disconnected> [IPv4 ('127.0.0.1',9092)]>: creating new socket
DEBUG:kafka.client:Initializing connection to node bootstrap-0 for Metadata request
DEBUG:kafka.client:Initiating connection to node bootstrap-0 at harshal-VirtualBox:9092
DEBUG:kafka.producer.sender:Shutdown of Kafka producer I/O thread has completed.
DEBUG:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <disconnected> [IPv4 ('127.0.0.1',9092)]>: setting socket option (6,1,1)
DEBUG:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <disconnected> [IPv4 ('127.0.0.1',9092)]>: creating new socket
DEBUG:kafka.metrics.metrics:Added sensor with name bytes-sent-received
DEBUG:kafka.producer.kafka:The Kafka producer has closed.
INFO:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connecting> [IPv4 ('127.0.0.1',9092)]>: connecting to harshal-VirtualBox:9092 [('127.0.0.1',9092) IPv4]
INFO:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connecting> [IPv4 ('127.0.0.1',9092) IPv4]
DEBUG:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <disconnected> [IPv4 ('127.0.0.1',1)
DEBUG:kafka.metrics.metrics:Added sensor with name bytes-sent
ERROR:kafka.conn:Connection attempt to <brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connecting> [IPv4 ('127.0.0.1',9092)]> timed out
ERROR:kafka.conn:Connection attempt to <brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connecting> [IPv4 ('127.0.0.1',9092)]> timed out
INFO:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connecting> [IPv4 ('127.0.0.1',9092)]>: Closing connection. KafkaConnectionError: timeout
DEBUG:kafka.metrics.metrics:Added sensor with name bytes-received
INFO:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connecting> [IPv4 ('127.0.0.1',9092)]>: Closing connection. KafkaConnectionError: timeout
INFO:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connecting> [IPv4 ('127.0.0.1',9092) IPv4]
DEBUG:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connecting> [IPv4 ('127.0.0.1',9092)]>: reconnect backoff 0.10767298112773331 after 2 failures
DEBUG:kafka.metrics.metrics:Added sensor with name request-latency
DEBUG:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connecting> [IPv4 ('127.0.0.1',9092)]>: reconnect backoff 0.08064906006470171 after 2 failures
ERROR:kafka.conn:Connection attempt to <brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connecting> [IPv4 ('127.0.0.1',9092)]> timed out
DEBUG:kafka.client:Initializing connection to node bootstrap-0 for Metadata request
DEBUG:kafka.metrics.metrics:Added sensor with name node-bootstrap-0.bytes-sent
DEBUG:kafka.client:Initializing connection to node bootstrap-0 for Metadata request
INFO:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connecting> [IPv4 ('127.0.0.1',9092)]>: Closing connection. KafkaConnectionError: timeout
DEBUG:kafka.metrics.metrics:Added sensor with name node-bootstrap-0.bytes-received
DEBUG:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connecting> [IPv4 ('127.0.0.1',9092)]>: reconnect backoff 0.08617990914810844 after 2 failures
DEBUG:kafka.metrics.metrics:Added sensor with name node-bootstrap-0.latency
DEBUG:kafka.client:Initializing connection to node bootstrap-0 for Metadata request
DEBUG:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <disconnected> [unspecified None]>: creating new socket
DEBUG:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <disconnected> [IPv4 ('127.0.0.1',1)
INFO:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connecting> [IPv4 ('127.0.0.1',9092)]>: established TCP connection
INFO:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connecting> [IPv4 ('127.0.0.1',9092)]>: Connection complete.
DEBUG:kafka.client:Node bootstrap-0 connected
DEBUG:kafka.client:Sending Metadata request MetadataRequest_v1(topics=['TestTopicNew']) to node bootstrap-0
DEBUG:kafka.protocol.parser:Sending request MetadataRequest_v1(topics=['TestTopicNew'])
DEBUG:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connected> [IPv4 ('127.0.0.1',9092)]> Request 1: MetadataRequest_v1(topics=['TestTopicNew'])
DEBUG:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <disconnected> [IPv4 ('127.0.0.1',9092)]>: established TCP connection
DEBUG:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <disconnected> [IPv4 ('127.0.0.1',9092)]>: creating new socket
INFO:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connecting> [IPv4 ('127.0.0.1',9092)]>: Connection complete.
DEBUG:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connecting> [IPv4 ('127.0.0.1',1)
DEBUG:kafka.client:Node bootstrap-0 connected
INFO:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connecting> [IPv4 ('127.0.0.1',9092)]>: Connection complete.
INFO:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connecting> [IPv4 ('127.0.0.1',9092) IPv4]
DEBUG:kafka.client:Sending Metadata request MetadataRequest_v1(topics=['TestTopicNew']) to node bootstrap-0
DEBUG:kafka.client:Node bootstrap-0 connected
DEBUG:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connecting> [IPv4 ('127.0.0.1',9092)]>: established TCP connection
DEBUG:kafka.protocol.parser:Sending request MetadataRequest_v1(topics=['TestTopicNew'])
DEBUG:kafka.client:Sending Metadata request MetadataRequest_v1(topics=['TestTopic']) to node bootstrap-0
INFO:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connecting> [IPv4 ('127.0.0.1',9092)]>: Connection complete.
DEBUG:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connected> [IPv4 ('127.0.0.1',9092)]> Request 1: MetadataRequest_v1(topics=['TestTopicNew'])
DEBUG:kafka.protocol.parser:Sending request MetadataRequest_v1(topics=['TestTopic'])
DEBUG:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connected> [IPv4 ('127.0.0.1',9092)]> Request 1: MetadataRequest_v1(topics=['TestTopic'])
DEBUG:kafka.client:Node bootstrap-0 connected
DEBUG:kafka.client:Sending Metadata request MetadataRequest_v1(topics=['TestTopic','TestTopicNew']) to node bootstrap-0
DEBUG:kafka.protocol.parser:Sending request MetadataRequest_v1(topics=['TestTopic','TestTopicNew'])
DEBUG:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connected> [IPv4 ('127.0.0.1',9092)]> Request 1: MetadataRequest_v1(topics=['TestTopic','TestTopicNew'])
WARNING:kafka.client:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connected> [IPv4 ('127.0.0.1',9092)]>: reconnect backoff 0.052615947599434616 after 1 failures
WARNING:kafka.client:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connected> [IPv4 ('127.0.0.1',9092)]>: reconnect backoff 0.046722132029283756 after 1 failures
WARNING:kafka.client:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connected> [IPv4 ('127.0.0.1',9092)]>: reconnect backoff 0.04573278045317557 after 1 failures
WARNING:kafka.client:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connected> [IPv4 ('127.0.0.1',9092)]>: reconnect backoff 0.04625942805975036 after 1 failures
DEBUG:kafka.client:Initializing connection to node bootstrap-0 for Metadata request
DEBUG:kafka.client:Initializing connection to node bootstrap-0 for Metadata request
DEBUG:kafka.client:Initializing connection to node bootstrap-0 for Metadata request
DEBUG:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <disconnected> [IPv4 ('127.0.0.1',9092)]>: Connection complete.
DEBUG:kafka.client:Node bootstrap-0 connected
DEBUG:kafka.client:Sending Metadata request MetadataRequest_v1(topics=['TestTopicNew']) to node bootstrap-0
DEBUG:kafka.protocol.parser:Sending request MetadataRequest_v1(topics=['TestTopicNew'])
DEBUG:kafka.client:Initializing connection to node bootstrap-0 for Metadata request
DEBUG:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <disconnected> [IPv4 ('127.0.0.1',9092)]>: creating new socket
DEBUG:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connected> [IPv4 ('127.0.0.1',9092)]>: established TCP connection
INFO:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connecting> [IPv4 ('127.0.0.1',9092)]>: Connection complete.
DEBUG:kafka.client:Node bootstrap-0 connected
DEBUG:kafka.client:Sending Metadata request MetadataRequest_v1(topics=['TestTopicNew']) to node bootstrap-0
DEBUG:kafka.protocol.parser:Sending request MetadataRequest_v1(topics=['TestTopicNew'])
DEBUG:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connected> [IPv4 ('127.0.0.1',9092)]>: Connection complete.
DEBUG:kafka.client:Node bootstrap-0 connected
DEBUG:kafka.client:Sending Metadata request MetadataRequest_v1(topics=['TestTopic']) to node bootstrap-0
DEBUG:kafka.protocol.parser:Sending request MetadataRequest_v1(topics=['TestTopic'])
DEBUG:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <connected> [IPv4 ('127.0.0.1',9092)]> Request 1: MetadataRequest_v1(topics=['TestTopic'])
DEBUG:kafka.conn:<brokerConnection node_id=bootstrap-0 host=harshal-VirtualBox:9092 <disconnected> [IPv4 ('127.0.0.1',9092)]>: Connection complete.
DEBUG:kafka.client:Node bootstrap-0 connected
DEBUG:kafka.client:Sending Metadata request MetadataRequest_v1(topics=['TestTopic','TestTopicNew'])
解决方法
暂无找到可以解决该程序问题的有效方法，小编努力寻找整理中！
如果你已经找到好的解决方法，欢迎将解决方案带上本链接一起发送给小编。
小编邮箱:dio#foxmail.com (将#修改为@）