问题描述
在我们的解决方案（LoRa WAN - Chirpstack 应用服务器和网络服务器）中，我们使用 RabbitMQ 服务器。使用 MQTT 传输帧，而 AMQP 取决于设备。在过去的 2 个月内，我们发现 AMQP 有 2 次错误。
为了解决这个问题，我们必须重新启动 chirpstack-application-server。但是在生产中不可能总是以这些方式修复它。你能帮我吗？
我在 v3.8.1 中看到了 * 修复 AMQP 重新连接问题。 ...但我不确定这个错误是否相同
网络服务器：3.8.0 应用服务器：3.9.0 RabbitMQ：3.8.2 + Erlang 22.0.7
journalctl 的输出：
May 17 11:24:19 loranet-srv-01.emitel.local chirpstack-application-server[32693]: time=“2021-05-17T11:24:19+02:00” level=error msg=“integration/multi: integration error” ctx_id=3c2a7651-03f0-4f34-9ac5-bbeaefce92f0 error=“publish event error: Exception (504) Reason: “channel/connection is not open”” integration="*amqp.Integration"
来自 RabbitMQ 崩溃日志的日志：
2021-06-03 03:59:06.403 [info] <0.25055.366> connection <0.25055.366> (127.0.0.1:59682 → 127.0.0.1:5672): user ‘logstash’ authenticated and granted access to vhost ‘/’
2021-06-03 03:59:06.409 [info] <0.24970.366> connection <0.24970.366> (127.0.0.1:59684 → 127.0.0.1:5672): user ‘logstash’ authenticated and granted access to vhost ‘/’
2021-06-03 04:01:17.436 [warning] <0.32299.304> closing AMQP connection <0.32299.304> (10.93.0.1:60538 → 10.93.0.2:5672,vhost: ‘/’,user: ‘chirpstack_as’):
client unexpectedly closed TCP connection
2021-06-03 04:01:18.517 [info] <0.25425.366> MQTT vhost picked using plugin configuration or default
2021-06-03 04:01:18.711 [warning] <0.24203.212> MQTT disconnecting client “172.20.102.1:60372 → 10.93.0.2:1883” with duplicate id ‘logger-1334’
2021-06-03 04:01:18.712 [info] <0.25425.366> accepting MQTT connection <0.25425.366> (172.20.102.1:51255 → 10.93.0.2:1883,client id: logger-1334)
2021-06-03 04:01:24.927 [info] <0.25471.366> MQTT vhost picked using plugin configuration or default
2021-06-03 04:01:26.472 [warning] <0.24249.212> MQTT disconnecting client “172.20.102.1:60371 → 10.93.0.2:1883” with duplicate id ‘logger-2755’
2021-06-03 04:01:26.474 [info] <0.25471.366> accepting MQTT connection <0.25471.366> (172.20.102.1:51297 → 10.93.0.2:1883,client id: logger-2755)
2021-06-03 04:01:30.096 [info] <0.24814.366> MQTT vhost picked using plugin configuration or default
2021-06-03 04:01:31.290 [error] <0.32645.304> MQTT: a socket write Failed,the socket might already be closed
2021-06-03 04:01:31.290 [info] <0.32645.304> MQTT connection “10.93.0.1:44382 → 10.93.0.2:1883” will terminate because peer closed TCP connection
2021-06-03 04:01:35.104 [error] <0.24814.366> MQTT cannot accept a connection: client ID registration timed out
2021-06-03 04:01:35.104 [error] <0.24814.366> MQTT cannot accept connection 10.93.0.1:35330 → 10.93.0.2:1883 due to an internal error or unavailable component
2021-06-03 04:01:36.291 [error] <0.32645.304> ** Generic server <0.32645.304> terminating
** Last message in was {tcp_closed,#Port<0.1814243>}
** When Server state == {state,#Port<0.1814243>,“10.93.0.1:44382 → 10.93.0.2:1883”,true,undefined,running,{none,<0.32296.304>},<0.32638.304>,false,none,{proc_state,#{“application/+/device/+/tx” => [0]},{<<“amq.ctag
-YOowwvGDd4Gio7wHkLEpDQ”>>,undefined},{0,nil},1,“AckMrCeAAx2bHcmZcnfOzg”,{<0.32618.304>,<0.32622.304>,<<“amq.topic”>>,{amqp_adapter_info,65535,2653,2},1883,1},4
4382,<<“10.93.0.1:44382 → 10.93.0.2:1883”>>,{‘MQTT’,“N/A”},[{variable_map,#{<<“client_id”>> => <<“AckMrCeAAx2bHcmZcnfOzg”>>}},{channels,{channel_max,{frame_max,0},{client_properties,[{<<“product”>>,longstr,<<“MQTT client”>>},{clie
nt_id,<<“AckMrCeAAx2bHcmZcnfOzg”>>}]},{ssl,false}]},<0.530.0>,{auth_state,<<“chirpstack_as”>>,{user,[‘Chirpstack’],[{rabbit_auth_backend_internal,none}]},<<"/">>},#Fun<rabbit_mqtt_processor.0.50925783>,#Fun<rabbit_mqtt_util.4.100928214>,#Fun<rabbit_mqtt_util.5.100928214>},{state,fine,5000,#Ref<0.2613465786.3301179395.25474>}}
** Reason for termination ==
** {{badmatch,{timeout,{mqtt_node,‘rabbit@loraapp-srv-01’}}},[{rabbit_mqtt_processor,close_connection,[{file,“src/rabbit_mqtt_processor.erl”},{line,900}]},{rabbit_mqtt_reader,do_terminate,2,“src/rabbit_mqtt_reader.erl”},24
2}]},{gen_server2,terminate,3,“src/gen_server2.erl”},1169}]},{proc_lib,wake_up,“proc_lib.erl”},259}]}]}
** In ‘terminate’ callback with reason ==
** {shutdown,conn_closed}
2021-06-03 04:01:36.291 [error] <0.32645.304> CRASH REPORT Process <0.32645.304> with 1 neighbours exited with reason: no match of right hand value {timeout,‘rabbit@loraapp-srv-01’}} in rabbit_mqtt_processor:close_connection/1
line 900 in gen_server2:terminate/3 line 1172
2021-06-03 04:01:36.292 [error] <0.32621.304> Supervisor {<0.32621.304>,amqp_channel_sup} had child channel started with amqp_channel:start_link(direct,<0.32619.304>,{<<“10.93.0.1:44382 → 10.93.0.2:1883”>>,1}) at <0.
32618.304> exit with reason no match of right hand value {timeout,‘rabbit@loraapp-srv-01’}} in rabbit_mqtt_processor:close_connection/1 line 900 in context child_terminated
2021-06-03 04:01:36.292 [error] <0.32640.304> Supervisor {<0.32640.304>,rabbit_mqtt_connection_sup} had child rabbit_mqtt_reader started with rabbit_mqtt_reader:start_link(<0.32638.304>,{acceptor,1883}) at <0.32645.304
exit with reason no match of right hand value {timeout,‘rabbit@loraapp-srv-01’}} in rabbit_mqtt_processor:close_connection/1 line 900 in context child_terminated
2021-06-03 04:01:36.292 [error] <0.32621.304> Supervisor {<0.32621.304>,1}) at <0.
32618.304> exit with reason reached_max_restart_intensity in context shutdown
2021-06-03 04:01:36.292 [error] <0.32640.304> Supervisor {<0.32640.304>,1883}) at <0.32645.304
exit with reason reached_max_restart_intensity in context shutdown
2021-06-03 04:01:36.292 [error] <0.32622.304> ** Generic server <0.32622.304> terminating
** Last message in was {‘EXIT’,<0.32645.304>,{{badmatch,[{f
ile,242}]},259}]}]}}
** When Server state == {state,amqp_direct_connection,‘rabbit@loraapp-srv-01’,<<"/">>,{amqp_params_direct,<<“SUhIVgLkkrUfG6R/boaoY
Fhn9qAYI+ygy5qch5P7abl3ORsRMBTw4JnorNUjeMKW”>>,44382,“3.1.1”},#{<<“c
lient_id”>> => <<“AckMrCeAAx2bHcmZcnfOzg”>>}},{client_id,[]},{amqp_adapter_inf
o,{client
_properties,<0.32624.304>,1621256199087},<0.32637.304>,[{<<“capabilities”>>,table,[{<<“publisher_confirms”>>,bool,true},{<<“exchange_exchange_bindings”>>,…]},…],…}
** Reason for termination ==
** “stopping because dependent process <0.32645.304> died: {{badmatch,\n {timeout,\n {mqtt_node,\n ‘rabbit@loraapp-srv-01’}}},\n [{rabbit_mqtt_processor,\n close_connection,\n [{file,\n “src/rabbit_mqtt_processor.erl”},\n {line,\n {rabbit_mqtt_reader,\n do_terminate,\n “src/rabbit_mqtt_reader.erl”},\n {gen_server2,\n terminate,\n “src/gen_server2.erl”},\n {proc_lib,\n “proc_lib.erl”},25…”
2021-06-03 04:01:36.293 [error] <0.32622.304> CRASH REPORT Process <0.32622.304> with 0 neighbours exited with reason: “stopping because dependent process <0.32645.304> died: {{badmatch,\n …” in gen_server:handle_common_reply/8 line 751
2021-06-03 04:01:36.293 [error] <0.32641.304> Supervisor {<0.32641.304>,amqp_connection_sup} had child connection started with amqp_gen_connection:start_link(<0.32651.304>,<<“SUhIVgLkkrUfG6R/boaoYFhn9qAYI+ygy5qch5P7abl3ORsRMBTw4Jn…”>>,…}) at <0.32622.304> exit with reason “stopping because dependent process <0.32645.304> died: {{badmatch,\n …” in context child_terminated
2021-06-03 04:01:36.293 [error] <0.32641.304> Supervisor {<0.32641.304>,…}) at <0.32622.304> exit with reason reached_max_restart_intensity in context shutdown
2021-06-03 04:01:36.293 [error] <0.295.305> ** Generic server <0.295.305> terminating
** Last message in was {‘EXIT’,<0.32618.304>,259}]}]}}
** When Server state == {ch,{conf,rabbit_framing_amqp_0_9_1,<<“mqtt-subscription-AckMrCeAAx2bHcmZcnfOzgqos0”>>,{<<“basic.nack”>>,{<<“consumer_cancel_notify”>>,{<<“connection.blocked”>>,{<<“authentication_failure_close”>>,true}],10,134217728,{lstate,<0.289.305>,false},{[],[]}},#{<0.458.0> => {resource,queue,<<“mqtt-subscription-fe80:1:3:26002:1:1:1:27qos1”>>},<0.24213.212> => {resource,<<“mqtt-subscription-logger-2755qos0”>>},<0.24242.212> => {resource,<<“mqtt-subscription-logger-1422qos0”>>},<0.32291.304> => {resource,<<“mqtt-subscription-AckMrCeAAx2bHcmZcnfOzgqos0”>>},<0.6870.353> => {resource,<<“mqtt-subscription-mqtt_15e1c9e9.577696qos1”>>}},#{<0.458.0> => #Ref<0.2613465786.3254779908.20375>,<0.24213.212> => #Ref<0.2613465786.3479699457.204624>,<0.24242.212> => #Ref<0.2613465786.3479699457.204623>,<0.32291.304> => #Ref<0.2613465786.3479699457.202162>,<0.6870.353> => #Ref<0.2613465786.3520331777.196060>},erlang},#{<<“amq.ctag-YOowwvGDd4Gio7wHkLEpDQ”>> => {{amqqueue,{resource,[],<0.32291.304>,live,#{user => <<“chirpstack_as”>>},rabbit_classic_queue,#{}},{true,[]}}},#{<0.32291.304> => {1,{<<“amq.ctag-YOowwvGDd4Gio7wHkLEpDQ”>>,…}}},…}
** Reason for termination ==
** {{badmatch,259}]}]}
2021-06-03 04:01:36.294 [error] <0.295.305> CRASH REPORT Process <0.295.305> with 0 neighbours exited with reason: no match of right hand value {timeout,‘rabbit@loraapp-srv-01’}} in rabbit_mqtt_processor:close_connection/1 line 900 in gen_server2:terminate/3 line 1183
2021-06-03 04:01:36.294 [error] <0.292.305> Supervisor {<0.292.305>,rabbit_channel_sup} had child channel started with rabbit_channel:start_link(1,…},…}) at <0.295.305> exit with reason no match of right hand value {timeout,‘rabbit@loraapp-srv-01’}} in rabbit_mqtt_processor:close_connection/1 line 900 in context child_terminated
2021-06-03 04:01:36.294 [error] <0.292.305> Supervisor {<0.292.305>,…}) at <0.295.305> exit with reason reached_max_restart_intensity in context shutdown
2021-06-03 04:01:37.105 [info] <0.24828.366> MQTT vhost picked using plugin configuration or default
2021-06-03 04:01:42.113 [error] <0.24828.366> MQTT cannot accept a connection: client ID registration timed out
2021-06-03 04:01:42.113 [error] <0.24828.366> MQTT cannot accept connection 10.93.0.1:35338 → 10.93.0.2:1883 due to an internal error or unavailable component
2021-06-03 04:01:46.115 [info] <0.25564.366> MQTT vhost picked using plugin configuration or default
2021-06-03 04:01:46.662 [info] <0.25564.366> accepting MQTT connection <0.25564.366> (10.93.0.1:35358 → 10.93.0.2:1883,client id: 1M4VrmghPWJsSOFRncBEeA)
2021-06-04 03:07:30.005 [info] <0.26138.369> accepting AMQP connection <0.26138.369> (10.104.1.131:46242 → 10.93.0.2:5672)
{bad_header,<<“GET / HT”>>}
2021-06-04 03:07:35.040 [error] <0.26349.369> MQTT cannot parse a frame on connection ‘10.104.1.131:60008 → 10.93.0.2:1883’,unparseable payload: <<73,102,0>>,error: {{badmatch,<<>>},[{rabbit_mqtt_frame,parse_frame,“src/rabbit_mqtt_frame.erl”},104}]},parse,358}]},process_received_bytes,289}]},handle_msg,1067}]},init_p_do_apply,249}]}]}
2021-06-04 03:07:35.047 [info] <0.24610.369> accepting AMQP connection <0.24610.369> (10.104.1.131:48490 → 10.93.0.2:5672)
2021-06-04 03:07:35.047 [error] <0.24610.369> closing AMQP connection <0.24610.369> (10.104.1.131:48490 → 10.93.0.2:5672):
{bad_header,<<0,22,128,1>>}
2021-06-04 03:07:35.054 [info] <0.26587.369> accepting AMQP connection <0.26587.369> (10.104.1.131:48500 → 10.93.0.2:5672)
2021-06-04 03:07:35.054 [error] <0.26587.369> closing AMQP connection <0.26587.369> (10.104.1.131:48500 → 10.93.0.2:5672):
{bad_version,0}}
2021-06-04 03:07:35.063 [info] <0.26586.369> accepting AMQP connection <0.26586.369> (10.104.1.131:48506 → 10.93.0.2:5672)
2021-06-04 03:07:35.068 [info] <0.26439.369> accepting AMQP connection <0.26439.369> (10.104.1.131:48512 → 10.93.0.2:5672)
2021-06-04 03:07:35.068 [error] <0.26439.369> closing AMQP connection <0.26439.369> (10.104.1.131:48512 → 10.93.0.2:5672):
{bad_version,{2,9,1}}
2021-06-04 03:07:36.067 [warning] <0.26586.369> closing AMQP connection <0.26586.369> (10.104.1.131:48506 → 10.93.0.2:5672):
client unexpectedly closed TCP connection
2021-06-04 03:08:22.338 [error] <0.28654.369> MQTT cannot parse a frame on connection ‘10.104.1.131:51810 → 10.93.0.2:1883’,unparseable payload: <<“GET / HTTP/1.1\r\nConnection: Close\r\nHost: loraapp-srv-01\r\nPragma: no-cache\r\nUser-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)\r\nAccept: image/gif,image/x-xbitmap,image/jpeg,image/pjpeg,image/png,/\r\nAccept-Language: en\r\nAccept-Charset: iso-8859-1,*,utf-8\r\n\r\n”>>,<<“T / HTTP/1.1\r\nConnection: Close\r\nHost: loraapp-srv-01\r\nPragma: no-cac”>>},249}]}]}
解决方法
暂无找到可以解决该程序问题的有效方法，小编努力寻找整理中！
如果你已经找到好的解决方法，欢迎将解决方案带上本链接一起发送给小编。
小编邮箱:dio#foxmail.com (将#修改为@）