问题描述
我们在我们的生产堆栈中偶尔会看到以下带有堆栈跟踪的异常（偶尔）。堆栈大部分时间都在平稳运行。我们使用 Axon 3.4.3 和 Spring Boot 2.1.6.RELEASE。
我认为这意味着 Axon 无法在分布式命令总线上调度命令，因为 Spring 命令总线连接器超时。
导致 Spring Command Bus 连接器超时的原因是什么？超时发生在 30 秒后，然后重试。有时我们会看到 HTTP 503，有时我们会看到 HTTP 500。
我已经知道这不是由运行缓慢的 DB 语句引起的，因为它在不到 3 毫秒的时间内返回。
http.request: GET /api/path/to/do/something
java.net.socketTimeoutException: Read timed out
at java.base/java.net.socketInputStream.socketRead0(Native Method)
at java.base/java.net.socketInputStream.socketRead(SocketInputStream.java:115)
at java.base/java.net.socketInputStream.read(SocketInputStream.java:168)
at java.base/java.net.socketInputStream.read(SocketInputStream.java:140)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at org.springframework.cloud.netflix.ribbon.apache.RetryableRibbonLoadBalancingHttpClient.lambda$execute$0(RetryableRibbonLoadBalancingHttpClient.java:125)
at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:287)
at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:180)
at org.springframework.cloud.netflix.ribbon.apache.RetryableRibbonLoadBalancingHttpClient.executeWithRetry(RetryableRibbonLoadBalancingHttpClient.java:180)
at org.springframework.cloud.netflix.ribbon.apache.RetryableRibbonLoadBalancingHttpClient.execute(RetryableRibbonLoadBalancingHttpClient.java:145)
at org.springframework.cloud.netflix.ribbon.apache.RetryableRibbonLoadBalancingHttpClient.execute(RetryableRibbonLoadBalancingHttpClient.java:60)
at org.springframework.cloud.netflix.zuul.filters.route.support.AbstractRibbonCommand.run(AbstractRibbonCommand.java:218)
at org.springframework.cloud.netflix.zuul.filters.route.support.AbstractRibbonCommand.run(AbstractRibbonCommand.java:55)
at com.netflix.hystrix.HystrixCommand$2.call(HystrixCommand.java:302)
at com.netflix.hystrix.HystrixCommand$2.call(HystrixCommand.java:298)
at rx.internal.operators.OnSubscribedefer.call(OnSubscribedefer.java:46)
at rx.internal.operators.OnSubscribedefer.call(OnSubscribedefer.java:35)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at datadog.trace.instrumentation.rxjava.TracedOnSubscribe.call(TracedOnSubscribe.java:48)
at datadog.trace.instrumentation.rxjava.TracedOnSubscribe.call(TracedOnSubscribe.java:15)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribedefer.call(OnSubscribedefer.java:51)
at rx.internal.operators.OnSubscribedefer.call(OnSubscribedefer.java:35)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribedoOnEach.call(OnSubscribedoOnEach.java:41)
at rx.internal.operators.OnSubscribedoOnEach.call(OnSubscribedoOnEach.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribedoOnEach.call(OnSubscribedoOnEach.java:41)
at rx.internal.operators.OnSubscribedoOnEach.call(OnSubscribedoOnEach.java:30)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribedoOnEach.call(OnSubscribedoOnEach.java:41)
at rx.internal.operators.OnSubscribedoOnEach.call(OnSubscribedoOnEach.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribedoOnEach.call(OnSubscribedoOnE
spring.handler：TheController.doSomething
org.axonframework.commandhandling.distributed.CommanddispatchException: An error occurred while trying to dispatch a command on the distributedCommandBus: 503 null
at org.axonframework.commandhandling.distributed.distributedCommandBus.dispatch(distributedCommandBus.java:139)
at org.axonframework.commandhandling.gateway.AbstractCommandGateway.send(AbstractCommandGateway.java:79)
at org.axonframework.commandhandling.gateway.DefaultCommandGateway.send(DefaultCommandGateway.java:95)
at org.axonframework.commandhandling.gateway.DefaultCommandGateway.send(DefaultCommandGateway.java:143)
http.request: POST /spring-command-bus-connector/command
java.io.IOException: Server returned HTTP response code: 503 for URL: http://10.244.2.46:8084/spring-command-bus-connector/command
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1924)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520)
at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527)
at org.springframework.http.client.SimpleClientHttpResponse.getRawStatusCode(SimpleClientHttpResponse.java:55)
at org.springframework.web.client.DefaultResponseErrorHandler.hasError(DefaultResponseErrorHandler.java:55)
at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:766)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:736)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:710)
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:628)
at org.axonframework.springcloud.commandhandling.SpringHttpCommandBusConnector.sendRemotely(SpringHttpCommandBusConnector.java:161)
at org.axonframework.springcloud.commandhandling.SpringHttpCommandBusConnector.lambda$send$1(SpringHttpCommandBusConnector.java:130)
at org.axonframework.common.DirectExecutor.execute(DirectExecutor.java:42)
at org.axonframework.springcloud.commandhandling.SpringHttpCommandBusConnector.send(SpringHttpCommandBusConnector.java:128)
at org.axonframework.commandhandling.distributed.distributedCommandBus.dispatch(distributedCommandBus.java:134)
at org.axonframework.commandhandling.gateway.AbstractCommandGateway.send(AbstractCommandGateway.java:79)
at org.axonframework.commandhandling.gateway.DefaultCommandGateway.send(DefaultCommandGateway.java:95)
at org.axonframework.commandhandling.gateway.DefaultCommandGateway.send(DefaultCommandGateway.java:143)
SpringHttpCommandBusConnector.receiveCommand
org.springframework.web.context.request.async.AsyncRequestTimeoutException
at org.springframework.web.context.request.async.TimeoutDeferredResultProcessingInterceptor.handleTimeout(TimeoutDeferredResultProcessingInterceptor.java:42)
at org.springframework.web.context.request.async.DeferredResultInterceptorChain.triggerAfterTimeout(DeferredResultInterceptorChain.java:79)
at org.springframework.web.context.request.async.WebAsyncManager.lambda$startDeferredResultProcessing$5(WebAsyncManager.java:427)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
at org.springframework.web.context.request.async.StandardServletAsyncWebRequest.onTimeout(StandardServletAsyncWebRequest.java:150)
at org.apache.catalina.core.AsyncListenerWrapper.fireOnTimeout(AsyncListenerWrapper.java:44)
at org.apache.catalina.core.AsyncContextImpl.timeout(AsyncContextImpl.java:133)
at org.apache.catalina.connector.CoyoteAdapter.asyncdispatch(CoyoteAdapter.java:153)
at org.apache.coyote.AbstractProcessor.dispatch(AbstractProcessor.java:241)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:53)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:853)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1587)
at org.apache.tomcat.util.net.socketProcessorBase.run(SocketProcessorBase.java:49)
at datadog.trace.bootstrap.instrumentation.java.concurrent.Wrapper.run(Wrapper.java:25)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:834)
解决方法
暂无找到可以解决该程序问题的有效方法，小编努力寻找整理中！
如果你已经找到好的解决方法，欢迎将解决方案带上本链接一起发送给小编。
小编邮箱:dio#foxmail.com (将#修改为@）