Cassandra节点正在运行，但无法连接？

编程问答 2022-06-20

问题描述

使用Cassandra 3.11.8版，openjdk-8u242-b08

在崩溃之前，我正在更改具有50k +列的表，因此这可能是所有这些的一个因素。理想情况下，我宁愿丢失提交中的数据（如果仍然永久插入积压的数据），这样我就可以连接到主机，从而可以恢复服务。

在开始错误之前，我开始执行alter table命令，每次调用1000次将许多列插入表中。最终，在完成其中的一半之后，我收到了所有节点的错误消息。

2020-09-10 15:34:29警告[控制连接]连接到时出错 127.0.0.3:9042：回溯（最近一次通话最近）：文件“ cassandra \ cluster.py”，行3522，在 cassandra.cluster.ControlConnection._reconnect_internal文件 “ cassandra \ cluster.py”，第3591行，在 cassandra.cluster.ControlConnection._try_connect文件 “ cassandra \ cluster.py”，第3588行，在 cassandra.cluster.ControlConnection._try_connect文件 “ cassandra \ cluster.py”，第3690行，在 cassandra.cluster.ControlConnection._refresh_schema文件 第142行中的“ cassandra \ metadata.py” cassandra.metadata.Metadata.refresh文件“ cassandra \ metadata.py”， cassandra.metadata.Metadata._rebuild_all文件中的第165行 get_all_keyspaces文件中的“ cassandra \ metadata.py”行2522 get_all_keyspaces文件中的“ cassandra \ metadata.py”行2031 “ cassandra \ metadata.py”，第2719行，在 cassandra.metadata.SchemaParserV3._query_all文件 在“ cassandra \ connection.py”行985中 cassandra.connection.Connection.wait_for_responses文件 “ cassandra \ connection.py”，第983行，在 cassandra.connection.Connection.wait_for_responses文件 第1435行的“ cassandra \ connection.py”在 cassandra.connection.ResponseWaiter.deliver cassandra.OperationTimedOut：errors = None，last_host = None

我正在服务器上运行8个节点。我已重置所有节点，并且握手已完成。但是我无法连接到任何节点上的集群。一旦cassandra开始运行，我的system.log和debug.log都有相似的日志。 gc.log已有一段时间没有更新，所以让我想知道怎么回事？有趣的一点是，我只检索表中的列列表总计3次，我在桌面上运行此代码时使用2个节点（资源少得多）没有问题，并且没有收到任何这些问题。

编辑：为清楚起见，我的应用程序/连接未运行，并且下面的这些日志是定期发生的事情。我试图查看计划的任务，但无法找到有关cassandra的信息。我想知道它是从什么积压书中读取的，是否可以阻止它。理想情况下，我想停止这种工作负担……

-------SYSTEM.LOG-------
INFO  [GossipStage:1] 2020-09-10 17:38:52,376 StorageService.java:2400 - Node /127.0.0.9 state jump to NORMAL
WARN  [OptionalTasks:1] 2020-09-10 17:38:54,802 CassandraRoleManager.java:377 - CassandraRoleManager skipped default role setup: some nodes were not ready
INFO  [OptionalTasks:1] 2020-09-10 17:38:54,802 CassandraRoleManager.java:416 - Setup task failed with error,rescheduling
INFO  [HANDSHAKE-/127.0.0.4] 2020-09-10 17:38:56,965 OutboundTcpConnection.java:561 - Handshaking version with /127.0.0.4
INFO  [HANDSHAKE-/127.0.0.4] 2020-09-10 17:38:58,262 OutboundTcpConnection.java:561 - Handshaking version with /127.0.0.4
INFO  [GossipStage:1] 2020-09-10 17:38:59,102 Gossiper.java:1139 - Node /127.0.0.4 has restarted,now UP
INFO  [GossipStage:1] 2020-09-10 17:38:59,103 TokenMetadata.java:497 - Updating topology for /127.0.0.4
INFO  [GossipStage:1] 2020-09-10 17:38:59,105 Gossiper.java:1103 - InetAddress /127.0.0.4 is now UP
INFO  [HANDSHAKE-/127.0.0.5] 2020-09-10 17:38:59,813 OutboundTcpConnection.java:561 - Handshaking version with /127.0.0.5
INFO  [GossipStage:1] 2020-09-10 17:39:00,104 StorageService.java:2400 - Node /127.0.0.4 state jump to NORMAL
INFO  [HANDSHAKE-/127.0.0.5] 2020-09-10 17:39:01,029 OutboundTcpConnection.java:561 - Handshaking version with /127.0.0.5
INFO  [GossipStage:1] 2020-09-10 17:39:01,266 Gossiper.java:1139 - Node /127.0.0.5 has restarted,now UP
INFO  [GossipStage:1] 2020-09-10 17:39:01,267 TokenMetadata.java:497 - Updating topology for /127.0.0.5
INFO  [GossipStage:1] 2020-09-10 17:39:01,270 Gossiper.java:1103 - InetAddress /127.0.0.5 is now UP
INFO  [GossipStage:1] 2020-09-10 17:39:04,271 StorageService.java:2400 - Node /127.0.0.5 state jump to NORMAL
INFO  [ScheduledTasks:1] 2020-09-10 17:43:05,805 NoSpamLogger.java:91 - Some operations were slow,details available at debug level (debug.log)
INFO  [ScheduledTasks:1] 2020-09-10 17:48:40,892 NoSpamLogger.java:91 - Some operations were slow,details available at debug level (debug.log)
INFO  [ScheduledTasks:1] 2020-09-10 17:54:35,999 NoSpamLogger.java:91 - Some operations were slow,details available at debug level (debug.log)
INFO  [ScheduledTasks:1] 2020-09-10 17:59:36,083 NoSpamLogger.java:91 - Some operations were slow,details available at debug level (debug.log)
INFO  [Service Thread] 2020-09-10 18:00:24,722 GCInspector.java:285 - ParNew GC in 237ms.  CMS Old Gen: 717168160 -> 887151520; Par Eden Space: 1718091776 -> 0; Par Survivor Space: 12757512 -> 214695936
INFO  [ScheduledTasks:1] 2020-09-10 18:04:56,160 NoSpamLogger.java:91 - Some operations were slow,details available at debug level (debug.log)

------DEBUG.LOG------
INFO  [Service Thread] 2020-09-10 18:00:24,722 GCInspector.java:285 - ParNew GC in 237ms.  CMS Old Gen: 717168160 -> 887151520; Par Eden Space: 1718091776 -> 0; Par Survivor Space: 12757512 -> 214695936
DEBUG [ScheduledTasks:1] 2020-09-10 18:00:26,102 MonitoringTask.java:173 - 1 operations were slow in the last 4996 msecs:
<SELECT * FROM system_schema.columns>,was slow 2 times: avg/min/max 1256/1232/1281 msec - slow timeout 500 msec
DEBUG [ScheduledTasks:1] 2020-09-10 18:00:56,110 MonitoringTask.java:173 - 1 operations were slow in the last 5007 msecs:
<SELECT * FROM system_schema.columns>,time 795 msec - slow timeout 500 msec
DEBUG [ScheduledTasks:1] 2020-09-10 18:01:01,111 MonitoringTask.java:173 - 1 operations were slow in the last 5003 msecs:
<SELECT * FROM system_schema.columns>,time 808 msec - slow timeout 500 msec
DEBUG [ScheduledTasks:1] 2020-09-10 18:03:41,143 MonitoringTask.java:173 - 1 operations were slow in the last 5002 msecs:
<SELECT * FROM system_schema.columns>,time 853 msec - slow timeout 500 msec
DEBUG [ScheduledTasks:1] 2020-09-10 18:04:06,148 MonitoringTask.java:173 - 1 operations were slow in the last 4996 msecs:
<SELECT * FROM system_schema.columns>,time 772 msec - slow timeout 500 msec
DEBUG [ScheduledTasks:1] 2020-09-10 18:04:26,153 MonitoringTask.java:173 - 1 operations were slow in the last 4991 msecs:
<SELECT * FROM system_schema.columns>,time 838 msec - slow timeout 500 msec
DEBUG [ScheduledTasks:1] 2020-09-10 18:04:31,154 MonitoringTask.java:173 - 1 operations were slow in the last 5009 msecs:
<SELECT * FROM system_schema.columns>,time 841 msec - slow timeout 500 msec
INFO  [ScheduledTasks:1] 2020-09-10 18:04:56,details available at debug level (debug.log)
DEBUG [ScheduledTasks:1] 2020-09-10 18:04:56,160 MonitoringTask.java:173 - 1 operations were slow in the last 5004 msecs:
<SELECT * FROM system_schema.columns>,time 772 msec - slow timeout 500 msec
DEBUG [ScheduledTasks:1] 2020-09-10 18:05:11,165 MonitoringTask.java:173 - 1 operations were slow in the last 4994 msecs:
<SELECT * FROM system_schema.columns>,time 808 msec - slow timeout 500 msec
DEBUG [ScheduledTasks:1] 2020-09-10 18:05:31,171 MonitoringTask.java:173 - 1 operations were slow in the last 5004 msecs:
<SELECT * FROM system_schema.columns>,time 834 msec - slow timeout 500 msec
DEBUG [ScheduledTasks:1] 2020-09-10 18:05:56,176 MonitoringTask.java:173 - 1 operations were slow in the last 5010 msecs:
<SELECT * FROM system_schema.columns>,was slow 2 times: avg/min/max 847/837/857 msec - slow timeout 500 msec
DEBUG [ScheduledTasks:1] 2020-09-10 18:07:16,196 MonitoringTask.java:173 - 1 operations were slow in the last 5003 msecs:
<SELECT * FROM system_schema.columns>,time 827 msec - slow timeout 500 msec
DEBUG [ScheduledTasks:1] 2020-09-10 18:07:31,200 MonitoringTask.java:173 - 1 operations were slow in the last 5007 msecs:
<SELECT * FROM system_schema.columns>,time 834 msec - slow timeout 500 msec
DEBUG [ScheduledTasks:1] 2020-09-10 18:08:01,207 MonitoringTask.java:173 - 1 operations were slow in the last 5000 msecs:
<SELECT * FROM system_schema.columns>,time 799 msec - slow timeout 500 msec
DEBUG [ScheduledTasks:1] 2020-09-10 18:08:16,211 MonitoringTask.java:173 - 1 operations were slow in the last 4999 msecs:
<SELECT * FROM system_schema.columns>,time 780 msec - slow timeout 500 msec
DEBUG [ScheduledTasks:1] 2020-09-10 18:08:36,217 MonitoringTask.java:173 - 1 operations were slow in the last 5000 msecs:
<SELECT * FROM system_schema.columns>,time 835 msec - slow timeout 500 msec
DEBUG [ScheduledTasks:1] 2020-09-10 18:09:01,221 MonitoringTask.java:173 - 1 operations were slow in the last 5002 msecs:
<SELECT * FROM system_schema.columns>,time 832 msec - slow timeout 500 msec
INFO  [ScheduledTasks:1] 2020-09-10 18:09:56,231 NoSpamLogger.java:91 - Some operations were slow,details available at debug level (debug.log)
DEBUG [ScheduledTasks:1] 2020-09-10 18:09:56,231 MonitoringTask.java:173 - 1 operations were slow in the last 4995 msecs:
<SELECT * FROM system_schema.columns>,time 778 msec - slow timeout 500 msec
DEBUG [ScheduledTasks:1] 2020-09-10 18:10:06,233 MonitoringTask.java:173 - 1 operations were slow in the last 5009 msecs:
<SELECT * FROM system_schema.columns>,time 1099 msec - slow timeout 500 msec

解决方法

超时是由于驱动程序试图在建立控制连接时解析架构而引起的。

驱动程序将控制连接用于管理任务，例如在初始化阶段发现集群的拓扑和架构。我已经在这篇帖子中进行了更详细的讨论-https://community.datastax.com/questions/7702/。

在您的情况下，驱动程序初始化在解析您提到的表中的数千列时超时。我必须承认这对我来说是新的。我从来没有使用过具有数千列的集群，所以我很好奇知道您的用例是什么，也许有一个更好的数据模型。

作为一种解决方法，您可以尝试消除默认超时，以查看驱动程序是否最终能够初始化。但是，这将成为一个临时解决方案，因为驱动程序每次发生DDL时都需要解析该架构。干杯！

cassandracassandra-3.0garbage-collection

相关问答

matplotlib报错：AttributeError: module 'backend_interagg' has no attribute 'FigureCanvas'. Did you mean: 'FigureCanvasAgg'?
使用本地python环境可以成功执行 import pandas as pd impor...
gitlab登录失败，报错：This challenge page was accidentally cached by an intermediary and is no longer available.
设置时间 控制面板
后端开发常见错误
错误1：Request method ‘DELETE‘ not supported 错误还原：...
docker常见错误
错误1：启动docker镜像时报错：Error response from daemon:...
idea常见错误
错误1：private field ‘xxx‘ is never assigned 按Alt...
pip安装依赖失败
报错如下，通过源不能下载，最后警告pip需升级版本 Requirem...