问题描述
在 k8 集群中，我们有 etcd:3.3.15 Pod 正在运行。但是当我们查询集群健康状况时，集群似乎降级了，问题是，pods/peers 进入 active 和 inactive 状态，我不知道他们为什么这样做，下面的代码可能会给人一些输入。
2021-03-17 13:39:32.775194 I | rafthttp: peer 68d1163017ce392 **became active**
2021-03-17 13:39:32.775221 I | rafthttp: established a TCP streaming connection with peer 68d1163017ce392 (stream Message writer)
2021-03-17 13:39:32.775437 I | rafthttp: established a TCP streaming connection with peer 68d1163017ce392 (stream MsgApp v2 writer)
2021-03-17 13:39:34.273399 E | rafthttp: Failed to dial 68d1163017ce392 on stream MsgApp v2 (dial tcp: I/O timeout)
2021-03-17 13:39:34.273433 I | rafthttp: peer 68d1163017ce392 **became inactive** (message send to peer Failed)
[root@k8s-master-0 ~]# kubectl -n mvnr-mtcil1-infra-mtcil-mtcil1 exec -it etcd-1 -- /bin/bash
bash-5.0# ETCDCTL_API=3 etcdctl endpoint health --cluster
{"level":"warn","ts":"2021-03-17T15:34:30.858Z","caller":"clientv3/retry_interceptor.go:61","msg":"retrying of unary invoker Failed","target":"endpoint://client-bdb76ddc-ac56-4ba3-b8f0-9d15a9f3571f/etcd-0.etcd.mvnr-mtcil1-infra-mtcil-mtcil1.svc.cluster.local:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
http://etcd-2.etcd.mvnr-mtcil1-infra-mtcil-mtcil1.svc.cluster.local:2379 is healthy: successfully committed proposal: took = 4.436385ms
http://etcd-1.etcd.mvnr-mtcil1-infra-mtcil-mtcil1.svc.cluster.local:2379 is healthy: successfully committed proposal: took = 4.21728ms
http://etcd-0.etcd.mvnr-mtcil1-infra-mtcil-mtcil1.svc.cluster.local:2379 is unhealthy: Failed to commit proposal: context deadline exceeded
Error: unhealthy cluster
bash-5.0# ETCDCTL_API=3 etcdctl endpoint health --cluster
http://etcd-1.etcd.mvnr-mtcil1-infra-mtcil-mtcil1.svc.cluster.local:2379 is healthy: successfully committed proposal: took = 76.860651ms
http://etcd-2.etcd.mvnr-mtcil1-infra-mtcil-mtcil1.svc.cluster.local:2379 is healthy: successfully committed proposal: took = 3.96122ms
http://etcd-0.etcd.mvnr-mtcil1-infra-mtcil-mtcil1.svc.cluster.local:2379 is healthy: successfully committed proposal: took = 77.39875ms
bash-5.0# ETCDCTL_API=3 etcdctl endpoint health --cluster
{"level":"warn","ts":"2021-03-17T15:34:52.874Z","target":"endpoint://client-7186ba57-fc81-49be-889a-552b5aba0c8d/etcd-0.etcd.mvnr-mtcil1-infra-mtcil-mtcil1.svc.cluster.local:2379","error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
{"level":"warn","target":"endpoint://client-867d9d6e-0925-491f-81e1-d92dc799c25a/etcd-2.etcd.mvnr-mtcil1-infra-mtcil-mtcil1.svc.cluster.local:2379","error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
http://etcd-1.etcd.mvnr-mtcil1-infra-mtcil-mtcil1.svc.cluster.local:2379 is healthy: successfully committed proposal: took = 88.035144ms
http://etcd-0.etcd.mvnr-mtcil1-infra-mtcil-mtcil1.svc.cluster.local:2379 is unhealthy: Failed to commit proposal: context deadline exceeded
http://etcd-2.etcd.mvnr-mtcil1-infra-mtcil-mtcil1.svc.cluster.local:2379 is unhealthy: Failed to commit proposal: context deadline exceeded
Error: unhealthy cluster
bash-5.0# ETCDCTL_API=3 etcdctl endpoint health --cluster
{"level":"warn","ts":"2021-03-17T15:35:22.927Z","target":"endpoint://client-bf153df7-fb4d-4ade-aafc-38003d4577f5/etcd-2.etcd.mvnr-mtcil1-infra-mtcil-mtcil1.svc.cluster.local:2379","ts":"2021-03-17T15:35:22.960Z","target":"endpoint://client-6f107368-f895-4b2b-8cd0-f952f36359f4/etcd-0.etcd.mvnr-mtcil1-infra-mtcil-mtcil1.svc.cluster.local:2379","error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
http://etcd-1.etcd.mvnr-mtcil1-infra-mtcil-mtcil1.svc.cluster.local:2379 is healthy: successfully committed proposal: took = 35.355891ms
http://etcd-2.etcd.mvnr-mtcil1-infra-mtcil-mtcil1.svc.cluster.local:2379 is unhealthy: Failed to commit proposal: context deadline exceeded
http://etcd-0.etcd.mvnr-mtcil1-infra-mtcil-mtcil1.svc.cluster.local:2379 is unhealthy: Failed to commit proposal: context deadline exceeded
Error: unhealthy cluster
bash-5.0# ETCDCTL_API=3 etcdctl endpoint health --cluster
{"level":"warn","ts":"2021-03-17T15:35:51.991Z","target":"endpoint://client-a20e8fd4-a047-4ac3-a28c-d00ec5c63bbb/etcd-2.etcd.mvnr-mtcil1-infra-mtcil-mtcil1.svc.cluster.local:2379","error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
http://etcd-0.etcd.mvnr-mtcil1-infra-mtcil-mtcil1.svc.cluster.local:2379 is healthy: successfully committed proposal: took = 68.568217ms
http://etcd-1.etcd.mvnr-mtcil1-infra-mtcil-mtcil1.svc.cluster.local:2379 is healthy: successfully committed proposal: took = 68.503005ms
http://etcd-2.etcd.mvnr-mtcil1-infra-mtcil-mtcil1.svc.cluster.local:2379 is unhealthy: Failed to commit proposal: context deadline exceeded
Error: unhealthy cluster
bash-5.0# ETCDCTL_API=3 etcdctl endpoint health --cluster
{"level":"warn","ts":"2021-03-17T15:36:01.490Z","target":"endpoint://client-cfcbf88f-7115-455b-ab5d-06a3c82c7a63/etcd-0.etcd.mvnr-mtcil1-infra-mtcil-mtcil1.svc.cluster.local:2379","ts":"2021-03-17T15:36:01.557Z","target":"endpoint://client-b7144958-b2ee-4e2e-a709-c18c10e6ac7c/etcd-2.etcd.mvnr-mtcil1-infra-mtcil-mtcil1.svc.cluster.local:2379","error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
http://etcd-1.etcd.mvnr-mtcil1-infra-mtcil-mtcil1.svc.cluster.local:2379 is healthy: successfully committed proposal: took = 71.045159ms
http://etcd-0.etcd.mvnr-mtcil1-infra-mtcil-mtcil1.svc.cluster.local:2379 is unhealthy: Failed to commit proposal: context deadline exceeded
http://etcd-2.etcd.mvnr-mtcil1-infra-mtcil-mtcil1.svc.cluster.local:2379 is unhealthy: Failed to commit proposal: context deadline exceeded
Error: unhealthy cluster
bash-5.0# ETCDCTL_API=3 etcdctl endpoint health --cluster
{"level":"warn","ts":"2021-03-17T15:36:11.667Z","target":"endpoint://client-24f1dd60-2da4-4c78-a3e4-be848fcc3bef/etcd-0.etcd.mvnr-mtcil1-infra-mtcil-mtcil1.svc.cluster.local:2379","ts":"2021-03-17T15:36:11.668Z","target":"endpoint://client-ee401b13-7d39-4243-9de6-15701874dc36/etcd-2.etcd.mvnr-mtcil1-infra-mtcil-mtcil1.svc.cluster.local:2379","error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
http://etcd-1.etcd.mvnr-mtcil1-infra-mtcil-mtcil1.svc.cluster.local:2379 is healthy: successfully committed proposal: took = 3.814891ms
http://etcd-0.etcd.mvnr-mtcil1-infra-mtcil-mtcil1.svc.cluster.local:2379 is unhealthy: Failed to commit proposal: context deadline exceeded
http://etcd-2.etcd.mvnr-mtcil1-infra-mtcil-mtcil1.svc.cluster.local:2379 is unhealthy: Failed to commit proposal: context deadline exceeded
Error: unhealthy cluster
所有配置似乎都很好。
解决方法
暂无找到可以解决该程序问题的有效方法，小编努力寻找整理中！
如果你已经找到好的解决方法，欢迎将解决方案带上本链接一起发送给小编。
小编邮箱:dio#foxmail.com (将#修改为@）