问题描述
我有一个kubespray部署的单节点群集。我尝试了2.14-同样的事情。群集在部署后成功启动，并且可以长时间工作。当我重新启动计算机时，除etcd外，所有docker容器（apiserver，调度程序，cni，calico等）均无法正常工作。 Systemd kubelet服务已启动，但其日志显示它不健康。
// main obj
MERGE (mainObj:Obj {code: row.objs.main.code})
SET mainObj.make = row.objs.main.make,mainObj.isMain = true,mainObj:MainObj
// secondary obj
MERGE (secondaryObj:Obj {code: row.objs.secondary.code})
SET secondaryObj.make = row.objs.secondary.make,secondaryObj.isSecondary = true,secondaryObj:SecondaryObj
基本上，与apiserver通讯失败很累。
我观察到的结果非常困惑，使apiserver容器和cni / calico容器（Sep 25 16:07:09 node1 systemd[1]: Started Kubernetes Kubelet Server.
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.378828 4273 flags.go:33] FLAG: --add-dir-header="false"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.378989 4273 flags.go:33] FLAG: --address="0.0.0.0"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.379003 4273 flags.go:33] FLAG: --allowed-unsafe-sysctls="[]"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.379009 4273 flags.go:33] FLAG: --alsologtostderr="false"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.379014 4273 flags.go:33] FLAG: --anonymous-auth="true"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.379019 4273 flags.go:33] FLAG: --application-metrics-count-limit="100"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.379023 4273 flags.go:33] FLAG: --authentication-token-webhook="false"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.379027 4273 flags.go:33] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.379032 4273 flags.go:33] FLAG: --authorization-mode="AlwaysAllow"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.379698 4273 flags.go:33] FLAG: --volume-stats-agg-period="1m0s"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.379734 4273 feature_gate.go:243] feature gates: &{map[]}
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.381339 4273 feature_gate.go:243] feature gates: &{map[]}
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.381391 4273 feature_gate.go:243] feature gates: &{map[]}
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.391492 4273 mount_linux.go:168] Detected OS with systemd
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.391633 4273 server.go:416] Version: v1.17.12
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.391728 4273 feature_gate.go:243] feature gates: &{map[]}
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.391804 4273 feature_gate.go:243] feature gates: &{map[]}
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.391934 4273 plugins.go:100] No cloud provider specified.
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.391956 4273 server.go:532] No cloud provider specified: "" from the config file: ""
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.391968 4273 server.go:822] Client rotation is on,will bootstrap in background
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.393945 4273 bootstrap.go:84] Current kubeconfig file contents are still valid,no bootstrap necessary
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.394040 4273 certificate_store.go:129] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.394294 4273 server.go:849] Starting client certificate rotation.
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.394311 4273 certificate_manager.go:275] Certificate rotation is enabled.
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.394462 4273 certificate_manager.go:531] Certificate expiration is 2021-09-25 15:58:29 +0000 UTC,rotation deadline is 2021-08-09 11:53:23.825031508 +0000 UTC
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.394497 4273 certificate_manager.go:281] Waiting 7627h46m14.430536992s for next certificate rotation
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.394860 4273 dynamic_cafile_content.go:166] Starting client-ca-bundle::/etc/kubernetes/ssl/ca.crt
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.395055 4273 manager.go:146] cAdvisor running in container: "/sys/fs/cgroup/cpu,cpuacct/system.slice/kubelet.service"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.414349 4273 fs.go:125] Filesystem UUIDs: map[2020-09-25-15-46-36-00:/dev/sr0 55122f25-8946-4857-952e-48dc1647b4e1:/dev/vda1 88DB-6C04:/dev/vda15]
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.414375 4273 fs.go:126] Filesystem partitions: map[/dev/shm:{mountpoint:/dev/shm major:0 minor:25 fsType:tmpfs blockSize:0} /dev/vda1:{mountpoint:/ major:252 minor:1 fsType:ext4 blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /run/lock:{mountpoint:/run/lock major:0 minor:26 fsType:tmpfs blockSize:0} /run/user/1000:{mountpoint:/run/user/1000 major:0 minor:61 fsType:tmpfs blockSize:0} /sys/fs/cgroup:{mountpoint:/sys/fs/cgroup major:0 minor:27 fsType:tmpfs blockSize:0}]
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.418995 4273 manager.go:193] Machine: {NumCores:8 cpuFrequency:2494136 MemoryCapacity:50641403904 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:6f481a3404734917921596637175f7df SystemUUID:6F481A34-0473-4917-9215-96637175F7DF BootID:4499a41b-6deb-4cf6-906f-a88d6794e657 Filesystems:[{Device:/sys/fs/cgroup DeviceMajor:0 DeviceMinor:27 Capacity:25320701952 Type:vfs Inodes:6181812 HasInodes:true} {Device:/run/user/1000 DeviceMajor:0 DeviceMinor:61 Capacity:5064138752 Type:vfs Inodes:6181812 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:5064142848 Type:vfs Inodes:6181812 HasInodes:true} {Device:/dev/vda1 DeviceMajor:252 DeviceMinor:1 Capacity:103880232960 Type:vfs Inodes:12902400 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:25 Capacity:25320701952 Type:vfs Inodes:6181812 HasInodes:true} {Device:/run/lock DeviceMajor:0 DeviceMinor:26 Capacity:5242880 Type:vfs Inodes:6181812 HasInodes:true}] diskMap:map[252:0:{Name:vda Major:252 Minor:0 Size:107374182400 Scheduler:none}] NetworkDevices:[{Name:ens3 MacAddress:fa:16:3e:26:40:ce Speed:-1 Mtu:1500}] Topology:[{Id:0 Memory:50641403904 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[{Size:16777216 Type:Unified Level:3}]} {Id:1 Memory:0 HugePages:[] Cores:[{Id:0 Threads:[1] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[{Size:16777216 Type:Unified Level:3}]} {Id:2 Memory:0 HugePages:[] Cores:[{Id:0 Threads:[2] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[{Size:16777216 Type:Unified Level:3}]} {Id:3 Memory:0 HugePages:[] Cores:[{Id:0 Threads:[3] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[{Size:16777216 Type:Unified Level:3}]} {Id:4 Memory:0 HugePages:[] Cores:[{Id:0 Threads:[4] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[{Size:16777216 Type:Unified Level:3}]} {Id:5 Memory:0 HugePages:[] Cores:[{Id:0 Threads:[5] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[{Size:16777216 Type:Unified Level:3}]} {Id:6 Memory:0 HugePages:[] Cores:[{Id:0 Threads:[6] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[{Size:16777216 Type:Unified Level:3}]} {Id:7 Memory:0 HugePages:[] Cores:[{Id:0 Threads:[7] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[{Size:16777216 Type:Unified Level:3}]}] CloudProvider:UnkNown InstanceType:UnkNown InstanceID:None}
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.420041 4273 manager.go:199] Version: {KernelVersion:4.15.0-118-generic ContainerOsversion:Ubuntu 18.04.5 LTS DockerVersion:18.09.7 DockerAPIVersion:1.39 CadvisorVersion: CadvisorRevision:}
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.420193 4273 server.go:642] --cgroups-per-qos enabled,but --cgroup-root was not specified. defaulting to /
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.420530 4273 container_manager_linux.go:265] container manager verified user specified cgroup-root exists: []
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.420548 4273 container_manager_linux.go:270] Creating Container Manager object based on Node Config: {RuntimeCgroupsName:/systemd/system.slice SystemCgroupsName: KubeletCgroupsName:/systemd/system.slice ContainerRuntime:docker CgroupsPerQOS:true Cgrouproot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemcpus: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[cpu:{i:{value:200 scale:-3} d:{Dec:<nil>} s:200m Format:DecimalSI} memory:{i:{value:512 scale:6} d:{Dec:<nil>} s:512M Format:DecimalSI}] SystemReserved:map[] HardevictionThresholds:[{Signal:imagefs.available Operator:Lessthan Value:{Quantity:<nil> Percentage:0.15} GracePeriod:0s MinReclaim:<nil>} {Signal:memory.available Operator:Lessthan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:Lessthan Value:{Quantity:<nil> Percentage:0.1} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.inodesFree Operator:Lessthan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalcpuManagerPolicy:none ExperimentalcpuManagerReconcilePeriod:10s ExperimentalPodPidsLimit:-1 EnforcecpuLimits:true cpuCFSQuotaPeriod:100ms ExperimentalTopologyManagerPolicy:none}
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.420695 4273 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.420702 4273 container_manager_linux.go:305] Creating device plugin manager: true
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.420712 4273 manager.go:126] Creating Device Plugin manager at /var/lib/kubelet/device-plugins/kubelet.sock
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.420731 4273 fake_topology_manager.go:39] [fake topologymanager] AddHintProvider HintProvider: &{kubelet.sock /var/lib/kubelet/device-plugins/ map[] {0 0} <nil> {{} [0 0 0]} 0x1b1eed0 0x6eb9ef8 0x1b1f7a0 map[] map[] map[] map[] map[] 0xc000a6d7d0 [0] 0x6eb9ef8}
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.420763 4273 state_mem.go:36] [cpumanager] initializing new in-memory state store
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.420837 4273 state_mem.go:84] [cpumanager] updated default cpuset: ""
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.420843 4273 state_mem.go:92] [cpumanager] updated cpuset assignments: "map[]"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.420851 4273 state_checkpoint.go:101] [cpumanager] state checkpoint: restored state from checkpoint
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.420855 4273 state_checkpoint.go:102] [cpumanager] state checkpoint: defaultcpuSet:
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.420862 4273 fake_topology_manager.go:39] [fake topologymanager] AddHintProvider HintProvider: &{{0 0} 0x6eb9ef8 10000000000 0xc00049ea20 <nil> <nil> <nil> <nil> map[cpu:{{200 -3} {<nil>} DecimalSI} memory:{{616857600 0} {<nil>} DecimalSI}] 0x6eb9ef8}
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.420925 4273 server.go:1056] Using root directory: /var/lib/kubelet
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.420947 4273 kubelet.go:286] Adding pod path: /etc/kubernetes/manifests
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.420974 4273 file.go:68] Watching path "/etc/kubernetes/manifests"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.420987 4273 kubelet.go:311] Watching apiserver
Sep 25 16:07:09 node1 kubelet[4273]: E0925 16:07:09.421729 4273 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/kubelet.go:449: Failed to list *v1.Service: Get https://10.34.12.170:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.34.12.170:6443: connect: connection refused
Sep 25 16:07:09 node1&limit=500&resourceVersion=0: dial tcp 10.34.12.170:6443: connect: connection refused
Sep 25 16:07:09 node1&limit=500&resourceVersion=0: dial tcp 10.34.12.170:6443: connect: connection refused
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.424494 4273 client.go:75] Connecting to docker on unix:///var/run/docker.sock
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.424517 4273 client.go:104] Start docker client with request timeout=2m0s
Sep 25 16:07:09 node1 kubelet[4273]: W0925 16:07:09.425742 4273 docker_service.go:564] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled,falling back to "hairpin-veth"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.425764 4273 docker_service.go:240] Hairpin mode set to "hairpin-veth"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.459931 4273 plugins.go:166] Loaded network plugin "cni"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.459991 4273 docker_service.go:255] Docker cri networking managed by cni
Sep 25 16:07:09 node1 Labels:[] ExperimentalBuild:false ServerVersion:18.09.7 ClusterStore: Clusteradvertise: Runtimes:map[runc:{Path:runc Args:[]}] DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:[] Nodes:0 Managers:0 Cluster:<nil> Warnings:[]} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:8fba4e9a7d01810a393d5d25a3621dc101981175 Expected:8fba4e9a7d01810a393d5d25a3621dc101981175} runcCommit:{ID:dc9208a3303feef5b3839f4323d9beb36df0a9dd Expected:dc9208a3303feef5b3839f4323d9beb36df0a9dd} InitCommit:{ID:fec3683 Expected:fec3683} SecurityOptions:[name=apparmor name=seccomp,profile=default] ProductLicense:Community Engine Warnings:[WARNING: No swap limit support]}
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.468858 4273 docker_service.go:273] Setting cgroupDriver to cgroupfs
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.468983 4273 kubelet.go:642] Starting the GRPC server for the docker CRI shim.
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.469073 4273 container_manager_linux.go:118] Configure resource-only container "/systemd/system.slice" with memory limit: 35448982732
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.469091 4273 docker_server.go:59] Start dockershim grpc server
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.478669 4273 remote_runtime.go:59] parsed scheme: ""
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.478689 4273 remote_runtime.go:59] scheme "" not registered,fallback to default scheme
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.478714 4273 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/var/run/dockershim.sock 0 <nil>}] <nil>}
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.478722 4273 clientconn.go:577] ClientConn switching balancer to "pick_first"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.478749 4273 remote_image.go:50] parsed scheme: ""
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.478755 4273 remote_image.go:50] scheme "" not registered,fallback to default scheme
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.478764 4273 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/var/run/dockershim.sock 0 <nil>}] <nil>}
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.478769 4273 clientconn.go:577] ClientConn switching balancer to "pick_first"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.478889 4273 balancer_conn_wrappers.go:127] pickfirstBalancer: HandleSubConnStateChange: 0xc00026e020,CONNECTING
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.478928 4273 balancer_conn_wrappers.go:127] pickfirstBalancer: HandleSubConnStateChange: 0xc000a42020,CONNECTING
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.479399 4273 balancer_conn_wrappers.go:127] pickfirstBalancer: HandleSubConnStateChange: 0xc00026e020,READY
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.479426 4273 balancer_conn_wrappers.go:127] pickfirstBalancer: HandleSubConnStateChange: 0xc000a42020,READY
Sep 25 16:07:09 node1 kubelet[4273]: E0925 16:07:09.870533 4273 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
Sep 25 16:07:09 node1 kubelet[4273]: For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.872026 4273 kuberuntime_manager.go:217] Container runtime docker initialized,version: 18.09.7,apiVersion: 1.39.0
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.872387 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/aws-ebs"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.872485 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/gce-pd"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.872550 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/cinder"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.872602 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/azure-disk"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.872651 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/azure-file"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.872701 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/vsphere-volume"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.872763 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/empty-dir"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.872813 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/git-repo"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.872872 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/host-path"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.872926 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/nfs"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.872976 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/secret"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.873041 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/iscsi"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.873091 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/glusterfs"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.873151 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/rbd"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.873200 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/quobyte"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.873251 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/cephfs"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.873298 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/downward-api"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.873347 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/fc"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.873397 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/flocker"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.873445 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/configmap"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.873496 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/projected"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.873556 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/portworx-volume"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.873623 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/scaleio"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.873681 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/local-volume"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.873729 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/storageos"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.873816 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/csi"
Sep 25 16:07:09 node1: dial tcp 10.34.12.170:6443: connect: connection refused
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.875105 4273 server.go:1114] Started kubelet
Sep 25 16:07:09 node1 kubelet[4273]: E0925 16:07:09.875145 4273 kubelet.go:1305] Image garbage collection Failed once. Stats initialization may not have completed yet: Failed to get imageFs info: unable to find data in memory cache
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.875186 4273 server.go:144] Starting to listen on 10.34.12.170:10250
Sep 25 16:07:09 node1 kubelet[4273]: E0925 16:07:09.875457 4273 event.go:272] Unable to write event: 'Post https://10.34.12.170:6443/api/v1/namespaces/default/events: dial tcp 10.34.12.170:6443: connect: connection refused' (may retry after sleeping)
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.876063 4273 fs_resource_analyzer.go:64] Starting FS ResourceAnalyzer
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.876153 4273 volume_manager.go:263] The desired_state_of_world populator starts
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.876163 4273 volume_manager.go:265] Starting Kubelet Volume Manager
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.876550 4273 desired_state_of_world_populator.go:138] Desired state populator starts to run
Sep 25 16:07:09 node1 kubelet[4273]: E0925 16:07:09.876729 4273 reflector.go:153] k8s.io/client-go/informers/factory.go:135: Failed to list *v1beta1.CSIDriver: Get https://10.34.12.170:6443/apis/storage.k8s.io/v1beta1/csidrivers?limit=500&resourceVersion=0: dial tcp 10.34.12.170:6443: connect: connection refused
Sep 25 16:07:09 node1?timeout=10s: dial tcp 10.34.12.170:6443: connect: connection refused
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.877999 4273 server.go:384] Adding debug handlers to kubelet server.
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.891555 4273 factory.go:356] Registering Docker factory
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.891578 4273 factory.go:54] Registering systemd factory
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.891705 4273 clientconn.go:104] parsed scheme: "unix"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.891714 4273 clientconn.go:104] scheme "unix" not registered,fallback to default scheme
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.891735 4273 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock 0 <nil>}] <nil>}
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.891743 4273 clientconn.go:577] ClientConn switching balancer to "pick_first"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.891797 4273 balancer_conn_wrappers.go:127] pickfirstBalancer: HandleSubConnStateChange: 0xc00037cd00,CONNECTING
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.892584 4273 balancer_conn_wrappers.go:127] pickfirstBalancer: HandleSubConnStateChange: 0xc00037cd00,READY
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.893173 4273 factory.go:137] Registering containerd factory
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.893533 4273 factory.go:101] Registering Raw factory
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.893662 4273 manager.go:1158] Started watching for new ooms in manager
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.894666 4273 manager.go:272] Starting recovery of all containers
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.897183 4273 kubelet_network_linux.go:150] Not using `--random-fully` in the MASQUERADE rule for iptables because the local version of iptables does not support it
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.898224 4273 status_manager.go:157] Starting to sync pod status with apiserver
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.898331 4273 kubelet.go:1823] Starting kubelet main sync loop.
Sep 25 16:07:09 node1 kubelet[4273]: E0925 16:07:09.898413 4273 kubelet.go:1847] skipping pod synchronization - [container runtime status check may not have completed yet,PLEG is not healthy: pleg has yet to be successful]
Sep 25 16:07:09 node1 kubelet[4273]: E0925 16:07:09.899415 4273 reflector.go:153] k8s.io/client-go/informers/factory.go:135: Failed to list *v1beta1.RuntimeClass: Get https://10.34.12.170:6443/apis/node.k8s.io/v1beta1/runtimeclasses?limit=500&resourceVersion=0: dial tcp 10.34.12.170:6443: connect: connection refused
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.904467 4273 manager.go:277] Recovery completed
Sep 25 16:07:09 node1 kubelet[4273]: W0925 16:07:09.923594 4273 docker_sandBox.go:394] Failed to read pod IP from plugin/docker: networkPlugin cni Failed on the status hook for pod "kubernetes-metrics-scraper-747b4fd5cd-f2gtr_kube-system": CNI Failed to retrieve network namespace path: cannot find network namespace for the terminated container "15b9965fb59f8b9f590ff1f01c7bf8c71746da8dcb0163a7277b7d1664f09b52"
Sep 25 16:07:09 node1 kubelet[4273]: W0925 16:07:09.928100 4273 docker_sandBox.go:394] Failed to read pod IP from plugin/docker: networkPlugin cni Failed on the status hook for pod "kubernetes-dashboard-77475cf576-xc6br_kube-system": CNI Failed to retrieve network namespace path: cannot find network namespace for the terminated container "3652b305aec8a60b50602b5ca12e9335a8e35c270cbcfcb7896139de97c73ace"
Sep 25 16:07:09 node1 kubelet[4273]: W0925 16:07:09.944457 4273 docker_sandBox.go:394] Failed to read pod IP from plugin/docker: networkPlugin cni Failed on the status hook for pod "dns-autoscaler-85f898cd5c-jv6cb_kube-system": CNI Failed to retrieve network namespace path: cannot find network namespace for the terminated container "55ee03cc56e650563ab445bccab7f327c7b8c66e086efbeb7d93bfb816aba838"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.947817 4273 kubelet_node_status.go:294] Setting node annotation to enable volume controller attach/detach
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.947980 4273 setters.go:73] Using node IP: "10.34.12.170"
Sep 25 16:07:09 node1
Sep 25 16:07:09 node1
Sep 25 16:07:09 node1
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.949354 4273 cpu_manager.go:173] [cpumanager] starting with none policy
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.949359 4273 cpu_manager.go:174] [cpumanager] reconciling every 10s
Sep 25 16:07:09 node1 kubelet[4273]: W0925 16:07:09.949360 4273 docker_sandBox.go:394] Failed to read pod IP from plugin/docker: networkPlugin cni Failed on the status hook for pod "coredns-76798d84dd-6bmth_kube-system": CNI Failed to retrieve network namespace path: cannot find network namespace for the terminated container "bde82b4692238cdcc8bb0860d9773ddbf57fc09d59388f655757224ef463ff1a"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.949365 4273 policy_none.go:43] [cpumanager] none policy: Start
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.950266 4273 manager.go:226] Starting Device Plugin manager
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.950956 4273 manager.go:268] Serving device plugin registration server on "/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.951022 4273 plugin_watcher.go:54] Plugin Watcher Start at /var/lib/kubelet/plugins_registry
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.951107 4273 plugin_manager.go:112] The desired_state_of_world populator (plugin watcher) starts
）都无法开始使用。它不是kubelet假设要启动这些吗，如果是这样，为什么它尝试不尝试启动api服务器就与它对话？如果没有kubelete，那么启动apiserver的原因是什么，则api服务器容器上的重新启动策略为否。
解决方法
暂无找到可以解决该程序问题的有效方法，小编努力寻找整理中！
如果你已经找到好的解决方法，欢迎将解决方案带上本链接一起发送给小编。
小编邮箱:dio#foxmail.com (将#修改为@）