Sentinel 有状态设置计划失败，无法找到要绑定的持久卷

问题描述

下午好

我真的需要一些帮助来建立一组哨兵，以便他们可以监视和执行我的 redis pod 的选举，这些 pod 运行没有问题。在这条消息的底部，我包含了哨兵配置，它说明了卷。第一个哨兵 sentinel0 处于 Pending 状态，而其余的 redis 实例都处于 READY 1/1，对于所有三个。

但他们没有被安排。当我尝试应用哨兵 statefulset 时，我收到以下计划错误。 sentinel statefulset 配置在这篇文章的底部

Warning FailedScheduling 5s default-scheduler 0/4 nodes are available: 1 node(s) has taint {node-role.kubernetes.io/master: },pod 不能容忍，3 node(s) did '找不到要绑定的可用持久卷。 警告 FailedScheduling 4s default-scheduler 0/4 nodes are available: 1 node(s) has taint {node-role.kubernetes.io/master: },pod 不能容忍，3 node(s) not found要绑定的可用持久卷。

关于我的 kubernetes 设置：

我正在运行一个四节点的裸机 kubernetes 集群；分别为1个master节点和3个worker节点。

对于存储，我使用跨节点共享的“本地存储”StorageClass。目前我正在使用一个单一的持久卷配置文件，它在三个节点上定义了三个卷。这似乎适用于 redis statefulset，但不适用于哨兵。 （底部的哨兵配置）

见下面持久卷的配置（pv-volume-node-0、1、2三个都绑定了）

kind: PersistentVolume
apiVersion: v1
metadata:
  name: ag1-pv-volume-node-0
  labels:
    type: local
spec:
  storageClassName: local-storage
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  local:
    path: "/var/opt/mssql"
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - k8s-node-0
---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: ag1-pv-volume-node-1
  labels:
    type: local
spec:
  storageClassName: local-storage
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  local:
    path: "/var/opt/mssql"
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - k8s-node-1
---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: ag1-pv-volume-node-2
  labels:
    type: local
spec:
  storageClassName: local-storage
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  local:
    path: "/var/opt/mssql"
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - k8s-node-2

注意：路径“/var/opt/mssql”是redis集群的有状态目录数据pt。这是一个用词不当，绝不反映一个 sql 数据库（我只是从演练中使用了这个目录），它可以工作。

目前，所有三个 redis pod 都已成功部署，并具有正常运行的 statefulset，请参阅下面的 redis 配置（所有工作）

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis
spec:
  serviceName: redis
  replicas: 3
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      initContainers:
      - name: config
        image: redis:6.0-alpine
        command: [ "sh","-c" ]
        args:
          - |
            cp /tmp/redis/redis.conf /etc/redis/redis.conf
            
            echo "finding master..."
            MASTER_FDQN=`hostname  -f | sed -e 's/redis-[0-9]\./redis-0./'`
            if [ "$(redis-cli -h sentinel -p 5000 ping)" != "PONG" ]; then
              echo "master not found,defaulting to redis-0"

              if [ "$(hostname)" == "redis-0" ]; then
                echo "this is redis-0,not updating config..."
              else
                echo "updating redis.conf..."
                echo "slaveof $MASTER_FDQN 6379" >> /etc/redis/redis.conf
              fi
            else
              echo "sentinel found,finding master"
              MASTER="$(redis-cli -h sentinel -p 5000 sentinel get-master-addr-by-name mymaster | grep -E '[0-9]{1,3}\.[0-9]{1,3}')"
              echo "master found : $MASTER,updating redis.conf"
              echo "slaveof $MASTER 6379" >> /etc/redis/redis.conf
            fi
        volumeMounts:
        - name: redis-config
          mountPath: /etc/redis/
        - name: config
          mountPath: /tmp/redis/
      containers:
      - name: redis
        image: redis:6.0-alpine
        command: ["redis-server"]
        args: ["/etc/redis/redis.conf"]
        ports:
        - containerPort: 6379
          name: redis
        volumeMounts:
        - name: data
          mountPath: /var/opt/mssql
        - name: redis-config
          mountPath: /etc/redis/
      volumes:
      - name: redis-config
        emptyDir: {}
      - name: config
        configMap:
          name: redis-config
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "local-storage"
      resources:
        requests:
          storage: 50Mi
---
apiVersion: v1
kind: Service
metadata:
  name: redis
spec:
  clusterIP: None
  ports:
  - port: 6379
    targetPort: 6379
    name: redis
  selector:
    app: redis

我遇到的真正问题，我相信源于我如何配置哨兵状态集。 pod 不会调度，其打印原因是它没有找到要绑定的持久卷。

SENTINEL STATEFULSET CONFIG，这里有问题，无法弄清楚如何使用我制作的卷正确设置它。

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: sentinel
spec:
  serviceName: sentinel
  replicas: 3
  selector:
    matchLabels:
      app: sentinel
  template:
    metadata:
      labels:
        app: sentinel
    spec:
      initContainers:
      - name: config
        image: redis:6.0-alpine
        command: [ "sh","-c" ]
        args:
          - |
            REDIS_PASSWORD=a-very-complex-password-here
            nodes=redis-0.redis.redis.svc.cluster.local,redis-1.redis.redis.svc.cluster.local,redis-2.redis.redis.svc.cluster.local

            for i in ${nodes//,/ }
            do
                echo "finding master at $i"
                MASTER=$(redis-cli --no-auth-warning --raw -h $i -a $REDIS_PASSWORD info replication | awk '{print $1}' | grep master_host: | cut -d ":" -f2)
                if [ "$MASTER" == "" ]; then
                    echo "no master found"
                    MASTER=
                else
                    echo "found $MASTER"
                    break
                fi
            done
            echo "sentinel monitor mymaster $MASTER 6379 2" >> /tmp/master

            echo "port 5000
            $(cat /tmp/master)
            sentinel down-after-milliseconds mymaster 5000
            sentinel failover-timeout mymaster 60000
            sentinel parallel-syncs mymaster 1
            sentinel auth-pass mymaster $REDIS_PASSWORD
            " > /etc/redis/sentinel.conf
            cat /etc/redis/sentinel.conf
        volumeMounts:
        - name: redis-config
          mountPath: /etc/redis/
      containers:
      - name: sentinel
        image: redis:6.0-alpine
        command: ["redis-sentinel"]
        args: ["/etc/redis/sentinel.conf"]
        ports:
        - containerPort: 5000
          name: sentinel
        volumeMounts:
        - name: redis-config
          mountPath: /etc/redis/
        - name: data
          mountPath: /var/opt/mssql
      volumes:
      - name: redis-config
        emptyDir: {}
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "local-storage"
      resources:
        requests:
          storage: 50Mi
---
apiVersion: v1
kind: Service
metadata:
  name: sentinel
spec:
  clusterIP: None
  ports:
  - port: 5000
    targetPort: 5000
    name: sentinel
  selector:
    app: sentinel

这是我在这里的第一篇文章。我是 stackoverflow 的忠实粉丝！

解决方法

您可以尝试使用此模板创建三个 PV：

kind: PersistentVolume
apiVersion: v1
metadata:
  name: ag1-pv-volume-node-0
  labels:
    type: local
spec:
  storageClassName: local-storage
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  claimRef:
    namespace: default
    name: data-redis-0
  local:
    path: "/var/opt/mssql"
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - k8s-node-0

这里的重要部分是 claimRef 字段，它通过 StatefulSet 将 PV 与 PVC 联系起来。 应该是特殊格式。

在此处阅读更多信息：https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/preexisting-pd#using_a_preexisting_disk_in_a_statefulset