为什么 EMQX Persistence 在本地 kubernetes 上工作时不能在 azure kubernetes 上工作？

编程问答 2022-05-26

问题描述

在本地机器上使用 kubernetes(minikube) statefulset 时，EMQX 规则持续存在，因为相同的 pod IP 被分配给 emqx 节点，例如 /opt/emqx/data/mnesia /emqx@172.17.0.9。即使我在新 pod 启动时删除了 pod，它也会被分配与以前相同的 IP。一切正常。

但是当我使用 aks(azure kubernetes) 在使用 azure 文件的 aks 集群上部署 EMQX 时，pod IP 每次都不同。例如，如果 /opt/emqx/data/mnesia/emqx@10.1.1.10 被分配给 EMQX 节点，那么如果我尝试删除 pod，则 /opt/emqx/data/ mnesia/emqx@10.1.1.11 可能已分配给它。

所以，没有什么是持久的。

本地代码

---

apiVersion: storage.k8s.io/v1
kind: StorageClass
Metadata:
  name: local-storage5
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

---

apiVersion: v1
kind: PersistentVolume
Metadata:
  name: emqx-pv5
spec:
  capacity:
    storage: 300Mi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage5
  local:
    path: /opt/emqx/data/mnesia
  nodeAffinity:
    required:
      nodeselectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - minikube

---

apiVersion: v1
kind: Service
Metadata:
  name: emqx-headless
spec:
  type: ClusterIP
  clusterIP: None
  selector:
    app: emqx
  ports:
    - name: mqtt
      port: 1883
      protocol: TCP
      targetPort: 1883
    - name: mqttssl
      port: 8883
      protocol: TCP
      targetPort: 8883
    - name: mgmt
      port: 8081
      protocol: TCP
      targetPort: 8081
    - name: websocket
      port: 8083
      protocol: TCP
      targetPort: 8083
    - name: wss
      port: 8084
      protocol: TCP
      targetPort: 8084
    - name: dashboard
      port: 18083
      protocol: TCP
      targetPort: 18083

---

apiVersion: apps/v1
kind: StatefulSet
Metadata:
  name: emqx-statefulset
  labels:
    app: emqx
spec:
  replicas: 1
  serviceName: emqx-headless
  selector:
    matchLabels:
      app: emqx
  template:
    Metadata:
      labels:
        app: emqx
    spec:
      containers:
        - name: emqx
          image: emqx/emqx:4.2.7
          ports:
            - name: emqx-dashboard
              containerPort: 18083
            - name: ssl-port
              containerPort: 8883
            - name: emqx-port
              containerPort: 1883
            - name: ssl-dashboard
              containerPort: 18084
          env:
            - name: EMQX_LOADED_PLUGINS
              value: emqx_management,emqx_recon,emqx_retainer,emqx_dashboard,emqx_rule_engine,emqx_auth_username
            - name: EMQX_CLUSTER__disCOVERY
              value: k8s
            - name: EMQX_NAME
              value: emqx
            - name: EMQX_CLUSTER__K8S__APISERVER
              value: https://kubernetes.default:443
            - name: EMQX_CLUSTER__K8S__SERVICE_NAME
              value: emqx
            - name: EMQX_CLUSTER__K8S__ADDRESS_TYPE
              value: ip
            - name: EMQX_CLUSTER__K8S__APP_NAME
              value: emqx
            - name: EMQX_ALLOW_ANONYMOUS
              value: "false"
            - name: EMQX_LISTENER__SSL__EXTERNAL__MAX_CONNECTIONS
              value: "1024000"
            - name: EMQX_AUTH__USER__PASSWORD_HASH
              value: sha256
            - name: EMQX_AUTH__USER__1__USERNAME
              value: 
            - name: EMQX_AUTH__USER__1__PASSWORD
              value: 
            - name: EMQX_DASHBOARD__DEFAULT_USER__LOGIN
              value: 
            - name: EMQX_DASHBOARD__DEFAULT_USER__PASSWORD
              value:
            - name: EMQX_DASHBOARD__LISTENER__HTTPS
              value: "18084"
            - name: MQX_DASHBOARD__LISTENER__HTTPS__ACCEPTORS
              value: "4"
            - name: EMQX_DASHBOARD__LISTENER__HTTPS__MAX_CLIENTS
              value: "512"
          tty: true
          volumeMounts:
            - name: emqx-mnesia
              mountPath: "/opt/emqx/data/mnesia"

  volumeClaimTemplates:
    - Metadata:
        name: emqx-mnesia
      spec:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: "local-storage5"
        resources:
          requests:
            storage: 300Mi

Azure Kubernetes 代码

apiVersion: v1
kind: ServiceAccount
Metadata:
  name: emqx
  namespace: emqx-test

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
Metadata:
  name: emqx
subjects:
  - kind: ServiceAccount
    name: emqx
    namespace: emqx-test
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
Metadata:
  name: emqx-files
provisioner: kubernetes.io/azure-file
mountOptions:
  - dir_mode=0777
  - file_mode=0777
  - uid=0
  - gid=0
  - mfsymlinks
  - cache=strict
  - actimeo=30
parameters:
  skuName: Standard_lrs
---
apiVersion: v1
kind: PersistentVolumeClaim
Metadata:
  name: emqx-pvc
  namespace: emqx-test
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: emqx-files
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: Service
Metadata:
  name: emqx
  namespace: emqx-test
spec:
  ports:
    - name: emqx-dashboard
      port: 80
      targetPort: 18083
      protocol: TCP
    - name: ssl-port
      port: 8883
      targetPort: ssl-port
      protocol: TCP
    - name: emqx-port
      port: 1883
      targetPort: emqx-port
      protocol: TCP
    - name: ssl-dashboard
      port: 443
      targetPort: 18084
      protocol: TCP
  selector:
    app: emqx
  type: LoadBalancer
---
apiVersion: apps/v1
kind: StatefulSet
Metadata:
  name: emqx
  labels:
    app: emqx
  namespace: emqx-test
spec:
  serviceName: "emqx"
  selector:
    matchLabels:
      app: emqx
  replicas: 1
  template:
    Metadata:
      labels:
        app: emqx
    spec:
      containers:
        - name: emqx
          image: emqx/emqx:4.2.7
          ports:
            - name: emqx-dashboard
              containerPort: 18083
            - name: ssl-port
              containerPort: 8883
            - name: emqx-port
              containerPort: 1883
            - name: ssl-dashboard
              containerPort: 18084
          env:
            - name: EMQX_LOADED_PLUGINS
              value: emqx_management,emqx_auth_username
            - name: EMQX_CLUSTER__disCOVERY
              value: k8s
            - name: EMQX_NAME
              value: emqx
            - name: EMQX_CLUSTER__K8S__APISERVER
              value: https://kubernetes.default:443
            - name: EMQX_CLUSTER__K8S__NAMESPACE
              value: emqx-test
            - name: EMQX_CLUSTER__K8S__SERVICE_NAME
              value: emqx
            - name: EMQX_CLUSTER__K8S__ADDRESS_TYPE
              value: ip
            - name: EMQX_CLUSTER__K8S__APP_NAME
              value: emqx
            - name: EMQX_ALLOW_ANONYMOUS
              value: "false"
            - name: EMQX_LISTENER__SSL__EXTERNAL__MAX_CONNECTIONS
              value: "1024000"
            - name: EMQX_AUTH__USER__PASSWORD_HASH
              value: sha256
            - name: EMQX_AUTH__USER__1__USERNAME
              value:
            - name: EMQX_AUTH__USER__1__PASSWORD
              value:
            - name: EMQX_DASHBOARD__DEFAULT_USER__LOGIN
              value:
            - name: EMQX_DASHBOARD__DEFAULT_USER__PASSWORD
              value:
            - name: EMQX_DASHBOARD__LISTENER__HTTPS
              value: "18084"
            - name: MQX_DASHBOARD__LISTENER__HTTPS__ACCEPTORS
              value: "4"
            - name: EMQX_DASHBOARD__LISTENER__HTTPS__MAX_CLIENTS
              value: "512"
          volumeMounts:
            - name: emqx-data
              mountPath: "/opt/emqx/data/mnesia"
          tty: true
      volumes:
        - name: emqx-data
          persistentVolumeClaim:
            claimName: emqx-pvc

解决方法

在 StatefulSet Basics 上的 k8s 文档中，您阅读了：

Pod 的序号、主机名、SRV 记录和 A 记录名称具有 未更改，但与 Pod 关联的 IP 地址可能已更改 改变了。在本教程使用的集群中，他们有。 这就是为什么 重要的是不要配置其他应用程序来连接到 Pod 在按 IP 地址的 StatefulSet 中。

这是预期的，正如您所看到的，文档中提到了这种行为。

但是为什么你在 minikube 上看到不同的行为，在 azure 上看到不同的行为？ IP 地址由 CNI 分配。在 minikube 默认 CNI 上它是 docker-bridge，在 azure 上是它的 Azure CNI，所以它是由 CNI 分配的地址。

最好始终假设您不能依赖 pod IP 地址来保持静态。使用 DNS for statefulsets 和 for other pods and services 进行通信，切勿直接使用硬编码的 pod ip 地址。

azure-aksazure-devopsemqkuberneteskubernetes-statefulset