mariadb 在具有主机路径卷的 kubernetes pod 内崩溃

编程问答 2022-05-31

问题描述

我正在尝试将 linux 服务器上的多个 docker 容器移动到在不同的 linux 机器上运行的基于 kubernets 的测试部署，我已经在 vagrant 虚拟机中安装了 kubernetes 作为 k3s 实例.

其中一个容器是 mariadb 容器实例，映射了绑定卷

这是我正在使用的 docker-compose 的相关部分：

  academy-db:
    image: 'docker.io/bitnami/mariadb:10.3-debian-10'
    container_name: academy-db
    environment:
      - ALLOW_EMPTY_PASSWORD=yes
      - MARIADB_USER=bn_moodle
      - MARIADB_DATABASE=bitnami_moodle
    volumes:
      - type: bind
        source: ./volumes/moodle/mariadb
        target: /bitnami/mariadb
    ports:
      - '3306:3306'

请注意，这可以正常工作。 （该容器被另一个连接到它的应用程序容器使用，并从数据库中读取数据没有问题）。

然后我尝试将其转换为 kubernetes 配置，将卷文件夹复制到目标机器并使用以下 kubernetes .yaml 部署文件。 这包括部署 .yaml、持久卷声明和持久卷，以及使容器可访问的 NodePort 服务。对于数据卷，我使用了一个简单的 hostPath 卷，指向从 docker-compose 的绑定安装复制的内容。

apiVersion: apps/v1
kind: Deployment
Metadata:
  name: academy-db
spec:
  replicas: 1
  selector:
    matchLabels:
      name: academy-db
  strategy:
    type: Recreate
  template:
    Metadata:
      labels:
        name: academy-db
    spec:
      containers:
        - env:
            - name: ALLOW_EMPTY_PASSWORD
              value: "yes"
            - name: MARIADB_DATABASE
              value: bitnami_moodle
            - name: MARIADB_USER
              value: bn_moodle
          image: docker.io/bitnami/mariadb:10.3-debian-10
          name: academy-db
          ports:
            - containerPort: 3306
          resources: {}
          volumeMounts:
            - mountPath: /bitnami/mariadb
              name: academy-db-claim
      restartPolicy: Always
      volumes:
        - name: academy-db-claim
          persistentVolumeClaim:
            claimName: academy-db-claim
---
apiVersion: v1
kind: PersistentVolume
Metadata:
  name: academy-db-pv
  labels:
    type: local
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  hostPath:
    path: "<...full path to deployment folder on the server...>/volumes/moodle/mariadb"
---
apiVersion: v1
kind: PersistentVolumeClaim
Metadata:
  name: academy-db-claim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: ""
  volumeName: academy-db-pv
---
apiVersion: v1
kind: Service
Metadata:
  name: academy-db-service
spec:
  type: NodePort
  ports:
    - name: "3306"
      port: 3306
      targetPort: 3306
  selector:
    name: academy-db

应用部署后，一切似乎都正常，因为 kubectl get ... pod 和卷似乎运行正常

kubectl get pods

NAME                             READY   STATUS    RESTARTS   AGE
academy-db-5547cdbc5-65k79       1/1     Running   9          15d
.
.
.

kubectl get pv
NAME                    CAPACITY   ACCESS MODES   RECLaim POLICY   STATUS   CLaim                              STORAGECLASS   REASON   AGE
academy-db-pv           1Gi        RWO            Retain           Bound    default/academy-db-claim                                   15d
.
.
.

kubectl get pvc
NAME                       STATUS   VOLUME                  CAPACITY   ACCESS MODES   STORAGECLASS   AGE
academy-db-claim           Bound    academy-db-pv           1Gi        RWO                           15d
.
.
.

这是 pod 的日志：

kubectl logs pod/academy-db-5547cdbc5-65k79

mariadb 10:32:05.66
mariadb 10:32:05.66 Welcome to the Bitnami mariadb container
mariadb 10:32:05.66 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mariadb
mariadb 10:32:05.66 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mariadb/issues
mariadb 10:32:05.66
mariadb 10:32:05.67 INFO  ==> ** Starting MariaDB setup **
mariadb 10:32:05.68 INFO  ==> Validating settings in MysqL_*/MARIADB_* env vars
mariadb 10:32:05.68 WARN  ==> You set the environment variable ALLOW_EMPTY_PASSWORD=yes. For safety reasons,do not use this flag in a production environment.
mariadb 10:32:05.69 INFO  ==> Initializing mariadb database
mariadb 10:32:05.69 WARN  ==> The mariadb configuration file '/opt/bitnami/mariadb/conf/my.cnf' is not writable. Configurations based on environment variables will not be applied for this file.
mariadb 10:32:05.70 INFO  ==> Using persisted data
mariadb 10:32:05.71 INFO  ==> Running MysqL_upgrade
mariadb 10:32:05.71 INFO  ==> Starting mariadb in background

和 describe pod 命令：

Name:         academy-db-5547cdbc5-65k79
Namespace:    default
Priority:     0
Node:         zdmp-kube/192.168.33.99
Start Time:   Tue,22 Dec 2020 13:33:43 +0000
Labels:       name=academy-db
              pod-template-hash=5547cdbc5
Annotations:  <none>
Status:       Running
IP:           10.42.0.237
IPs:
  IP:           10.42.0.237
Controlled By:  replicaset/academy-db-5547cdbc5
Containers:
  academy-db:
    Container ID:   containerd://68af105f15a1f503bbae8a83f1b0a38546a84d5e3188029f539b9c50257d2f9a
    Image:          docker.io/bitnami/mariadb:10.3-debian-10
    Image ID:       docker.io/bitnami/mariadb@sha256:1d8ca1757baf64758e7f13becc947b9479494128969af5c0abb0ef544bc08815
    Port:           3306/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Thu,07 Jan 2021 10:32:05 +0000
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Thu,07 Jan 2021 10:22:03 +0000
      Finished:     Thu,07 Jan 2021 10:32:05 +0000
    Ready:          True
    Restart Count:  9
    Environment:
      ALLOW_EMPTY_PASSWORD:  yes
      MARIADB_DATABASE:      bitnami_moodle
      MARIADB_USER:          bn_moodle
      MARIADB_PASSWORD:      bitnami
    Mounts:
      /bitnami/mariadb from academy-db-claim (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-x28jh (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  academy-db-claim:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  academy-db-claim
    ReadOnly:   false
  default-token-x28jh:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-x28jh
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason          Age                  From     Message
  ----    ------          ----                 ----     -------
  normal  Pulled          15d (x8 over 15d)    kubelet  Container image "docker.io/bitnami/mariadb:10.3-debian-10" already present on machine
  normal  Created         15d (x8 over 15d)    kubelet  Created container academy-db
  normal  Started         15d (x8 over 15d)    kubelet  Started container academy-db
  normal  SandBoxChanged  18m                  kubelet  Pod sandBox changed,it will be killed and re-created.
  normal  Pulled          8m14s (x2 over 18m)  kubelet  Container image "docker.io/bitnami/mariadb:10.3-debian-10" already present on machine
  normal  Created         8m14s (x2 over 18m)  kubelet  Created container academy-db
  normal  Started         8m14s (x2 over 18m)  kubelet  Started container academy-db

后来，我注意到客户端应用程序在连接时出现问题。经过一些调查，我得出结论，虽然 pod 正在运行，但在它内部运行的 mariadb 进程可能在启动后就崩溃了。如果我使用 kubectl exec 进入容器并尝试运行例如我得到的 MysqL 客户端：

kubectl  exec -it pod/academy-db-5547cdbc5-65k79 -- /bin/bash

I have no name!@academy-db-5547cdbc5-65k79:/$ MysqL
ERROR 2002 (HY000): Can't connect to local MysqL server through socket '/opt/bitnami/mariadb/tmp/MysqL.sock' (2)

是否知道可能导致问题的原因，或者我如何进一步调查该问题？ （注意：我不是 Kubernetes 的专家，但最近才开始学习）

编辑：按照@Novo 的评论，我尝试删除卷文件夹并让 mariadb 从头开始​​重新创建部署。 现在我的 pod 甚至没有启动，在 CrashLoopBackOff 中终止！

通过比较 pod 日志，我注意到在之前的 mariabd 日志中有一条消息：

...
mariadb 10:32:05.69 WARN  ==> The mariadb configuration file '/opt/bitnami/mariadb/conf/my.cnf' is not writable. Configurations based on environment variables will not be applied for this file.
mariadb 10:32:05.70 INFO  ==> Using persisted data
mariadb 10:32:05.71 INFO  ==> Running MysqL_upgrade
mariadb 10:32:05.71 INFO  ==> Starting mariadb in background

现在替换为

...
mariadb 14:15:57.32 INFO  ==> Updating 'my.cnf' with custom configuration
mariadb 14:15:57.32 INFO  ==> Setting user option
mariadb 14:15:57.35 INFO  ==> Installing database

会不会是主机vagrant机器中的volume文件夹的访问权限问题？

解决方法

默认情况下，hostPath 目录的创建权限为 755，由 kubelet 的用户和组拥有。要使用该目录，您可以尝试将以下内容添加到您的部署中：

  spec:
     securityContext:
       fsGroup: <gid>

其中 gid 是容器中进程使用的组。

此外，您可以通过更改要装入容器的文件夹的权限来解决主机本身的问题：

chown-R <uid>:<gid> /path/to/volume

其中 uid 和 gid 是您应用中的 userId 和 groupId。

chmod -R 777 /path/to/volume

这应该可以解决您的问题。

但总的来说，在这种情况下，部署不是您想要创建的，因为部署不应该有状态。对于有状态应用程序，Kubernetes 中有“StatefulSets”。将它们与“VolumeClaimTemplate”和 spec.securityContext.fsgroup 一起使用，k3s 将为您创建持久卷和持久卷声明，使用它的默认存储类，即本地存储（在您的节点上）。