MySQL Kubernetes Pod 进入 CrashLoopBackOff 并且无法解密具有静态加密的 .ibd MySQL pod 不会回到运行状态

问题描述

MysqL pods 重新启动时,由于表空间损坏或无法解密某些表空间 .ibd 文件,pods 不会回到运行状态。当通过 TDE 加密对表进行静态加密时,会特别观察到这种情况。

kubectl get pods -n t005-u000025
NAME                           READY   STATUS             RESTARTS   AGE
pbinnocluster-db-msql-node-01     1/2     CrashLoopBackOff   10         29m
pbinnocluster-db-msql-node-02     2/2     Running            0          22d
pbinnocluster-db-msql-node-03     1/2     CrashLoopBackOff   2981       10d
pbinnocluster-db-msql-router-01   1/1     Running            0          22d
pbinnocluster-db-msql-router-02   1/1     Running            0          12d

这是进入 CrashLoopBackOff 状态的 Pod 的日志

[Entrypoint] MysqL Docker Image 8.0.22-1.1.18
[Entrypoint] Creating MysqL directories
[Entrypoint] Generating/copying MysqL init files
2021-05-31T19:02:44.724547+09:00 0 [Warning] [MY-011068] [Server] The Syntax 'expire-logs-days' is deprecated and will be removed in a future release. Please use binlog_expire_logs_seconds instead.
2021-05-31T19:02:44.724702+09:00 0 [Warning] [MY-010915] [Server] 'NO_ZERO_DATE','NO_ZERO_IN_DATE' and 'ERROR_FOR_DIVISION_BY_ZERO' sql modes should be used with strict mode. They will be merged with strict mode in a future release.
2021-05-31T19:02:44.724939+09:00 0 [Warning] [MY-010101] [Server] Insecure configuration for --secure-file-priv: Location is accessible to all OS users. Consider choosing a different directory.
2021-05-31T19:02:44.735653+09:00 0 [Warning] [MY-000067] [Server] unkNown variable 'loose_group_replication_transaction_size_limit=2147483647'.
2021-05-31T19:02:44.735737+09:00 0 [Warning] [MY-000067] [Server] unkNown variable 'loose_group_replication_group_name=c6e27afa-b086-11eb-abdf-d7eb1299ca28'.
2021-05-31T19:02:44.735771+09:00 0 [Warning] [MY-000067] [Server] unkNown variable 'loose_group_replication_single_primary_mode=ON'.
2021-05-31T19:02:44.735808+09:00 0 [Warning] [MY-000067] [Server] unkNown variable 'loose_group_replication_enforce_update_everywhere_checks=OFF'.
2021-05-31T19:02:44.735847+09:00 0 [Warning] [MY-000067] [Server] unkNown variable 'loose_group_replication_recovery_use_ssl=ON'.
2021-05-31T19:02:44.735898+09:00 0 [Warning] [MY-000067] [Server] unkNown variable 'loose_group_replication_ssl_mode=required'.
2021-05-31T19:02:44.735951+09:00 0 [Warning] [MY-000067] [Server] unkNown variable 'loose_group_replication_local_address=planbudget-db-msql-node-01:11221'.
2021-05-31T19:02:44.736004+09:00 0 [Warning] [MY-000067] [Server] unkNown variable 'loose_group_replication_ip_allowlist=240b:c0e0:0204:5400:b414:0002:0000:0c57,240b:c0e0:0204:5400:b414:0002:0000:0e3c,240b:c0e0:0204:5400:b414:0002:0000:0e60'.
2021-05-31T19:02:44.736062+09:00 0 [Warning] [MY-000067] [Server] unkNown variable 'loose_group_replication_exit_state_action=READ_ONLY'.
2021-05-31T19:02:44.736119+09:00 0 [Warning] [MY-000067] [Server] unkNown variable 'loose_group_replication_start_on_boot=ON'.
2021-05-31T19:02:44.736183+09:00 0 [Warning] [MY-000067] [Server] unkNown variable 'loose_group_replication_group_seeds=planbudget-db-msql-node-02:11221,planbudget-db-msql-node-03:11221'.
2021-05-31T19:02:44.736445+09:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'MyISAM'
2021-05-31T19:02:44.736482+09:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'CSV'
[Entrypoint] Starting MysqL 8.0.22-1.1.18
2021-05-31T19:02:45.269735+09:00 0 [Warning] [MY-011068] [Server] The Syntax 'expire-logs-days' is deprecated and will be removed in a future release. Please use binlog_expire_logs_seconds instead.
2021-05-31T19:02:45.269849+09:00 0 [Warning] [MY-010915] [Server] 'NO_ZERO_DATE','NO_ZERO_IN_DATE' and 'ERROR_FOR_DIVISION_BY_ZERO' sql modes should be used with strict mode. They will be merged with strict mode in a future release.
2021-05-31T19:02:45.272196+09:00 0 [Warning] [MY-010101] [Server] Insecure configuration for --secure-file-priv: Location is accessible to all OS users. Consider choosing a different directory.
2021-05-31T19:02:45.272287+09:00 0 [Note] [MY-010949] [Server] Basedir set to /usr/.
2021-05-31T19:02:45.272305+09:00 0 [System] [MY-010116] [Server] /usr/sbin/MysqLd (MysqLd 8.0.22) starting as process 30
2021-05-31T19:02:45.281146+09:00 0 [Note] [MY-012366] [InnoDB] Using Linux native AIO
2021-05-31T19:02:45.284719+09:00 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2021-05-31T19:02:45.284990+09:00 1 [Note] [MY-013546] [InnoDB] Atomic write enabled
2021-05-31T19:02:45.285204+09:00 1 [Note] [MY-012932] [InnoDB] PUNCH HOLE support available
2021-05-31T19:02:45.285398+09:00 1 [Note] [MY-012943] [InnoDB] Mutexes and rw_locks use GCC atomic builtins
2021-05-31T19:02:45.285590+09:00 1 [Note] [MY-012944] [InnoDB] Uses event mutexes
2021-05-31T19:02:45.285759+09:00 1 [Note] [MY-012945] [InnoDB] GCC builtin __atomic_thread_fence() is used for memory barrier
2021-05-31T19:02:45.285955+09:00 1 [Note] [MY-012948] [InnoDB] Compressed tables use zlib 1.2.11
2021-05-31T19:02:45.295049+09:00 1 [Note] [MY-013251] [InnoDB] Number of pools: 1
2021-05-31T19:02:45.295702+09:00 1 [Note] [MY-012951] [InnoDB] Using cpu crc32 instructions
2021-05-31T19:02:45.298047+09:00 1 [Note] [MY-012203] [InnoDB] Directories to scan './'
2021-05-31T19:02:45.298559+09:00 1 [Note] [MY-012204] [InnoDB] Scanning './'
2021-05-31T19:02:45.371901+09:00 1 [Note] [MY-012208] [InnoDB] Completed space ID check of 65 files.
2021-05-31T19:02:45.374537+09:00 1 [Note] [MY-012955] [InnoDB] Initializing buffer pool,total size = 4.000000G,instances = 4,chunk size =128.000000M 
2021-05-31T19:02:45.834521+09:00 1 [Note] [MY-012957] [InnoDB] Completed initialization of buffer pool
2021-05-31T19:02:45.859219+09:00 0 [Note] [MY-011952] [InnoDB] If the MysqLd execution user is authorized,page cleaner thread priority can be changed. See the man page of setpriority().
2021-05-31T19:02:45.859826+09:00 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_0.dblwr' for doublewrite
2021-05-31T19:02:45.860750+09:00 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_1.dblwr' for doublewrite
2021-05-31T19:02:45.958296+09:00 1 [Note] [MY-013566] [InnoDB] Double write buffer files: 2
2021-05-31T19:02:45.958477+09:00 1 [Note] [MY-013565] [InnoDB] Double write buffer pages per instance: 4
2021-05-31T19:02:45.958640+09:00 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_0.dblwr' for doublewrite
2021-05-31T19:02:45.958823+09:00 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_1.dblwr' for doublewrite
2021-05-31T19:02:45.959450+09:00 1 [Note] [MY-012560] [InnoDB] The log sequence number 27798434 in the system tablespace does not match the log sequence number 525964267 in the ib_logfiles!
2021-05-31T19:02:45.959601+09:00 1 [Note] [MY-012551] [InnoDB] Database was not shutdown normally!
2021-05-31T19:02:45.959749+09:00 1 [Note] [MY-012552] [InnoDB] Starting crash recovery.
2021-05-31T19:02:45.960943+09:00 1 [Note] [MY-013086] [InnoDB] Starting to parse redo log at lsn = 525963788,whereas checkpoint_lsn = 525964267
2021-05-31T19:02:45.961123+09:00 1 [Note] [MY-012550] [InnoDB] Doing recovery: scanned up to log sequence number 525967677
2021-05-31T19:02:45.978463+09:00 1 [Note] [MY-013083] [InnoDB] Log background threads are being started...
2021-05-31T19:02:45.987563+09:00 1 [Note] [MY-012532] [InnoDB] Applying a batch of 15 redo log records ...
2021-05-31T19:02:45.990749+09:00 1 [ERROR] [MY-012657] [InnoDB] Encryption can't find master key,please check the keyring plugin is loaded.
2021-05-31T19:02:45.990990+09:00 1 [ERROR] [MY-012226] [InnoDB] Encryption information in datafile: ./PLAN_BUDGET/QRTZ_SCHEDULER_STATE.ibd can't be decrypted,please confirm the keyfile is match and keyring plugin is loaded.
2021-05-31T19:02:45.993855+09:00 1 [Note] [MY-012533] [InnoDB] 100%
2021-05-31T19:02:46.494224+09:00 1 [Note] [MY-012535] [InnoDB] Apply batch completed!
2021-05-31T19:02:46.494774+09:00 1 [Warning] [MY-013541] [InnoDB] Doublewrite page 17 for {space: 55,page_no:4} Could not be restored. File name unkNown for tablespace ID 55
2021-05-31T19:02:46.495394+09:00 1 [ERROR] [MY-012179] [InnoDB] Could not find any file associated with the tablespace ID: 55
2021-05-31T19:02:46.495884+09:00 1 [ERROR] [MY-012964] [InnoDB] Use --innodb-directories to find the tablespace files. If that fails then use --innodb-force-recovery=1 to ignore this and to permanently lose all changes to the missing tablespace(s)
2021-05-31T19:02:46.596669+09:00 1 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Generic error.
2021-05-31T19:02:46.994715+09:00 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine
2021-05-31T19:02:46.995569+09:00 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization Failed.
2021-05-31T19:02:46.996580+09:00 0 [ERROR] [MY-010119] [Server] Aborting
2021-05-31T19:02:46.996677+09:00 0 [Note] [MY-010120] [Server] binlog end
2021-05-31T19:02:46.996980+09:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'MyISAM'
2021-05-31T19:02:46.997071+09:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'InnoDB'
2021-05-31T19:02:46.997154+09:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'CSV'
2021-05-31T19:02:46.998977+09:00 0 [System] [MY-010910] [Server] /usr/sbin/MysqLd: Shutdown complete (MysqLd 8.0.22)  MysqL Community Server - GPL.

请帮助我们找出 MysqL 不返回运行状态的原因?即使密钥环路径和密钥环文件在初始化时对 MysqL 可用,也不解密加密表空间的确切原因是什么。

解决方法

暂无找到可以解决该程序问题的有效方法,小编努力寻找整理中!

如果你已经找到好的解决方法,欢迎将解决方案带上本链接一起发送给小编。

小编邮箱:dio#foxmail.com (将#修改为@)