目录
1.环境版本
操作系统版本
cat /proc/version
Linux version 3.10.0-1062.9.1.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC) ) #1 SMP Fri Dec 6 15:49:49 UTC 2019数据库版本
MysqL> select version();
+-----------+
| version() |
+-----------+
| 8.0.17 |
+-----------+
1 row in set (0.00 sec)2.问题描述
研发兄弟反馈说，他们的压测数据库中有一些表不能正常查询。我试了一下查询会报如下错误：
MysqL> select * from test_part;
ERROR 2013 (HY000): Lost connection to MysqL server during query
日志中报错如下：
2020-07-06T17:42:52.413990+08:00 64 [ERROR] [MY-013183] [InnoDB] Assertion failure: btr0pcur.cc:318:page_is_comp(next_page) == page_is_comp(page) thread 140622855923456
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.MysqL.com.
InnoDB: If you get repeated assertion failures or crashes,even
InnoDB: immediately after the MysqLd startup,there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.MysqL.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
09:42:52 UTC - MysqLd got signal 6 ;
Most likely,you have hit a bug,but this error can also be caused by malfunctioning hardware.
Thread pointer: 0x7fe0d80102c0
Attempting backtrace. You can use the following information to find out
where MysqLd died. If you see no messages after this,something went
terribly wrong...
stack_bottom = 7fe54f5fdc30 thread_stack 0x46000
/usr/sbin/MysqLd(my_print_stacktrace(unsigned char*,unsigned long)+0x3d) [0x1e435bd]
/usr/sbin/MysqLd(handle_fatal_signal+0x333) [0xf25e83]
/lib64/libpthread.so.0(+0xf5f0) [0x7fe5cf88a5f0]
/lib64/libc.so.6(gsignal+0x37) [0x7fe5cdb9f337]
/lib64/libc.so.6(abort+0x148) [0x7fe5cdba0a28]
/usr/sbin/MysqLd() [0xca40da]
/usr/sbin/MysqLd(btr_pcur_t::move_to_next_page(mtr_t*)+0x1b8) [0x212d458]
/usr/sbin/MysqLd(row_search_mvcc(unsigned char*,page_cur_mode_t,row_prebuilt_t*,unsigned long,unsigned long)+0x118a) [0x20620ea]
/usr/sbin/MysqLd(ha_innobase::general_fetch(unsigned char*,unsigned int,unsigned int)+0xd3) [0x1f07d53]
/usr/sbin/MysqLd(handler::ha_rnd_next(unsigned char*)+0x66) [0x102f606]
/usr/sbin/MysqLd(TableScanIterator::Read()+0x1d) [0xd51b1d]
/usr/sbin/MysqLd(JOIN::exec()+0x45e) [0xdbbe6e]
/usr/sbin/MysqLd(sql_cmd_dml::execute_inner(THD*)+0x2cc) [0xe4088c]
/usr/sbin/MysqLd(sql_cmd_dml::execute(THD*)+0x418) [0xe47e18]
/usr/sbin/MysqLd(MysqL_execute_command(THD*,bool)+0x14a0) [0xdf4360]
/usr/sbin/MysqLd(MysqL_parse(THD*,Parser_state*)+0x353) [0xdf8703]
/usr/sbin/MysqLd(dispatch_command(THD*,COM_DATA const*,enum_server_command)+0x2d10) [0xdfb920]
/usr/sbin/MysqLd(do_command(THD*)+0x1b4) [0xdfc404]
/usr/sbin/MysqLd() [0xf17630]
/usr/sbin/MysqLd() [0x233495c]
/lib64/libpthread.so.0(+0x7e65) [0x7fe5cf882e65]
/lib64/libc.so.6(clone+0x6d) [0x7fe5cdc6788d]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7fe0d9d17b28): select * from test_part
Connection ID (thread ID): 64
Status: NOT_KILLED
The manual page at http://dev.MysqL.com/doc/MysqL/en/crashing.html contains
information that should help you find out what is causing the crash.日志开头 Assertion failure: btr0pcur.cc:318:page_is_comp(next_page) == page_is_comp(page)，扫描全表时遇到坏块
MysqL> check table t_order_viapoint;
ERROR 2013 (HY000): Lost connection to MysqL server during query
check table 报错后查看errorlog 日志
2020-07-06T17:43:28.385547+08:00 16 [ERROR] [MY-013051] [InnoDB] In pages [page id: space=105,page number=21] and [page id: space=105,page number=22] of index `PRIMARY` of table `saic_trip_order2`.`test_part`
InnoDB: broken FIL_PAGE_NEXT or FIL_PAGE_PREV links
2020-07-06T17:43:28.385628+08:00 16 [ERROR] [MY-013051] [InnoDB] In pages [page id: space=105,page number=22] of index `PRIMARY` of table `saic_trip_order2`.`test_part`
InnoDB: 'compact' flag mismatch
2020-07-06T17:43:28.385646+08:00 16 [ERROR] [MY-011866] [InnoDB] Page index id 0 != data dictionary index id 443
2020-07-06T17:43:28.385658+08:00 16 [ERROR] [MY-013183] [InnoDB] Assertion failure: btr0btr.cc:4256:!page_is_empty(page) || (level == 0 && page_get_page_no(page) == dict_index_get_page(index)) thread 139819373106944
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.MysqL.com.
InnoDB: If you get repeated assertion failures or crashes,there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.MysqL.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
09:43:28 UTC - MysqLd got signal 6 ;
Most likely,but this error can also be caused by malfunctioning hardware.
Thread pointer: 0x7f25940008c0
Attempting backtrace. You can use the following information to find out
where MysqLd died. If you see no messages after this,something went
terribly wrong...
stack_bottom = 7f2a3c110c30 thread_stack 0x46000
/usr/sbin/MysqLd(my_print_stacktrace(unsigned char*,unsigned long)+0x3d) [0x1e435bd]
/usr/sbin/MysqLd(handle_fatal_signal+0x333) [0xf25e83]
/lib64/libpthread.so.0(+0xf5f0) [0x7f2a6e6475f0]
/lib64/libc.so.6(gsignal+0x37) [0x7f2a6c95c337]
/lib64/libc.so.6(abort+0x148) [0x7f2a6c95da28]
/usr/sbin/MysqLd() [0xca40da]
/usr/sbin/MysqLd() [0x211789b]
/usr/sbin/MysqLd(btr_validate_index(dict_index_t*,trx_t const*,bool)+0x3f5) [0x2117eb5]
/usr/sbin/MysqLd(ha_innobase::check(THD*,HA_CHECK_OPT*)+0x3c6) [0x1f0c7a6]
/usr/sbin/MysqLd(handler::ha_check(THD*,HA_CHECK_OPT*)+0x13b) [0x1032aeb]
/usr/sbin/MysqLd() [0x120becb]
/usr/sbin/MysqLd(sql_cmd_check_table::execute(THD*)+0x99) [0x120ccd9]
/usr/sbin/MysqLd(MysqL_execute_command(THD*,enum_server_command)+0x2d10) [0xdfb920]
/usr/sbin/MysqLd(do_command(THD*)+0x1b4) [0xdfc404]
/usr/sbin/MysqLd() [0xf17630]
/usr/sbin/MysqLd() [0x233495c]
/lib64/libpthread.so.0(+0x7e65) [0x7f2a6e63fe65]
/lib64/libc.so.6(clone+0x6d) [0x7f2a6ca2488d]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f2594008a98): check table test_part
Connection ID (thread ID): 16
Status: NOT_KILLED
The manual page at http://dev.MysqL.com/doc/MysqL/en/crashing.html contains
information that should help you find out what is causing the crash.InnoDB: broken FIL_PAGE_NEXT or FIL_PAGE_PREV links，innodb 数据页双向链表出现问题(比如FILE_PAGE_NEXT指定下一个页是 page:32,但是读取到的页号却不是32)
2020-07-06T17:43:28.385646+08:00 16 [ERROR] [MY-011866] [InnoDB] Page index id 0 != data dictionary index id 443
2020-07-06T17:43:28.385658+08:00 16 [ERROR] [MY-013183] [InnoDB] Assertion failure: btr0btr.cc:4256:!page_is_empty(page) || (level == 0 && page_get_page_no(page) == dict_index_get_page(index)) thread 139819373106944
数据字典中该表和索引相关元数据信息如下：
select t.TABLE_ID,t.NAME table_name,t.ROW_FORMAT,t.SPACE table_sapce,i.INDEX_ID,i.NAME index_name,i.TYPE index_type,i.PAGE_NO index_page,i.SPACE index_space from information_schema.INNODB_TABLES t,information_schema.INNODB_INDEXES i where t.TABLE_ID=i.TABLE_ID and i.INDEX_ID=443;
+----------+-----------------------------------+------------+-------------+----------+------------+------------+------------+-------------+
| TABLE_ID | table_name | ROW_FORMAT | table_sapce | INDEX_ID | index_name | index_type | index_page | index_space |
+----------+-----------------------------------+------------+-------------+----------+------------+------------+------------+-------------+
| 1162 | test_shao/test_part | Dynamic | 105 | 443 | PRIMARY | 3 | 4 | 105 |
+----------+-----------------------------------+------------+-------------+----------+------------+------------+------------+-------------+3.问题处理
恢复备份数据
