问题描述
我使用 Prestashop+ Nginx + PHP-FPM，有超过 30.000 个列表+变体，由外部 API 服务器同步。
我的 prestashop 正在运行：
AMD Ryzen™ 5 3600（6 核）+ 64GB RAM + 2xNVME 500GB (RAID 1) + 千兆以太网/网络
我不是专家，我很想得到一些建议
/etc/my.cnf
[MysqL]
port = 3306
[MysqLd]
# === required Settings ===
basedir = /usr
bind_address = 127.0.0.1 # Change to 0.0.0.0 to allow remote connections
datadir = /var/lib/MysqL
max_allowed_packet = 256M
max_connect_errors = 1000000
port = 3306
skip_external_locking
skip_name_resolve
tmpdir = /tmp
user = MysqL
# === sql Compatibility Mode ===
# Enable for b/c with databases created in older MysqL/MariaDB versions
# (e.g. when using null dates)
#sql_mode = ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION,ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES
# === InnoDB Settings ===
default_storage_engine = InnoDB
innodb_buffer_pool_instances = 42 # Use 1 instance per 1GB of InnoDB pool size
innodb_buffer_pool_size = 42G # Use up to 70-80% of RAM
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 0
innodb_flush_method = O_DIRECT
innodb_log_buffer_size = 16M
innodb_log_file_size = 2G
innodb_stats_on_Metadata = 0
performance_schema = ON
#innodb_temp_data_file_path = ibtmp1:64M:autoextend:max:20G # Control the maximum size for the ibtmp1 file
#innodb_thread_concurrency = 4 # Optional: Set to the number of cpus on your system (minus 1 or 2) to better
# contain cpu usage. E.g. if your system has 8 cpus,try 6 or 7 and check
# the overall load produced by MysqL/MariaDB.
innodb_read_io_threads = 64
innodb_write_io_threads = 64
#innodb_io_capacity = 1000 # Max is 2000
# === MyISAM Settings ===
# The following 3 options are ONLY supported by MariaDB & up to MysqL 5.7
# Do NOT un-comment on MysqL 8.x+
query_cache_limit = 10M # UPD
query_cache_size = 50000M # UPD
query_cache_type = 1 # Enabled by default
key_buffer_size = 32M # UPD
low_priority_updates = 1
concurrent_insert = 2
# === Connection Settings ===
max_connections = 750 # UPD - Important: high no. of connections = high RAM consumption
back_log = 512
thread_cache_size = 100
thread_stack = 192K
interactive_timeout = 180
wait_timeout = 180
# For MysqL 5.7+ only (disabled by default)
#max_execution_time = 30000 # Set a timeout limit for SELECT statements (value in milliseconds).
# This option may be useful to address aggressive crawling on large sites,# but it can also cause issues (e.g. with backups). So use with extreme caution and test!
# More info at: https://dev.MysqL.com/doc/refman/5.7/en/server-system-variables.html#sysvar_max_execution_time
# For MariaDB 10.1.1+ only (disabled by default)
#max_statement_time = 30 # The equivalent of "max_execution_time" in MysqL 5.7+ (set above)
# The variable is of type double,thus you can use subsecond timeout.
# For example you can use value 0.01 for 10 milliseconds timeout.
# More info at: https://mariadb.com/kb/en/aborting-statements/
# === Buffer Settings ===
innodb_sort_buffer_size = 2M # UPD
join_buffer_size = 8M # UPD
read_buffer_size = 3M # UPD
read_rnd_buffer_size = 4M # UPD
sort_buffer_size = 4M # UPD
# === Table Settings ===
# In systemd managed systems like Ubuntu 16.04+ or CentOS 7+,you need to perform an extra action for table_open_cache & open_files_limit
# to be overriden (also see comment next to open_files_limit).
# E.g. for MysqL 5.7,please check: https://dev.MysqL.com/doc/refman/5.7/en/using-systemd.html
# and for MariaDB check: https://mariadb.com/kb/en/library/systemd/
table_deFinition_cache = 40000 # UPD
table_open_cache = 40000 # UPD
open_files_limit = 80000 # UPD - This can be 2x to 3x the table_open_cache value or match the system's
# open files limit usually set in /etc/sysctl.conf or /etc/security/limits.conf
# In systemd managed systems this limit must also be set in:
# /etc/systemd/system/MysqLd.service.d/override.conf (for MysqL 5.7+) and
# /etc/systemd/system/mariadb.service.d/override.conf (for MariaDB)
max_heap_table_size = 128M
tmp_table_size = 128M
# === Search Settings ===
ft_min_word_len = 3 # Minimum length of words to be indexed for search results
# === Logging ===
log_error = /var/lib/MysqL/MysqL_error.log
log_queries_not_using_indexes = 1
long_query_time = 5
slow_query_log = 1 # disabled for production
slow_query_log_file = /var/lib/MysqL/MysqL_slow.log
[MysqLdump]
# Variable reference
# For MysqL 5.7+: https://dev.MysqL.com/doc/refman/5.7/en/MysqLdump.html
# For MariaDB: https://mariadb.com/kb/en/library/MysqLdump/
quick
quote_names
max_allowed_packet = 512M
tuning-primer.sh
[root@server2 /]# curl -L https://raw.githubusercontent.com/BMDan/tuning-primer.sh/master/tuning-primer.sh | bash
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 63986 100 63986 0 0 201k 0 --:--:-- --:--:-- --:--:-- 202k
-- MysqL PERFORMANCE TUNING PRIMER --
- By: Matthew Montgomery -
MysqL Version 10.2.38-MariaDB-log x86_64
Uptime = 3 days 21 hrs 47 min 51 sec
Avg. qps = 18
Total Questions = 6119342
Threads Connected = 11
Server has been running for over 48hrs.
It should be safe to follow these recommendations
To find out more information on how each of these
runtime variables effects performance visit:
http://dev.MysqL.com/doc/refman/10.2/en/server-system-variables.html
Visit http://www.MysqL.com/products/enterprise/advisors.html
for info about MysqL's Enterprise Monitoring and Advisory Service
SLOW QUERIES
The slow query log is enabled.
Current long_query_time = 5.000000 sec.
Since startup,102804 out of 6119356 queries have taken longer than <long_query_time-when-they-were-executed> to complete.
Your long_query_time may be too high,I typically set this under 5 sec.
BINARY UPDATE LOG
The binary update log is NOT enabled.
You will not be able to do point in time recovery
See http://dev.MysqL.com/doc/refman/10.2/en/point-in-time-recovery.html
WORKER THREADS
Current thread_cache_size = 100
Current threads_cached = 42
Current threads_per_sec = 0
Historic threads_per_sec = 0
Your thread_cache_size is fine
MAX CONNECTIONS
Current max_connections = 750
Current threads_connected = 11
Historic max_used_connections = 54
The number of used connections is 7% of the configured maximum.
You are using less than 10% of your configured max_connections.
Lowering max_connections Could help to avoid an over-allocation of memory
See "MEMORY USAGE" section to make sure you are not over-allocating
INNODB STATUS
Current InnoDB index space = 137 M
Current InnoDB data space = 255 M
Current InnoDB buffer pool free = 99 %
Current innodb_buffer_pool_size = 42.00 G
Depending on how much space your innodb indexes take up it may be safe
to increase this value to up to 2 / 3 of total system memory
MEMORY USAGE
Max Memory Ever Allocated : 91.88 G
Configured Max Per-thread Buffers : 14.05 G
Configured Max Global Buffers : 90.87 G
Configured Max Memory Limit : 104.92 G
Physical Memory : 62.75 G
Max memory limit exceeds 90% of physical memory
KEY BUFFER
Current MyISAM index space = 185 K
Current key_buffer_size = 32 M
Key cache miss rate is 1 : 697
Key buffer free ratio = 81 %
Your key_buffer_size seems to be fine
QUERY CACHE
Query cache is enabled
Current query_cache_size = 48.82 G
Current query_cache_used = 1.17 G
Current query_cache_limit = 10 M
Current Query cache Memory fill ratio = 2.39 %
Current query_cache_min_res_unit = 4 K
Your query_cache_size seems to be too high.
Perhaps you can use these resources elsewhere
MysqL won't cache query results that are larger than query_cache_limit in size
SORT OPERATIONS
Current sort_buffer_size = 4 M
Current read_rnd_buffer_size = 4 M
Sort buffer seems to be fine
JOINS
Current join_buffer_size = 8.00 M
You have had 15352 queries where a join Could not use an index properly
join_buffer_size >= 4 M
This is not advised
You should enable "log-queries-not-using-indexes"
Then look for non indexed joins in the slow query log.
OPEN FILES LIMIT
Current open_files_limit = 32768 files
The open_files_limit should typically be set to at least 2x-3x
that of table_cache if you have heavy MyISAM usage.
Your open_files_limit value seems to be fine
TABLE CACHE
Current table_open_cache = 15994 tables
Current table_deFinition_cache = 40000 tables
You have a total of 483 tables
You have 557 open tables.
The table_cache value seems to be fine
TEMP TABLES
Current max_heap_table_size = 128 M
Current tmp_table_size = 128 M
Of 146965 temp tables,6% were created on disk
Created disk tmp tables ratio seems fine
TABLE SCANS
Current read_buffer_size = 3 M
Current table scan ratio = 88 : 1
read_buffer_size seems to be fine
TABLE LOCKING
Current Lock Wait ratio = 1 : 37
You may benefit from selective use of InnoDB.
解决方法
当前 query_cache_size = 48.82 G -- 不！停在50M左右；当它那么大时，它会变得非常低效。
在slowlog上使用pt-query-digest；然后让我们讨论前几个查询。这可能对性能有很大帮助。
log_queries_not_using_indexes = 1 -- 关闭；否则它会使慢日志变得混乱。
我们将在第二遍中解决这个问题：
max_connections = 750 # UPD
table_definition_cache = 40000 # UPD
table_open_cache = 40000 # UPD
如果您想了解更多分析，请参阅http://mysql.rjweb.org/doc.php/mysql_analysis#tuning
摘要
查询 1
UPDATE ps_stock_available sa
INNER JOIN ps_mmo_connector_product_map pm
ON sa.id_product = pm.id_product_shop
SET sa.quantity = '10'
WHERE sa.quantity > '10'\G
摘要中的第一个查询是似乎限制 quantity 的更新。据推测，您可以通过永远不将值更改为超过 10 来消除它？这将消除服务器的一半负担。如果您不想这样做，请提供这些以便我检查索引：
SHOW CREATE TABLE `presta`.`ps_stock_available`\G
SHOW CREATE TABLE `presta`.`ps_mmo_connector_product_map`\G
但是……有什么问题吗？这意味着没有太多事情发生：
# Overall: 281.15k total,799 unique,0.54 QPS,0.01x concurrency
这说明“最差”的查询（即更新）不是很慢或很频繁：
# Exec time 48 1313s 18ms 1s 85ms 253ms 88ms 53ms
如果您希望服务器变得更加繁忙，那么我会深入挖掘。但现在，服务器唯一能做的就是打鼾。
在获取这些 Creates 的同时，也抓住这些，以便我可以检查接下来的几个查询：
SHOW CREATE TABLE `presta`.`ps_category_product`\G
SHOW CREATE TABLE `presta`.`ps_category`\G
SHOW CREATE TABLE `presta`.`ps_category_shop`\G
SHOW CREATE TABLE `presta`.`ps_search_word`\G
SHOW CREATE TABLE `presta`.`ps_search_index`\G
SHOW TABLE STATUS FROM `presta` LIKE 'ps_tag_count'\G
我看到一个 REPLACE 似乎正在重新计算标签计数。乍一看，这似乎是一种非常低效的方法。但是看看查询有多复杂，改进它可能不切实际。你是查询 4 的意图吗？