CeleryD似乎忽略了并发参数

问题描述

我最近将Django项目升级到了Celery 4.4.6，事情进展不顺利。 我当前的首要问题是任务的并发性。因为这些任务锁定数据库表，而有些任务占用大量内存，所以没有机会同时运行八个任务。我也只有2个处理器的机器可用。但是，芹菜就是这样做的。

以前，我只能同时运行两个任务。

该工作程序被守护，只有一个工作程序处于活动状态（一个节点）。我将并发设置为两个。这是我的/ etc / default / celeryd：

#   most people will only start one node:
CELERYD_NODES="worker1"
#   but you can also start multiple and configure settings
#   for each in CELERYD_OPTS
#CELERYD_NODES="worker1 worker2 worker3"
#   alternatively,you can specify the number of nodes to start:
#CELERYD_NODES=3

# Absolute or relative path to the 'celery' command:
CELERY_BIN="/home/ubuntu/dev/bin/python -m celery"
#CELERY_BIN="/virtualenvs/def/bin/celery"

# App instance to use
# comment out this line if you don't use an app
CELERY_APP="match2"
# or fully qualified:
#CELERY_APP="proj.tasks:app"

# Where to chdir at start.
export DJANGO_SETTINGS_MODULE="match2.settings"
CELERYD_CHDIR="/home/ubuntu/dev/match2/match2"

# Extra command-line arguments to the worker
CELERYD_OPTS="--concurrency=2"
# Configure node-specific settings by appending node name to arguments:
#CELERYD_OPTS="--time-limit=300 -c 8 -c:worker2 4 -c:worker3 2 -Ofair:worker1"

# Set logging level to DEBUG
CELERYD_LOG_LEVEL="INFO"

# %n will be replaced with the first part of the nodename.
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_PID_FILE="/var/run/celery/%n.pid"

# Workers should run as an unprivileged user.
#   You need to create this user manually (or you can choose
#   a user/group combination that already exists (e.g.,nobody).
CELERYD_USER="ubuntu"
CELERYD_GROUP="users"

# If enabled pid and log directories will be created if missing,# and owned by the userid/group configured.
CELERY_CREATE_Dirs=1

我非常假设此行将负责可以同时执行多少个任务：CELERYD_OPTS="--concurrency=2" 但是似乎它仍然从RabbitMQ消息队列中拾取多达八项。

任何帮助表示赞赏。

解决方法

因此，经过一番往返并加入Google网上论坛，我终于得到了答案：

如果您希望Celery表现得像个好工人 ，并且在完成旧任务之前不承担其他任务，那么您需要在设置文件中同时包含这两个任务：

task_acks_late = True
worker_prefetch_multiplier = 1

如果您随后在带有旧式大写设置的Django项目中使用它（请参见：https://docs.celeryproject.org/en/stable/userguide/configuration.html#new-lowercase-settings），则可以将其转换为：

CELERY_WORKER_PREFETCH_MULTIPLIER = 1
CELERY_TASK_ACKS_LATE = True