Docker，Celery，撰写方法失败

编程问答 2022-08-11

问题描述

我正在尝试将芹菜，redis和rabitMQ作为后台任务的基于FastAPI的API。 在进行docker-compose up后，redis，兔子和花朵的零件正常工作，我可以访问花朵的仪表板。

但是它被卡在芹菜部分。

错误：

 rabbitmq_1       | 2020-09-08 06:32:38.552 [info] <0.716.0> connection <0.716.0> (172.22.0.6:49290 -> 172.22.0.2:5672): user 'user' authenticated and granted access to vhost '/'
celery-flower_1  | [W 200908 06:32:41 control:44] 'stats' inspect method Failed
celery-flower_1  | [W 200908 06:32:41 control:44] 'active_queues' inspect method Failed
celery-flower_1  | [W 200908 06:32:41 control:44] 'registered' inspect method Failed
celery-flower_1  | [W 200908 06:32:41 control:44] 'scheduled' inspect method Failed
celery-flower_1  | [W 200908 06:32:41 control:44] 'active' inspect method Failed
celery-flower_1  | [W 200908 06:32:41 control:44] 'reserved' inspect method Failed
celery-flower_1  | [W 200908 06:32:41 control:44] 'revoked' inspect method Failed
celery-flower_1  | [W 200908 06:32:41 control:44] 'conf' inspect method Failed

我的docker-compose文件：

version: "3.7"

services:
  rabbitmq:
    image: "bitnami/rabbitmq:3.7"
    ports:
      - "4000:4000"
      - "5672:5672"
    volumes:
      - "rabbitmq_data:/bitnami"

  redis:
    image: "bitnami/redis:5.0.4"
    environment:
      - REdis_PASSWORD=password123
    ports:
      - "5000:5000"
    volumes:
      - "redis_data:/bitnami/redis/data"

  celery-flower:
    image: gregsi/latest-celery-flower-docker:latest
    environment:
      - AMQP_USERNAME=user
      - AMQP_PASSWORD=bitnami
      - AMQP_ADMIN_USERNAME=user
      - AMQP_ADMIN_PASSWORD=bitnami
      - AMQP_HOST=rabbitmq
      - AMQP_PORT=5672
      - AMQP_ADMIN_HOST=rabbitmq
      - AMQP_ADMIN_PORT=15672
      - FLOWER_BASIC_AUTH=user:test
    ports:
      - "5555:5555"
    depends_on:
      - rabbitmq
      - redis

  fastapi:
    build: .
    ports:
      - "8000:8000"
    depends_on:
      - rabbitmq
      - redis
    volumes:
      - "./:/app"
    command: "poetry run uvicorn app/app/main:app --bind 0.0.0.0:8000"

  worker:
    build: .
    depends_on:
      - rabbitmq
      - redis
    volumes:
      - "./:/app"
    command: "poetry run celery worker -A app.app.worker.celery_worker -l info -Q test-queue -c 1"

volumes:
  rabbitmq_data:
    driver: local
  redis_data:
    driver: local

我的芹菜应用程序：

celery_app = Celery(
    "worker",backend="redis://:password123@redis:6379/0",broker="amqp://user:bitnami@rabbitmq:5672//"
)

celery_app.conf.task_routes = {
    "app.app.worker.celery_worker.compute_stock_indicators": "stocks-queue"
}

celery_app.conf.update(task_track_started=True)

芹菜工人：

@celery_app.task(acks_late=True)
def compute_stock_indicators(stocks: list,background_task):
    stocks_with_indicators = {}
    for stock in stocks:
        current_task.update_state(state=Actions.STARTED,Meta={f"starting to fetch {stock}'s indicators"})

        stock_indicators = fetch_stock_indicators(stock)  # Fetch the stock most recent indicators
        current_task.update_state(state=Actions.FINISHED,Meta={f"{stock}'s indicators fetched"})

        stocks_with_indicators.update({stock: stock_indicators})

        current_task.update_state(state=Actions.PROGRESS,Meta={f"predicting {stocks}s..."})

快速API函数：

log = logging.getLogger(__name__)
rabbit = RabbitMQHandler(host='localhost',port=5672,level="DEBUG")
log.addHandler(rabbit)


def celery_on_message(body):
    """
    Logs the initiation of the function
    """
    log.warning(body)


def background_on_message(task):
    """
    logs the function when it is added to queue
    """
    log.warning(task.get(on_message=celery_on_message,propagate=False))


app = FastAPI(debug=True)


@app.post("/")
async def initiator(stocks: FrozenSet,background_task: BackgroundTasks,):
    """
    :param stocks: stocks to be analyzed
    :type stocks: set
    :param background_task: initiate the tasks queue
    :type background_task: starlette.background.BackgroundTasks
    """
    log.warning(msg=f'beginning analysis on: {stocks}')
    task_name = "app.app.worker.celery_worker.compute_stock_indicators"

    task = celery_app.send_task(task_name,args=[stocks,background_task])
    background_task.add_task(background_on_message,task)
    return {"message": "Stocks indicators successfully calculated,stocks sent to prediction"}

解决方法

在docker-compose的{​​{1}}部分，worker读取：

command

因此，实质上，您是在要求工作人员“监视”名为command: "poetry run celery worker -A app.app.worker.celery_worker -l info -Q test-queue -c 1" 的队列。
但是在test-queue的以下部分中：

celery_app

您正在定义一个名为celery_app.conf.task_routes = { "app.app.worker.celery_worker.compute_stock_indicators": "stocks-queue" } 的队列。

更改stocks-queue或docker-compose的队列名称以使其彼此匹配。

dockerdockerfastapiflowerpython