Helm + Kubernetes + Airflow - Worker Pod 未使用环境变量启动

编程问答 2022-08-06

问题描述

在 github 气流稳定的 helm 存储库中对以下问题的任何帮助都会很棒。

气流：1.10.14
Kubernetes：v1.18.8
Minikube：v1.17.1
掌舵：v3.5.2

https://github.com/airflow-helm/charts/issues/77

我正在尝试使用 extraEnv 设置一些环境变量，我希望这些变量应该传播到 Worker pod。情况似乎并非如此，我可以看到所有值都在调度程序 pod 中正确设置，但是当它产生一个在 extraEnv 下设置的工作器值时似乎不存在，我在这里做错了什么?还是这是一个错误？

气流：1.10.14 Kubernetes：v1.18.8 Minikube：v1.17.1 掌舵：v3.5.2

以下是我的自定义值文件，

airflow:
  image:
    repository: airflow-global-local
    tag: 1.0.0
    pullPolicy: IfNotPresent

  executor: KubernetesExecutor

  fernetKey: "xxxxxx"

  config:
    # Security
    AIRFLOW__CORE__SECURE_MODE: "False"
    AIRFLOW__API__AUTH_BACKEND: "airflow.api.auth.backend.deny_all"
    AIRFLOW__WEBSERVER__EXPOSE_CONfig: "False"
    AIRFLOW__WEBSERVER__RBAC: "False"

    # DAGS
    AIRFLOW__CORE__LOAD_EXAMPLES: "False"

    #K8s
    AIRFLOW__KUBERNETES__DAGS_IN_IMAGE: "True"
    AIRFLOW__KUBERNETES__DELETE_WORKER_PODS: "True"
    AIRFLOW__KUBERNETES__DELETE_WORKER_PODS_ON_FAILURE: "True"

    #remote logging to s3
    AIRFLOW__CORE__REMOTE_LOGGING: "True"
    AIRFLOW__CORE__REMOTE_BASE_LOG_FOLDER: "s3://xxxx"
    AIRFLOW__CORE__REMOTE_LOG_CONN_ID: "aws_logging"

    ## disable noisy "Handling signal: ttou" Gunicorn log messages
    GUNICORN_CMD_ARGS: "--log-level WARNING"
    ENVIRONMENT: "dev"
    
  extraEnv:
    - name: VAULT_ROLE_ID
      valueFrom:
        secretKeyRef:
          name: airflow-core-secrets
          key: VAULT_ROLE_ID
    - name: VAULT_SECRET_ID
      valueFrom:
        secretKeyRef:
          name: airflow-core-secrets
          key: VAULT_SECRET_ID    
    - name: TEAMS_WEBHOOK
      valueFrom:
        secretKeyRef:
          name: airflow-core-secrets
          key: TEAMS_WEBHOOK

externalDatabase:
  type: postgres
  host: xxxxx
  port: 5432
  database: airflow
  user: root
  passwordSecret: "airflow-core-secrets"
  passwordSecretKey: "DB_PASSWORD"

scheduler:
  connections:
    - id: aws_logging
      type: aws
      extra: |
        {
          "aws_access_key_id": "xxxxx","aws_secret_access_key": "xxxxx","region_name":"eu-west-1"
        }
  livenessProbe:
    enabled: true
    initialDelaySeconds: 300
    periodSeconds: 30
    failureThreshold: 5


postgresql:
  enabled: False

redis:
  enabled: False

flower:
  enabled: False

workers:
  enabled: False

非常感谢任何帮助。

解决方法

暂无找到可以解决该程序问题的有效方法，小编努力寻找整理中！

如果你已经找到好的解决方法，欢迎将解决方案带上本链接一起发送给小编。

小编邮箱:dio#foxmail.com (将#修改为@）

airflowairflowkuberneteskubernetes-helmworkerworker