从Airflow Celery Executor连接到启用TLS的Elasticache Redis

问题描述

我当前在Airflow 1.10上使用Celery Executor。我的经纪人是AWS Elasticache Redis（v。5.0.6）。如何启用传输加密？

根据Airflow源代码，ssl_keyfile，ssl_certfile和ssl_ca_certs是必需的。

elif 'redis://' in broker_url:
    broker_use_ssl = {'ssl_keyfile': conf.get('celery','SSL_KEY'),'ssl_certfile': conf.get('celery','SSL_CERT'),'ssl_ca_certs': conf.get('celery','SSL_CACERT'),'ssl_cert_reqs': ssl.CERT_required}
 

https://github.com/apache/airflow/blob/1.10.10/airflow/config_templates/default_celery.py#L68-L72

但是Elasticache Redis不提供那些TLS证书。官方文档仅解释了redis-cli的解决方案，该解决方案使用了stunnel的TLS隧道技术。

https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/in-transit-encryption.html#connect-tls

Airflow Celery Executor是否在Elasticache Redis上支持加密传输中？如果是这样，我们如何实现呢？

解决方法

以下气流配置适用于Elasticache和Redis SSL 请注意'

上的多余subfields

db.collection.aggregate([
  {
    $set: {
      subfields: { $objectToArray: [ "$field2" ] }
    }
  },{
    $set: {
      subfields: {
        $reduce: {
          input: { $concatArrays: [ "$subfields.v" ] },initialValue: [],in: { $concatArrays: [ "$$value","$$this" ] }
        }
      }
    }
  },{$match: {subfields: "whatever you are looking for"}},{$unset: "subfields"}
])

s