AWS Batch：使用云形成将 efs 卷挂载到容器中

问题描述

我有一个带有作业定义的 AWS Batch 计算环境。

我使用 Cloud Formation 创建了这一切。

现在我想向此作业定义添加一个 EFS 卷（名称：EFS-000，文件系统 ID：fs-9999999）和一个 MountPoint。

我读过

• https://docs.aws.amazon.com/AmazonECS/latest/developerguide/tutorial-efs-volumes.html
• https://aws.amazon.com/es/premiumsupport/knowledge-center/batch-mount-efs

在第一个链接中，我们有一个任务定义示例（AWS ECS 而不是 AWS Batch 概念）

viewmodel Scope

将正确的代码添加到我的 Cloud Formation 配方中似乎很容易（我选择了 yaml 语法）。在我的 ContainerDeFinition 中，我添加了...

{
    "containerDeFinitions": [
        {
            "memory": 128,"portMappings": [
                {
                    "hostPort": 80,"containerPort": 80,"protocol": "tcp"
                }
            ],"essential": true,"mountPoints": [
                {
                    "containerPath": "/usr/share/Nginx/html","sourceVolume": "efs-html"
                }
            ],"name": "Nginx","image": "Nginx"
        }
    ],"volumes": [
        {
            "name": "efs-html","efsVolumeConfiguration": {
                "fileSystemId": "fs-1324abcd","transitEncryption": "ENABLED"
            }
        }
    ],"family": "efs-tutorial"
}

但是当我运行 Cloud Formation 配方时，我得到....

Volumes:
  - Name: SRV 
    EfsVolumeConfiguration:
      FileSystemId: fs-9999999
      TransitEncryption: ENABLED

The following resource(s) Failed to update: [ContentJob1].

如果 EfsVolumeConfiguration 不是有效的属性...

如何使用 Cloud Formation 将 EFS 卷添加到 AWS Batch 作业定义？

解决方法

在 CloudFormation 支持 EFS 之前，我使用了这个解决方案。不过还是有用的。

我创建了一个启动模板

  LaunchTemplate:
    Type: AWS::EC2::LaunchTemplate
    Properties: 
      LaunchTemplateName: !Join ['',[!Ref ServiceName,'-EC2-',LaunchTemplate]]
      LaunchTemplateData: 
        UserData:
          Fn::Base64: !Sub |
            MIME-Version: 1.0 
            Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="

            --==MYBOUNDARY==
            Content-Type: text/cloud-config; charset="us-ascii"

            runcmd:
            - mkdir /mnt/efs-misc
            - mkdir /mnt/efs-jobs
            - echo "${EfsJobs}:/ /mnt/efs-jobs nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=300,retrans=2,noresvport 0 0" >> /etc/fstab
            - echo "${EfsMisc}:/ /mnt/efs-misc nfs4 nfsvers=4.1,noresvport 0 0" >> /etc/fstab
            - mount -a
            --==MYBOUNDARY==--
        InstanceMarketOptions:
          MarketType: spot
          SpotOptions:
            MaxPrice: 0.096

然后我创建了一个使用这个模板的计算环境。

  BatchCompute:
    Type: 'AWS::Batch::ComputeEnvironment'
    DependsOn: LaunchTemplate
    Properties:
      ComputeEnvironmentName: !Join ['','-EC2']]
      ComputeResources:
        AllocationStrategy: SPOT_CAPACITY_OPTIMIZED
        DesiredvCpus: 0
        Ec2KeyPair: !Join ['-',[MyComputeEnv,!Ref EnvironmentLow]]
        InstanceRole: !GetAtt BatchInstanceProfile.Arn
        InstanceTypes:
          - c4.large
          - c5.large
        LaunchTemplate: 
          LaunchTemplateId: !Ref LaunchTemplate
          Version: $Latest
        MaxvCpus: 256
        MinvCpus: 0
        DesiredvCpus: 0
        SecurityGroupIds:
          - !Ref DefaultSecurityGroup
        Subnets: !Split [',',!Join [',[!Ref SubnetA,!Ref SubnetB,!Ref SubnetC ]]]
        Type: SPOT
        SpotIamFleetRole: SpotFleetTagServiceRole
      ServiceRole: !Ref BatchServiceRole
      State: ENABLED
      Type: Managed

最后，我使用了 AWS::Batch::JobDefinition 的 ContainerProperties/[Volumes|MountPoints] 来添加我的 EFS 卷。

  ContentJob0:
    Type: 'AWS::Batch::JobDefinition'
    Properties:
      Type: Container
      ContainerProperties:
        Command: 
          - ls
        Environment:
          - Name: AAA
            Value: AAA
        ExecutionRoleArn: !GetAtt BatchJobRole.Arn
        Image: !Join ['',[!Ref 'AWS::AccountId','.dkr.ecr.',!Ref 'AWS::Region','.amazonaws.com/',!Ref Image ]]
        LogConfiguration:
          LogDriver: awslogs
        JobRoleArn: !Ref BatchContainerRole
        Memory: 2048
        Vcpus: 2
        Volumes:
          - Name: MISC
            Host:
              SourcePath: '/mnt/efs-0'
          - Name: JOBS
            Host:
              SourcePath: '/mnt/efs-1'
        MountPoints:
          - SourceVolume: MISC
            ContainerPath: '/mnt/efs-0'
          - SourceVolume: JOBS
            ContainerPath: '/mnt/efs-1'
      JobDefinitionName: !Join ['-',['MyJob',!Ref ServiceName,'EC2','0']]
      PlatformCapabilities:
        - EC2
      PropagateTags: true
      RetryStrategy: 
        Attempts: 3
      Timeout: 
        AttemptDurationSeconds: 300

由于我需要在 LaunchTemplate 中设置 SPOT 实例的价格和其他东西，这对我来说是一个很好的解决方案。

        InstanceMarketOptions:
          MarketType: spot
          SpotOptions:
            MaxPrice: 0.096

,

AWS 批处理 added EFS Volume support on April 1,2021。

用户文档位于 https://docs.aws.amazon.com/batch/latest/userguide/efs-volumes.html

博客文章位于 https://aws.amazon.com/blogs/hpc/introducing-support-for-per-job-amazon-efs-volumes-in-aws-batch/