问题描述
我正在为将dynamo db导出到s3创建数据管道。模板 为无服务器yaml指定的代码不适用于“ PAY_PER_REQUEST”结算 模式
ServerlessError: An error occurred: UrlReportDataPipeline - Pipeline DeFinition Failed to validate because of following Errors: [{ObjectId = 'TableBackupActivity',errors = [Object references invalid id: 's3://dynamodb-dpl-#{myDDBRegion}/emr-ddb-storage-handler/4.11.0/emr-dynamodb-tools-4.11.0-SNAPSHOT-jar-with-dependencies.jar,org.apache.hadoop.dynamodb.tools.DynamoDBExport,#{output.directoryPath},#{input.tableName},#{input.readThroughputPercent}']}] and Warnings: [].
有人可以帮我吗?使用控制台创建的管道可以与表备份活动中的step值相同而完美地工作。
管道模板粘贴在下面
UrlReportDataPipeline:
Type: AWS::DataPipeline::Pipeline
Properties:
Name: ***pipeline name****
Activate: true
ParameterObjects:
- Id: "myDDBReadThroughputRatio"
Attributes:
- Key: "description"
StringValue: "DynamoDB read throughput ratio"
- Key: "type"
StringValue: "Double"
- Key: "default"
StringValue: "0.9"
- Id: "myOutputS3Loc"
Attributes:
- Key: "description"
StringValue: "S3 output bucket"
- Key: "type"
StringValue: "AWS::S3::ObjectKey"
- Key: "default"
StringValue:
!Join [ "",[ "s3://",Ref: "UrlReportBucket" ] ]
- Id: "myDDBTableName"
Attributes:
- Key: "description"
StringValue: "DynamoDB Table Name"
- Key: "type"
StringValue: "String"
- Id: "myDDBRegion"
Attributes:
- Key: "description"
StringValue: "DynamoDB region"
ParameterValues:
- Id: "myDDBTableName"
StringValue:
Ref: "UrlReport"
- Id: "myDDBRegion"
StringValue: "eu-west-1"
PipelineObjects:
- Id: "S3BackupLocation"
Name: "copy data to this S3 location"
Fields:
- Key: "type"
StringValue: "S3Datanode"
- Key: "dataFormat"
RefValue: "DDBExportFormat"
- Key: "directoryPath"
StringValue: "#{myOutputS3Loc}/#{format(@scheduledStartTime,'YYYY-MM-dd-HH-mm-ss')}"
- Id: "DDBSourceTable"
Name: "DDBSourceTable"
Fields:
- Key: "tableName"
StringValue: "#{myDDBTableName}"
- Key: "type"
StringValue: "DynamoDBDatanode"
- Key: "dataFormat"
RefValue: "DDBExportFormat"
- Key: "readThroughputPercent"
StringValue: "#{myDDBReadThroughputRatio}"
- Id: "DDBExportFormat"
Name: "DDBExportFormat"
Fields:
- Key: "type"
StringValue: "DynamoDBExportDataFormat"
- Id: "TableBackupActivity"
Name: "TableBackupActivity"
Fields:
- Key: "resizeClusterBeforeRunning"
StringValue: "true"
- Key: "type"
StringValue: "EmrActivity"
- Key: "input"
RefValue: "DDBSourceTable"
- Key: "runsOn"
RefValue: "EmrClusterForBackup"
- Key: "output"
RefValue: "S3BackupLocation"
- Key: "step"
RefValue: "s3://dynamodb-dpl-#{myDDBRegion}/emr-ddb-storage-handler/4.11.0/emr-dynamodb-tools-4.11.0-SNAPSHOT-jar-with-dependencies.jar,#{input.readThroughputPercent}"
- Id: "DefaultSchedule"
Name: "Every 1 day"
Fields:
- Key: "occurrences"
StringValue: "1"
- Key: "startDateTime"
StringValue: "2020-09-17T1:00:00"
- Key: "type"
StringValue: "Schedule"
- Key: "period"
StringValue: "1 Day"
- Id: "Default"
Name: "Default"
Fields:
- Key: "type"
StringValue: "Default"
- Key: "scheduleType"
StringValue: "cron"
- Key: "failureAndRerunMode"
StringValue: "CASCADE"
- Key: "role"
StringValue: "DatapipelineDefaultRole"
- Key: "resourceRole"
StringValue: "DatapipelineDefaultResourceRole"
- Key: "schedule"
RefValue: "DefaultSchedule"
- Id: "EmrClusterForBackup"
Name: "EmrClusterForBackup"
Fields:
- Key: "terminateAfter"
StringValue: "2 Hours"
- Key: "masterInstanceType"
StringValue: "m3.xlarge"
- Key: "coreInstanceType"
StringValue: "m3.xlarge"
- Key: "coreInstanceCount"
StringValue: "1"
- Key: "type"
StringValue: "EmrCluster"
- Key: "releaseLabel"
StringValue: "emr-5.23.0"
- Key: "region"
StringValue: "#{myDDBRegion}"
解决方法
Step具有一个指向多个资源的refValue,并且看起来也将它们指定为字符串。根据无服务器文档,refValue是
您指定为同一管道定义中另一个对象的标识符的字段值。
如果您在哪里使用S3BackupLocation,它将在PipelineObjects下创建,然后使用其ID进行引用。
对于步骤,您将refValue使用字符串作为其值,然后该字符串带有逗号,因此看起来好像在指定多个对象。
我不确定这意味着什么,但是如果要使用refValue,请在模板的其他位置创建它并在此处使用其ID?
也可以尝试在此处使用字符串值而不是引用值
,伙计们,我与AWS支持团队解决了。截至今天,以下是 yaml代码,可按需按需创建数据数据 动态表
如果需要,还可以将其转换为json
UrlReportBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: ***bucketname***
UrlReportDataPipeline:
Type: AWS::DataPipeline::Pipeline
Properties:
Name: ***pipelinename***
Activate: true
ParameterObjects:
- Id: "myDDBReadThroughputRatio"
Attributes:
- Key: "description"
StringValue: "DynamoDB read throughput ratio"
- Key: "type"
StringValue: "Double"
- Key: "default"
StringValue: "0.9"
- Id: "myOutputS3Loc"
Attributes:
- Key: "description"
StringValue: "S3 output bucket"
- Key: "type"
StringValue: "AWS::S3::ObjectKey"
- Key: "default"
StringValue:
!Join [ "",[ "s3://",Ref: "UrlReportBucket" ] ]
- Id: "myDDBTableName"
Attributes:
- Key: "description"
StringValue: "DynamoDB Table Name"
- Key: "type"
StringValue: "String"
- Id: "myDDBRegion"
Attributes:
- Key: "description"
StringValue: "DynamoDB region"
ParameterValues:
- Id: "myDDBTableName"
StringValue:
Ref: "UrlReport"
- Id: "myDDBRegion"
StringValue: "eu-west-1"
PipelineObjects:
- Id: "S3BackupLocation"
Name: "Copy data to this S3 location"
Fields:
- Key: "type"
StringValue: "S3DataNode"
- Key: "dataFormat"
RefValue: "DDBExportFormat"
- Key: "directoryPath"
StringValue: "#{myOutputS3Loc}/#{format(@scheduledStartTime,'YYYY-MM-dd-HH-mm-ss')}"
- Id: "DDBSourceTable"
Name: "DDBSourceTable"
Fields:
- Key: "tableName"
StringValue: "#{myDDBTableName}"
- Key: "type"
StringValue: "DynamoDBDataNode"
- Key: "dataFormat"
RefValue: "DDBExportFormat"
- Key: "readThroughputPercent"
StringValue: "#{myDDBReadThroughputRatio}"
- Id: "DDBExportFormat"
Name: "DDBExportFormat"
Fields:
- Key: "type"
StringValue: "DynamoDBExportDataFormat"
- Id: "TableBackupActivity"
Name: "TableBackupActivity"
Fields:
- Key: "resizeClusterBeforeRunning"
StringValue: "true"
- Key: "type"
StringValue: "EmrActivity"
- Key: "input"
RefValue: "DDBSourceTable"
- Key: "runsOn"
RefValue: "EmrClusterForBackup"
- Key: "output"
RefValue: "S3BackupLocation"
- Key: "step"
StringValue: "s3://dynamodb-dpl-#{myDDBRegion}/emr-ddb-storage-handler/4.11.0/emr-dynamodb-tools-4.11.0-SNAPSHOT-jar-with-dependencies.jar,org.apache.hadoop.dynamodb.tools.DynamoDBExport,#{output.directoryPath},#{myDDBTableName},#{myDDBReadThroughputRatio}"
- Id: "DefaultSchedule"
Name: "Every 1 day"
Fields:
- Key: "occurrences"
StringValue: "1"
- Key: "startDateTime"
StringValue: "2020-09-23T1:00:00"
- Key: "type"
StringValue: "Schedule"
- Key: "period"
StringValue: "1 Day"
- Id: "Default"
Name: "Default"
Fields:
- Key: "type"
StringValue: "Default"
- Key: "scheduleType"
StringValue: "cron"
- Key: "failureAndRerunMode"
StringValue: "CASCADE"
- Key: "role"
StringValue: "DatapipelineDefaultRole"
- Key: "resourceRole"
StringValue: "DatapipelineDefaultResourceRole"
- Key: "schedule"
RefValue: "DefaultSchedule"
- Id: "EmrClusterForBackup"
Name: "EmrClusterForBackup"
Fields:
- Key: "terminateAfter"
StringValue: "2 Hours"
- Key: "masterInstanceType"
StringValue: "m3.xlarge"
- Key: "coreInstanceType"
StringValue: "m3.xlarge"
- Key: "coreInstanceCount"
StringValue: "1"
- Key: "type"
StringValue: "EmrCluster"
- Key: "releaseLabel"
StringValue: "emr-5.23.0"
- Key: "region"
StringValue: "#{myDDBRegion}"