问题描述
我使用aws提供的s3-rds模板创建了数据管道。我已经为此工作了几天，但一直无法正常工作。
我的数据管道位于eu-west-1，而我要将数据导入到的rds实例位于eu-west-2。因此，我已在eu-west-2中启动了EC2资源，并为其分配了打开端口3306的“内部连接”安全组，RDS服务器也正在使用该安全组。
运行管道时，出现以下错误-Unable to establish connection to jdbc:MysqL://my-rds-instance.cv3sh7p5gd9.eu-west-2.rds.amazonaws.com:3306/null Could not create connection to database server.
我决定使用密钥对启动EC2实例，以便可以SSH到服务器，然后使用netcat来查看它是否可以连接到MysqL数据库，并且运行良好。
这里是数据管道定义，以防万一。
{
"objects": [
{
"output": {
"ref": "DestinationRDSTable"
},"input": {
"ref": "S3InputDataLocation"
},"dependsOn": {
"ref": "RdsMysqLTableCreateActivity"
},"name": "DataLoadActivity","id": "DataLoadActivity","runsOn": {
"ref": "Ec2Instance"
},"type": "copyActivity"
},{
"*password": "#{*myRDSPassword}","name": "rds_MysqL","jdbcProperties": "allowMultiQueries=true","id": "rds_MysqL","region": "eu-west-2","type": "RdsDatabase","rdsInstanceId": "#{myRDSInstanceId}","username": "#{myRDSUsername}"
},{
"failureAndRerunMode": "CASCADE","resourceRole": "DataPipelineDefaultResourceRole","role": "DataPipelineDefaultRole","pipelineLogUri": "s3://bucket/logs/","scheduleType": "ondemand","name": "Default","id": "Default"
},{
"instanceType": "t2.small","name": "Ec2Instance","actionOnTaskFailure": "terminate","securityGroups": "#{myEc2RdsSecurityGrps}","keyPair": "Rod-Dell-London","id": "Ec2Instance","type": "Ec2Resource","terminateAfter": "2 Hours"
},{
"database": {
"ref": "rds_MysqL"
},"name": "RdsMysqLTableCreateActivity","id": "RdsMysqLTableCreateActivity","type": "sqlActivity","script": "''"
},{
"name": "DataFormat1","id": "DataFormat1","type": "CSV"
},"name": "DestinationRDSTable","insertQuery": "#{myRDSTableInsertsql}","id": "DestinationRDSTable","type": "sqlDatanode","table": "#{myRDSTableName}","selectQuery": "select * from #{table}"
},{
"directoryPath": "#{myInputS3Loc}","dataFormat": {
"ref": "DataFormat1"
},"name": "S3InputDataLocation","id": "S3InputDataLocation","type": "S3Datanode"
}
],"parameters": [
{
"description": "RDS MysqL password","id": "*myRDSPassword","type": "String"
},{
"watermark": "security group name","helpText": "The names of one or more EC2 security groups that have access to the RDS MysqL cluster.","description": "RDS MysqL security group(s)","isArray": "true","optional": "true","id": "myEc2RdsSecurityGrps",{
"description": "RDS MysqL username","id": "myRDSUsername",{
"description": "Input S3 file path","id": "myInputS3Loc","type": "AWS::S3::ObjectKey"
},{
"helpText": "The sql statement to insert data into the RDS MysqL table.","watermark": "INSERT INTO #{table} (col1,col2,col3) VALUES(?,?,?) ;","description": "Insert sql query","id": "myRDSTableInsertsql",{
"helpText": "The name of an existing table or a new table that will be created based on the create table sql query parameter below.","description": "RDS MysqL table name","id": "myRDSTableName",{
"watermark": "CREATE TABLE pet IF NOT EXISTS (name VARCHAR(20),owner VARCHAR(20),species VARCHAR(20),gender CHAR(1),birth DATE,death DATE);","helpText": "The idempotent sql statement to create the RDS MysqL table if it does not already exist.","description": "Create table sql query","id": "myRDSCreateTablesql",{
"watermark": "DB Instance","description": "RDS Instance ID","id": "myRDSInstanceId","type": "String"
}
],"values": {
"myRDSInstanceId": "my-rds-instance.cv3sh7p5gd9.eu-west-2","myRDSUsername": "admin","myRDSTableInsertsql": "INSERT INTO probe_request_test (device_uuid,ts,device_vendor_id,drone_id,RSSi,venue_id) VALUES (?,?)","*myRDSPassword": "","myEc2RdsSecurityGrps": [
"Internal connectivity","SSH Inbound","Inbound - Outbound Office ONLY Access"
],"myInputS3Loc": "s3://bucket/2020-10-12/probe_data/519e4475-82f3-486a-8205-5c22f4bf6872-000000","myRDSTableName": "probe_request_test"
}
}
解决方法
暂无找到可以解决该程序问题的有效方法，小编努力寻找整理中！
如果你已经找到好的解决方法，欢迎将解决方案带上本链接一起发送给小编。
小编邮箱:dio#foxmail.com (将#修改为@）