问题描述
我正在尝试将数据从Amazon Kinesis Data Firehose发送到Amazon Elasticsearch Service，但是它记录了一条错误消息body {
margin: 0;
font-family: 'Open Sans',sans-serif;
}
/* added display flex,specified row and space-between */
.header {
overflow: hidden;
background-color: #262e28;
padding: 10px 10px;
Box-shadow: 0px 2px 8px #888888;
display: flex;
flex-direction: row;
justify-content: space-between;
}
.header a {
color: white;
text-align: center;
padding: 12px;
text-decoration: none;
font-size: 14px;
line-height: 25px;
border-radius: 4px;
}
.header a.logo {
font-size: 25px;
color: white;
display: flex;
justify-content: center;
align-items: flex-end;
padding-top: 0;
}
.header a:hover {
color: #aeaeae;
}
.header a.active {
color: #aeaeae;
}
/* added display flex properties */
.header-right {
display: flex;
justify-content: flex-end;
align-items: center;
}
/* added display flex properties */
.header-left {
display: flex;
justify-content: flex-start;
align-items: center;
}
@media screen and (max-width: 500px) {
.header a {
float: none;
display: block;
text-align: left;
}
.header-right {
float: none;
}
}
。但是，我可以到达Elasticsearch端点（503 Service Unavailable）并对其进行查询。我还经历了How can I prevent HTTP 503 Service Unavailable errors in Amazon Elasticsearch Service?，可以确认我的设置有足够的资源。
这是我在保存失败日志的S3备份存储区中遇到的错误：
https://vpc-XXX.<region>.es.amazonaws.com任何人都知道如何解决此问题并将数据索引到Elasticsearch中吗？
解决方法
结果是，我的问题是选择了错误的安全组。
我正在使用与附加到Elasticsearch实例相同的安全组（我命名为elasticsearch-${domain_name}）（该安全组允许TCP从firehose_es安全组进/出端口443）。我应该选择firehose_es安全组。
根据评论中的要求，这是firehose_es SG的Terraform配置。
resource "aws_security_group" "firehose_es" {
name = "firehose_es"
description = "Firehose to send logs to Elasticsearch"
vpc_id = module.networking.aws_vpc_id
}
resource "aws_security_group_rule" "firehose_es_https_ingress" {
type = "ingress"
from_port = 443
to_port = 443
protocol = "tcp"
security_group_id = aws_security_group.firehose_es.id
cidr_blocks = ["10.0.0.0/8"]
}
resource "aws_security_group_rule" "firehose_es_https_egress" {
type = "egress"
from_port = 443
to_port = 443
protocol = "tcp"
security_group_id = aws_security_group.firehose_es.id
source_security_group_id = aws_security_group.elasticsearch.id
}
在问这个问题（也许是为什么有些人问这个问题）之前，我要解决的另一件事是使用正确的角色并将正确的策略附加到该角色。这是我的角色（作为Terraform配置）
// https://docs.aws.amazon.com/firehose/latest/dev/controlling-access.html
data "aws_iam_policy_document" "firehose_es_policy_specific" {
statement {
actions = [
"s3:AbortMultipartUpload","s3:GetBucketLocation","s3:GetObject","s3:ListBucket","s3:ListBucketMultipartUploads","s3:PutObject"
]
resources = [
aws_s3_bucket.firehose.arn,"${aws_s3_bucket.firehose.arn}/*"
]
}
statement {
actions = [
"es:DescribeElasticsearchDomain","es:DescribeElasticsearchDomains","es:DescribeElasticsearchDomainConfig","es:ESHttpPost","es:ESHttpPut"
]
resources = [
var.elasticsearch_domain_arn,"${var.elasticsearch_domain_arn}/*",]
}
statement {
actions = [
"es:ESHttpGet"
]
resources = [
"${var.elasticsearch_domain_arn}/_all/_settings","${var.elasticsearch_domain_arn}/_cluster/stats","${var.elasticsearch_domain_arn}/${var.name_prefix}${var.name}_${var.app}*/_mapping/type-name","${var.elasticsearch_domain_arn}/_nodes","${var.elasticsearch_domain_arn}/_nodes/stats","${var.elasticsearch_domain_arn}/_nodes/*/stats","${var.elasticsearch_domain_arn}/_stats","${var.elasticsearch_domain_arn}/${var.name_prefix}${var.name}_${var.app}*/_stats"
]
}
statement {
actions = [
"ec2:DescribeVpcs","ec2:DescribeVpcAttribute","ec2:DescribeSubnets","ec2:DescribeSecurityGroups","ec2:DescribeNetworkInterfaces","ec2:CreateNetworkInterface","ec2:CreateNetworkInterfacePermission","ec2:DeleteNetworkInterface",]
resources = [
"*"
]
}
}
resource "aws_kinesis_firehose_delivery_stream" "ecs" {
name = "${var.name_prefix}${var.name}_${var.app}"
destination = "elasticsearch"
s3_configuration {
role_arn = aws_iam_role.firehose_es.arn
bucket_arn = aws_s3_bucket.firehose.arn
buffer_interval = 60
compression_format = "GZIP"
}
elasticsearch_configuration {
domain_arn = var.elasticsearch_domain_arn
role_arn = aws_iam_role.firehose_es.arn
# If Firehose cannot deliver to Elasticsearch,logs are sent to S3
s3_backup_mode = "FailedDocumentsOnly"
buffering_interval = 60
buffering_size = 5
index_name = "${var.name_prefix}${var.name}_${var.app}"
index_rotation_period = "OneMonth"
vpc_config {
subnet_ids = var.elasticsearch_subnet_ids
security_group_ids = [var.firehose_security_group_id]
role_arn = aws_iam_role.firehose_es.arn
}
}
}
再次阅读 Controlling Access with Amazon Kinesis Data Firehose 文章后，我能够弄清自己的错误。