ISTIO无法通过目的地规则向后端服务发起HTTPS请求

编程问答 2022-08-15

问题描述

我刚刚开始在我的EKS集群上玩Istio。在当前设置下，我 AWS ELB 位于最前面，然后它将通过 K8s-ingress 将请求路由到istio-ingressgateway。

我正在尝试做这样的事情：

AWS ALB ==(HTTP 80)==> istio-ingressgateway ==(HTTPS-5601)==> kibana-backend-service

我的Kibana当前在HTTPS端口5601上公开（已启用xpack安全）

这是我尝试过的：

  1. 我将AWS Ingress设置为指向istio-ingressgateway:80 
apiVersion: extensions/v1beta1
kind: Ingress
Metadata:
  namespace: istio-system
  annotations:
    alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect","RedirectConfig":{
      "Protocol": "HTTPS","Port": "443","StatusCode": "HTTP_301"}}'
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:eu-west-1:12345678910:certificate/26229a3b-26a9-4dce-ba02-95274978cac9
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80},{"HTTPS": 443}]'
    alb.ingress.kubernetes.io/scheme: internet-facing
    #alb.ingress.kubernetes.io/backend-protocol: HTTPS
    kubernetes.io/ingress.class: alb
  labels:
    app.kubernetes.io/instance: ingress-sandBoxa
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress
    helm.sh/chart: ingress-0.1.0
  name: istio-ingress-sandBoxa
spec:
  rules:
  - host: errbit.eu-west-1.sandBox.test.com
    http:
      paths:
      - backend:
          serviceName: ssl-redirect
          servicePort: use-annotation
        path: /*
      - backend:
          serviceName: istio-ingressgateway
          servicePort: 80
        path: /*
  - host: grafana.eu-west-1.sandBox.test.com
    http:
      paths:
      - backend:
          serviceName: ssl-redirect
          servicePort: use-annotation
        path: /*
      - backend:
          serviceName: istio-ingressgateway
          servicePort: 80
        path: /*
  - host: kibana.eu-west-1.sandBox.test.com
    http:
      paths:
      - backend:
          serviceName: ssl-redirect
          servicePort: use-annotation
        path: /*
      - backend:
          serviceName: istio-ingressgateway
          servicePort: 80
        path: /*
  1. 我设置了一个监听HTTP:80的网关
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
Metadata:
  namespace: istio-system
  name: kibana
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - kibana.eu-west-1.sandBox.test.com
  1. 然后，我创建一个指向该网关的VirtualService并设置DestinationRule，以通过tls:SIMPLE向后端服务容器发起HTTPS 调用。
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
Metadata:
  namespace: istio-system
  name: kibana
spec:
  hosts:
  - kibana.eu-west-1.sandBox.test.com
  gateways:
  - istio-system/kibana
  http:
  - match:
    - uri:
        prefix: /
    rewrite:
       uri: /
    route:
    - destination:
        host: monpack-sandBoxa-kibana.monitoring.svc.cluster.local
        subset: v1
        port:
          number: 5601

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
Metadata:
  namespace: istio-system
  name: kibana
spec:
  host: monpack-sandBoxa-kibana.monitoring.svc.cluster.local
  subsets:
  - name: v1
    labels:
      release: monpack-sandBoxa
    trafficPolicy:
      portLevelSettings:
      - port:
          number: 5601
        loadBalancer:
          simple: ROUND_ROBIN
      tls:
        mode: SIMPLE # initiates HTTPS
        privateKey: /etc/istio/ingressgateway-certs/tls.key
        caCertificates: /etc/istio/ingressgateway-certs/tls.crt

但是，这次似乎情况不太好。

当我尝试执行curl -v https://kibana.eu-west-1.sandBox.test.com时，我得到503并显示以下错误：upstream connect error or disconnect/reset before headers 

* Rebuilt URL to: https://kibana.eu-west-1.sandBox.test.com/
*   Trying 54.76.109.48...
* TCP_NODELAY set
* Connected to kibana.eu-west-1.sandBox.test.com (54.76.109.48) port 443 (#0)
* ALPN,offering h2
* ALPN,offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT),TLS handshake,Client hello (1):
* TLSv1.3 (IN),Server hello (2):
* TLSv1.2 (IN),Certificate (11):
* TLSv1.2 (IN),Server key exchange (12):
* TLSv1.2 (IN),Server finished (14):
* TLSv1.2 (OUT),Client key exchange (16):
* TLSv1.2 (OUT),TLS change cipher,Client hello (1):
* TLSv1.2 (OUT),Finished (20):
* TLSv1.2 (IN),Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN,server accepted to use h2
* Server certificate:
*  subject: CN=*.eu-west-1.sandBox.test.com
*  start date: Aug 21 00:00:00 2020 GMT
*  expire date: Sep 20 12:00:00 2021 GMT
*  subjectAltName: host "kibana.eu-west-1.sandBox.test.com" matched cert's "*.eu-west-1.sandBox.test.com"
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
* Using HTTP2,server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fffcdebc580)
> GET / HTTP/2
> Host: kibana.eu-west-1.sandBox.test.com
> User-Agent: curl/7.58.0
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 503
< date: Mon,24 Aug 2020 12:47:27 GMT
< content-type: text/plain
< content-length: 95
< server: istio-envoy
< x-envoy-upstream-service-time: 62
<
* Connection #0 to host kibana.eu-west-1.sandBox.test.com left intact
upstream connect error or disconnect/reset before headers. reset reason: connection termination

当我查看 istio-ingressgateway的窗格中的日志时，我发现了这一点：

"GET / HTTP/1.1" 503 URX "-" "-" 0 95 62 62 "59.102.86.159,10.199.64.125" "curl/7.58.0" "16085fc0-823b-95cb-a571-0d43e1cf117f" "kibana.eu-west-1.sandBox.test.com" "10.199.64.26:5601" outbound|5601|v1|monpack-sandBoxa-kibana.monitoring.svc.cluster.local 10.199.64.175:44888 10.199.64.175:8080 10.199.64.125:24894 - -

然后在 Kibana的窗格中的istio-proxy容器上，我找到了以下日志：

"GET / HTTP/1.1" 503 UC "-" "-" 0 95 4 - "59.102.86.159,10.199.64.125" "curl/7.58.0" "16085fc0-823b-95cb-a571-0d43e1cf117f" "kibana.eu-west-1.sandBox.test.com" "127.0.0.1:5601" inbound|5601|http|monpack-sandBoxa-kibana.monitoring.svc.cluster.local 127.0.0.1:38532 10.199.64.26:5601 10.199.64.125:0 outbound_.5601_.v1_.monpack-sandBoxa-kibana.monitoring.svc.cluster.local default

此外，在Kibana的容器内的 kibana 容器上，我找到了此日志：

{"type":"error","@timestamp":"2020-08-24T12:47:27Z","tags":["connection","client","error"],"pid":6,"level":"error","error":{"message":"140484513027968:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n","name":"Error","stack":"Error: 140484513027968:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n"},"message":"140484513027968:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n"}

如果我错了，请纠正我，但查看上面的日志，看来请求已按预期通过：

AWS ALB => istio-ingress-gateway => pod's istio(envoy) proxy => pod

尽管最终它会被后端容器拒绝（如SSL3错误日志所示）。

简而言之，我试图做的是让 AWS-ALB 捕获外部https 流量，然后在该流量中终止那里的SSL，然后通过* HTTP将其转发到istio-ingressgateway，然后让istio以 HTTPS 的形式将此流量转发到后端服务。

因此，我想知道我是否错过了这里的任何步骤，或者还有其他更好的方法来实现它？ ：）

其他说明：

  • 我使用的是Istio 1.6.8，通过下面的istioctl安装： 
    istioctl install \
    --set profile=demo \
    --set values.grafana.enabled=false \
    --set values.prometheus.enabled=false \
    --set values.gateways.istio-ingressgateway.type=NodePort
  • 我还没有触摸过mTLS设置
  • 我已经填充了istio-ingressgateway-certs K8的秘密

非常感谢

解决方法

暂无找到可以解决该程序问题的有效方法，小编努力寻找整理中！

如果你已经找到好的解决方法，欢迎将解决方案带上本链接一起发送给小编。

小编邮箱:dio#foxmail.com (将#修改为@）

aws-albhttpshttpsistiokibanassl