Istio 和 Hashicorpt Vault 代理 Sidecar 无法正常工作

编程问答 2022-05-30

问题描述

我使用的是本地 k8s v1.19 和带有 1.8.0 的 Istio.. 当我将 istio 网格注入到我们运行微服务的 hub-dev 时，我无法正确运行它们。保险柜正在运行 dev 命名空间。

我遇到的第一个问题是 Vault 和 Istio sidecar 以某种方式无法正常运行，应用程序无法如下初始化。我尝试使用以下注释来初始化第一个保险库，但没有解决以下问题。

  • vault.hashicorp.com/agent-init-first：真实
  • vault.hashicorp.com/agent-inject：真实

这里是 pod 状态和描述的输出

$ kubectl get pods -n hub-dev
    NAME                                     READY   STATUS     RESTARTS   AGE
    oneapihub-mp-dev-59f7685455-5kmft        0/3     Init:0/2   0          19
    
$ kubectl describe pod oneapihub-mp-dev-59f7685455-5kmft -n hub-dev

Init Containers:
  vault-agent-init:
    Container ID:  
    State:          Running
      Started:      Fri,15 Jan 2021 13:54:30 +0300
    Ready:          False
  istio-validation:
    Container ID:
    Image:         reg-dhc.app.corpintra.net/i3-mirror/docker.io_istio_proxyv2:1.8.0
    State:          Waiting
     Reason:       PodInitializing
    Ready:          False
Containers:
      oneapihub-mp:
        Container ID:
        State:          Waiting
          Reason:       PodInitializing
        Ready:          False
      istio-proxy:
        Container ID:
        State:          Waiting
          Reason:       PodInitializing
        Ready:          False
  istio-proxy:
    Container ID:
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False

    normal  Pulled     16m   kubelet,xx-kube-node07  Container image "docker.io_vault:1.5.2" already present on machine
    normal  Created    16m   kubelet,xx-kube-node07  Created container vault-agent-init
    normal  Started    16m   kubelet,xx-kube-node07  Started container vault-agent-init

当我尝试下面的注解时，它解决了上述问题，但是这次当 pod 开始运行时，它无法找到 /vault/secrets 路径，但之后以某种方式在我检查代理日志时可以读取它应用程序和 /vault/secrets 文件夹存在于 pod 中。

 - vault.hashicorp.com/agent-pre-populate: "false"

这里是应用程序的日志，即使文件夹存在

$ kubectl get pods -n hub-dev
oneapihub-mp-dev-78449b8cf6-qbqhn        3/3     Running   0          9m31s

$ kubectl logs -f oneapihub-mp-dev-78449b8cf6-qbqhn -n hub-dev -c oneapihub-mp

> market-place@1.0.0 start:docker /usr/src/app
> node app.js

{"message""devMessage":"SECRET_READ_ERROR","data":"","exception":"ENOENT: no such file or directory,open '/vault/secrets/database'","stack":"Error: ENOENT: no such file or directory,open '/vault/secrets/database'->

/ $ cd /vault/secrets
/vault/secrets $ ls
database  jenkins
/vault/secrets $

这里我有一些 PUT 错误，可能与 Vault 本身有关，但我很困惑 Vault 如何注入机密。

 $ kubectl logs -f oneapihub-mp-dev-78449b8cf6-qbqhn -n hub-dev -c vault-agent

2021-01-15T11:21:13.477Z [ERROR] auth.handler: error authenticating: error="Put "http://vault.dev.svc:8200/v1/auth/kubernetes/login": dial tcp 10.254.30.115:8200: connect: connection refused" backoff=2.464775515
==> Vault agent started! Log data will stream in below:

==> Vault agent configuration:

                     Cgo: disabled
               Log Level: info
                 Version: Vault v1.5.2
             Version Sha: 685fdfa60d607bca069c09d2d52b6958a7a2febd

2021-01-15T11:21:15.942Z [INFO]  auth.handler: authenticating
2021-01-15T11:21:15.966Z [INFO]  auth.handler: authentication successful,sending token to sinks
2021-01-15T11:21:15.966Z [INFO]  sink.file: token written: path=/home/vault/.vault-token

最后，当我检查 istio-proxy 日志时，我可以看到 GET 或 PUT 请求返回 200。

$ kubectl logs -f oneapihub-mp-dev-78449b8cf6-h8s8j -n hub-dev -c istio-proxy

021-01-15T11:35:04.352221Z  warning envoy filter    mTLS PERMISSIVE mode is used,connection can be either plaintext or TLS,and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2021-01-15T11:35:05.557Z] "PUT /v1/auth/kubernetes/login HTTP/1.1" 200 - "-" 1294 717 8 8 "-" "Go-http-client/1.1" "a082698b-d1f7-4aa5-9db5-01d86d5093ef" "vault.dev.svc:8200" "10.6.24.55:8200" outbound|8200||vault.dev.svc.cluster.local 10.6.19.226:55974 10.254.30.115:8200 10.6.19.226:60478 - default
2021-01-15T11:35:05.724833Z info    Envoy proxy is ready
[2021-010.6.19.226:41888 - default
[2021-01-15T11:35:05.596Z] "GET /v1/secret/data/oneapihub-marketplace/database HTTP/1.1" 200 - "-" 0 400 0 0 "-" "Go-http-client/1.1" "d7d10c1f-c445-44d1-b0e3-bb9ae7bbc2f0" "vault.dev.svc:8200" "10.6.24.55:8200" outbound|8200||vault.dev.svc.cluster.local 10.6.19.226:55974 10.254.30.115:8200 10.6.19.226:41900 - default
[2021-01-15T11:35:05.591Z] "PUT /v1/auth/token/renew-self HTTP/1.1" 200 - "-" 15 717 8 8 "-" "Go-http-client/1.1" "56705e5c-c966-4bc8-8187-7ca5bb2b4abe" "vault.dev.svc:8200" "10.6.24.55:8200" outbound|8200||vault.dev.svc.cluster.local 10.6.19.226:37388 10.254.30.115:8200 10.6.19.226:41890 - default
[2021-01-15T11:35:05.602Z] "GET /v1/secret/data/oneapihub-marketplace/jenkins HTTP/1.1" 200 - "-" 0 284 0 0 "-" "Go-http-client/1.1" "1b6d8601-18df-4f32-8722-162aa785c476" "vault.dev.svc:8200" "10.6.24.55:8200" outbound|8200||vault.dev.svc.cluster.local 10.6.19.226:55974 10.254.30.115:8200 10.6.19.226:41902 - default

解决方法

在下面添加了对我有用的注释。

  template:
    metadata:
      annotations:
        traffic.sidecar.istio.io/excludeOutboundPorts: "8200"
        vault.hashicorp.com/agent-init-first: "true"
        vault.hashicorp.com/agent-inject: "true"
,

下面的注释对我来说也很好用。如果我们使用“traffic.sidecar.istio.io/excludeOutboundPorts”，这意味着从保管库读取机密时，流量不会通过 Istio sidecar，并且很可能未加密。

template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-init-first: "true"
        vault.hashicorp.com/agent-inject: "true"
hashicorp-vaultistioistio-sidecarkubernetes