现在k8s上服务暴露方式用的最多就是nginx-ingress，今天我们来讲讲nginx-ingress的具体使用，我们在tke上实践下，如何部署使用nginx-ingress，以及nginx的一些注解功能的使用。

今天我们主要是在腾讯云上的tke上进行实践测试，不过测试的功能也是支持在其他类型的k8s上使用，只要部署了nignx-ingress就行。现在tke上为了方便用户能够快速使用nginx-ingress推了了通过组件的方式的来部署，具体部署方式可以参考文档https://cloud.tencent.com/document/product/457/50503，如果在集群上找不到这个组件，可以提交工单进行开通，如果想自己搭建nginx-ingress可以参考https://cloud.tencent.com/document/product/457/47293进行部署。今天我们不说具体的部署，主要讲讲nginx-ingress一些常用的功能如何进行配置使用。

1. https的配置

有的时候我们需要给域名配置ssl证书来进行https的安全访问，首先你需要给域名申请证书，可以到现在的公有云上为域名申请免费的证书，申请后将证书通过secret挂载到k8s集群中

kubectl create secret tls example-ssl --key 2_example.tke.niewx.cn.key --cert 1_example.tke.niewx.cn_bundle.crt

然后再在ingress中配置下，这里我们配置下tls这个字段

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.cloud.tencent.com/direct-access: "false"
    kubernetes.io/ingress.class: nwx-ingress
    kubernetes.io/ingress.extensiveParameters: '{"AddressIPVersion":"IPV4"}'
    kubernetes.io/ingress.http-rules: "null"
    kubernetes.io/ingress.https-rules: '[{"host":"example.tke.niewx.cn","path":"/","backend":{"serviceName":"springboot","servicePort":"8080"}}]'
    kubernetes.io/ingress.rule-mix: "true"
    nginx.ingress.kubernetes.io/use-regex: "true"
  name: example-ingress
  namespace: test
spec:
  rules:
  - host: example.tke.niewx.cn
    http:
      paths:
      - backend:
          serviceName: springboot
          servicePort: 8080
        path: /
  tls:
  - hosts:
    - example.tke.niewx.cn
    secretName: example-ssl
status:
  loadBalancer:
    ingress:
    - ip: 81.71.131.235

这样浏览器输入example.tke.niewx.cn会通过https访问。

2. 域名登录认证

有时候我们的服务没有提供登录认证，但是有不希望将服务提供给所有的人都能访问，那么可以通过ingress上的认证控制访问。今天我们讲讲常用的2种认证方式。

2.1 基本身份认证

[root@VM-0-13-centos nginx-ingress]# htpasswd -c auth admin
New password: 
Re-type new password: 
Adding password for user admin
[root@VM-0-13-centos nginx-ingress]# kubectl create secret generic basic-auth --from-file=auth -n test
secret/basic-auth created

接下来我们在ingress中配置下这个secret，这里我们配置账号密码是admin/admin

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.cloud.tencent.com/direct-access: "false"
    kubernetes.io/ingress.class: nwx-ingress
    kubernetes.io/ingress.extensiveParameters: '{"AddressIPVersion":"IPV4"}'
    kubernetes.io/ingress.http-rules: "null"
    kubernetes.io/ingress.https-rules: '[{"host":"example.tke.niewx.cn","path":"/","backend":{"serviceName":"springboot","servicePort":"8080"}}]'
    kubernetes.io/ingress.rule-mix: "true"
    nginx.ingress.kubernetes.io/auth-realm: Authentication Required - admin
    nginx.ingress.kubernetes.io/auth-secret: basic-auth
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/use-regex: "true"
  name: example-ingress
  namespace: test
spec:
  rules:
  - host: example.tke.niewx.cn
    http:
      paths:
      - backend:
          serviceName: springboot
          servicePort: 8080
        path: /
  tls:
  - hosts:
    - example.tke.niewx.cn
    secretName: example-ssl
status:
  loadBalancer:
    ingress:
    - ip: 81.71.131.235

nginx.ingress.kubernetes.io/auth-realm: Authentication Required - admin
nginx.ingress.kubernetes.io/auth-secret: basic-auth
nginx.ingress.kubernetes.io/auth-type: basic

我们在ingress中加入这3条注解，再登录就会需要输入账号密码了。

2.2 外部身份验证

有时候我们有自己的鉴权中心，也是可以使用外部身份进行认证的，这里我们采用https://httpbin.org/basic-auth/user/passwd这个作为外部身份，这个默认账号和密码user/passwd

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.cloud.tencent.com/direct-access: "false"
    kubernetes.io/ingress.class: nwx-ingress
    kubernetes.io/ingress.extensiveParameters: '{"AddressIPVersion":"IPV4"}'
    kubernetes.io/ingress.http-rules: "null"
    kubernetes.io/ingress.https-rules: '[{"host":"example.tke.niewx.cn","path":"/","backend":{"serviceName":"springboot","servicePort":"8080"}}]'
    kubernetes.io/ingress.rule-mix: "true"
    nginx.ingress.kubernetes.io/auth-url: https://httpbin.org/basic-auth/user/passwd
    nginx.ingress.kubernetes.io/use-regex: "true"

这里我们只需要配置下面这个注解就会用外部身份认证

nginx.ingress.kubernetes.io/auth-url: https://httpbin.org/basic-auth/user/passwd

这里我们通过curl的方式验证下

[root@VM-0-3-centos ~]# curl -k https://example.tke.niewx.cn/ -v -H 'Host: example.tke.niewx.cn'
* About to connect() to example.tke.niewx.cn port 443 (#0)
*   Trying 81.71.131.235...
* Connected to example.tke.niewx.cn (81.71.131.235) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* 	subject: CN=Kubernetes Ingress Controller Fake Certificate,O=Acme Co
* 	start date: Dec 28 06:14:11 2020 GMT
* 	expire date: Dec 28 06:14:11 2021 GMT
* 	common name: Kubernetes Ingress Controller Fake Certificate
* 	issuer: CN=Kubernetes Ingress Controller Fake Certificate,O=Acme Co
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Accept: */*
> Host: example.tke.niewx.cn
> 
< HTTP/1.1 401 Unauthorized
< Date: Wed, 30 Dec 2020 11:36:43 GMT
< Content-Type: text/html
< Content-Length: 172
< Connection: keep-alive
< WWW-Authenticate: Basic realm="Fake Realm"
< Strict-Transport-Security: max-age=15724800; includeSubDomains
< 
<html>
<head><title>401 Authorization Required</title></head>
<body>
<center><h1>401 Authorization Required</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Connection #0 to host example.tke.niewx.cn left intact
[root@VM-0-3-centos ~]# curl -k https://example.tke.niewx.cn/ -v -H 'Host: example.tke.niewx.cn' -u 'user:passwd'
* About to connect() to example.tke.niewx.cn port 443 (#0)
*   Trying 81.71.131.235...
* Connected to example.tke.niewx.cn (81.71.131.235) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* 	subject: CN=Kubernetes Ingress Controller Fake Certificate,O=Acme Co
* 	start date: Dec 28 06:14:11 2020 GMT
* 	expire date: Dec 28 06:14:11 2021 GMT
* 	common name: Kubernetes Ingress Controller Fake Certificate
* 	issuer: CN=Kubernetes Ingress Controller Fake Certificate,O=Acme Co
* Server auth using Basic with user 'user'
> GET / HTTP/1.1
> Authorization: Basic dXNlcjpwYXNzd2Q=
> User-Agent: curl/7.29.0
> Accept: */*
> Host: example.tke.niewx.cn
> 
< HTTP/1.1 200 
< Date: Wed, 30 Dec 2020 11:37:14 GMT
< Content-Type: text/plain;charset=UTF-8
< Content-Length: 23
< Connection: keep-alive
< Strict-Transport-Security: max-age=15724800; includeSubDomains
< 
* Connection #0 to host example.tke.niewx.cn left intact
you access path is root[root@VM-0-3-centos ~]# curl -k https://example.tke.niewx.cn/ -v -H 'Host: example.tke.niewx.cn' -u 'user:user'
* About to connect() to example.tke.niewx.cn port 443 (#0)
*   Trying 81.71.131.235...
* Connected to example.tke.niewx.cn (81.71.131.235) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* 	subject: CN=Kubernetes Ingress Controller Fake Certificate,O=Acme Co
* 	start date: Dec 28 06:10:15 2020 GMT
* 	expire date: Dec 28 06:10:15 2021 GMT
* 	common name: Kubernetes Ingress Controller Fake Certificate
* 	issuer: CN=Kubernetes Ingress Controller Fake Certificate,O=Acme Co
* Server auth using Basic with user 'user'
> GET / HTTP/1.1
> Authorization: Basic dXNlcjp1c2Vy
> User-Agent: curl/7.29.0
> Accept: */*
> Host: example.tke.niewx.cn
> 
< HTTP/1.1 401 Unauthorized
< Date: Wed, 30 Dec 2020 11:37:20 GMT
< Content-Type: text/html
< Content-Length: 172
< Connection: keep-alive
* Authentication problem. Ignoring this.
< WWW-Authenticate: Basic realm="Fake Realm"
< Strict-Transport-Security: max-age=15724800; includeSubDomains
< 
<html>
<head><title>401 Authorization Required</title></head>
<body>
<center><h1>401 Authorization Required</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Connection #0 to host example.tke.niewx.cn left intact

我们分别通过不带用户名密码、正确的用户名密码、和不到用户名密码进行测试，最终只有正确输入用户名和密码才返回200。

3. rewrite

有时候我们需要创建ingress对象来为Controller制定转发规则

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.cloud.tencent.com/direct-access: "false"
    kubernetes.io/ingress.class: ingress
    nginx.ingress.kubernetes.io/use-regex: "true"
  name: ingress-v3
  namespace: test
spec:
  rules:
  - host: example.tke.niewx.cn
    http:
      paths:
      - backend:
          serviceName: springboot
          servicePort: 8080
        path: /api
  tls:
  - hosts:
    - example.tke.niewx.cn
    secretName: example-ssl

表示会将https://example.tke.niewx.cn/api/URL转发到集群中的https://example.tke.niewx.cn/api/URL上，也就是说最后一行的path: /api也会当作请求的一部分，追加到url中。

但是，如果nginx实际的请求地址为https://svc-springboot/api/URL，则会报404，找不到服务。

有的时候我们需要将前端用户访问的路径指向后端真实的路径，这里我们就需要用到ingress的rewite来实现了

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.cloud.tencent.com/direct-access: "false"
    kubernetes.io/ingress.class: ingress
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/use-regex: "true"
  name: example-ingress
  namespace: test
spec:
  rules:
  - host: example.tke.niewx.cn
    http:
      paths:
      - backend:
          serviceName: springboot
          servicePort: 8080
        path: /api(/|$)(.*)
  tls:
  - hosts:
    - example.tke.niewx.cn
    secretName: example-ssl

这里我们会进行如下重定向

example.tke.niewx.cn/api 重写到 example.tke.niewx.cn/
example.tke.niewx.cn/api/ 重写到 example.tke.niewx.cn/
example.tke.niewx.cn/api/v1 重写到 example.tke.niewx.cn/v1

正常我们会打印出是访问的实际路径，由于被重写，导致本应打印的应该是/api/v1，现在打印的是v1，说明真实访问的后端是example.tke.niewx.cn/v1

我们也可以将域名的根目录重写到某一个后端的路径下，这样访问根目录下就会重写到后的真实路径，我们测试下将根路径重写到/api/v2下面

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.cloud.tencent.com/direct-access: "false"
    kubernetes.io/ingress.class: nwx-ingress
    kubernetes.io/ingress.extensiveParameters: '{"AddressIPVersion":"IPV4"}'
    kubernetes.io/ingress.http-rules: "null"
    kubernetes.io/ingress.https-rules: '[{"host":"example.tke.niewx.cn","path":"/","backend":{"serviceName":"springboot","servicePort":"8080"}}]'
    kubernetes.io/ingress.rule-mix: "true"
    nginx.ingress.kubernetes.io/app-root: /api/v2
    nginx.ingress.kubernetes.io/auth-url: https://httpbin.org/basic-auth/user/passwd
    nginx.ingress.kubernetes.io/use-regex: "true"
  name: example-ingress
  namespace: test
spec:
  rules:
  - host: example.tke.niewx.cn
    http:
      paths:
      - backend:
          serviceName: springboot
          servicePort: 8080
        path: /
  tls:
  - hosts:
    - example.tke.niewx.cn
    secretName: example-ssl
status:
  loadBalancer:
    ingress:
    - ip: 81.71.131.235

这里我们测试发现访问根目录会跳转到/api/v2下，浏览器输入也会强制跳转到/api/v2下

[root@VM-0-3-centos ~]# curl -I -k https://example.tke.niewx.cn
HTTP/1.1 302 Moved Temporarily
Date: Wed, 30 Dec 2020 12:24:30 GMT
Content-Type: text/html
Content-Length: 138
Connection: keep-alive
Location: https://example.tke.niewx.cn/api/v2

rewrite功能可以让我们进行丰富的访问路径配置，这样方便了我们可以通过不同的路径去访问不通的后端。

4. 访问白名单

有时候我们需要给域名配置下访问白名单，我只希望部分ip可以访问我的服务，这时候需要用到ingress的whitelist-source-range，我们可以通过这个注解来配置我们希望放通访问的ip。下面我们只放通10.0.0.0/24可以访问

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.cloud.tencent.com/direct-access: "false"
    kubernetes.io/ingress.class: nwx-ingress
    kubernetes.io/ingress.extensiveParameters: '{"AddressIPVersion":"IPV4"}'
    kubernetes.io/ingress.http-rules: "null"
    kubernetes.io/ingress.https-rules: '[{"host":"example.tke.niewx.cn","path":"/","backend":{"serviceName":"springboot","servicePort":"8080"}}]'
    kubernetes.io/ingress.rule-mix: "true"
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/whitelist-source-range: 10.0.0.0/24

我们测试下看看，我们在10.0.0.3和10.0.5.33分别访问下域名看看

[root@VM-5-33-tlinux ~]# curl -I -k https://example.tke.niewx.cn
HTTP/1.1 403 Forbidden
Date: Wed, 30 Dec 2020 12:39:35 GMT
Content-Type: text/html
Content-Length: 146
Connection: keep-alive
Strict-Transport-Security: max-age=15724800; includeSubDomains

[root@VM-5-33-tlinux ~]# ip addr | grep eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet 10.0.5.33/24 brd 10.0.5.255 scope global eth0

[root@VM-0-3-centos ~]# curl -I -k https://example.tke.niewx.cn
HTTP/1.1 200 
Date: Wed, 30 Dec 2020 12:36:05 GMT
Content-Type: text/plain;charset=UTF-8
Content-Length: 23
Connection: keep-alive
Strict-Transport-Security: max-age=15724800; includeSubDomains

[root@VM-0-3-centos ~]# ip addr | grep eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet 10.0.0.3/24 brd 10.0.0.255 scope global eth0

可以发现10.0.0.3这个服务器上可以访问，但是10.0.5.33确无法访问，这是因为10.0.0.3在我们配置的白名单里。

5. 永久重定向

我们可以在ingress配置域名的重定向，可以配置访问到其他链接，同时也可以配置重定向错误码，这里错误码值的范围是 300~308 ，超出这个范围就重置为默认301的值。

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.cloud.tencent.com/direct-access: "false"
    kubernetes.io/ingress.class: nwx-ingress
    kubernetes.io/ingress.extensiveParameters: '{"AddressIPVersion":"IPV4"}'
    kubernetes.io/ingress.http-rules: "null"
    kubernetes.io/ingress.https-rules: '[{"host":"example.tke.niewx.cn","path":"/","backend":{"serviceName":"springboot","servicePort":"8080"}}]'
    kubernetes.io/ingress.rule-mix: "true"
    nginx.ingress.kubernetes.io/permanent-redirect: https://www.baidu.com
    nginx.ingress.kubernetes.io/use-regex: "true"

这里我们在浏览器中输入example.tke.niewx.cn会强制跳转到百度页面

[root@VM-0-3-centos ~]# curl -I  -k  https://example.tke.niewx.cn
HTTP/1.1 301 Moved Permanently
Date: Wed, 30 Dec 2020 12:55:21 GMT
Content-Type: text/html
Content-Length: 162
Connection: keep-alive
Location: https://www.baidu.com
Strict-Transport-Security: max-age=15724800; includeSubDomains

下面我们重定向到baidu，然后让返回码是308

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.cloud.tencent.com/direct-access: "false"
    kubernetes.io/ingress.class: nwx-ingress
    kubernetes.io/ingress.extensiveParameters: '{"AddressIPVersion":"IPV4"}'
    kubernetes.io/ingress.http-rules: "null"
    kubernetes.io/ingress.https-rules: '[{"host":"example.tke.niewx.cn","path":"/","backend":{"serviceName":"springboot","servicePort":"8080"}}]'
    kubernetes.io/ingress.rule-mix: "true"
    nginx.ingress.kubernetes.io/permanent-redirect: https://www.baidu.com
    nginx.ingress.kubernetes.io/permanent-redirect-code: "308"
    nginx.ingress.kubernetes.io/use-regex: "true"

我们访问域名测试下，可以发下最终后端是baidu，并且返回吗是308

[root@VM-0-3-centos ~]# curl -I  -k  https://example.tke.niewx.cn
HTTP/1.1 308 Permanent Redirect
Date: Wed, 30 Dec 2020 12:57:28 GMT
Content-Type: text/html
Content-Length: 164
Connection: keep-alive
Location: https://www.baidu.com
Strict-Transport-Security: max-age=15724800; includeSubDomains

5. 客户端请求body的大小

我通过ingress给我的harbor配置一个域名，但是我上次镜像的时间报错了，报错413 Request Entity Too Large，这个是客户端上传镜像到仓库的请求body太大，需要怎么解决呢？其实ingress中提供了注解用来配置请求体的大小即可，加上注解nginx.ingress.kubernetes.io/proxy-body-size: 1000m，具体的大小限制根据自身需求配置即可。

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.cloud.tencent.com/direct-access: "false"
    kubernetes.io/ingress.class: nwx-ingress
    kubernetes.io/ingress.extensiveParameters: '{"AddressIPVersion":"IPV4"}'
    kubernetes.io/ingress.http-rules: '[{"host":"harbor.tke.niewx.cn","path":"/","backend":{"serviceName":"harbor-1610088954-harbor-portal","servicePort":"80"}}]'
    kubernetes.io/ingress.https-rules: "null"
    kubernetes.io/ingress.rule-mix: "false"
    nginx.ingress.kubernetes.io/proxy-body-size: 1000m
    nginx.ingress.kubernetes.io/use-regex: "true"
  name: harbor-ingress
  namespace: harbor

6. http2-max-requests配置

问题现象：通过公网长连接访问后ingress后, 脚本每次请求1000次后,到1001次， grpc抛出: connections to all backends failing

github相关问题issue：https://github.com/kubernetes/ingress-nginx/issues/3028

解决方案：这里是因为nginx中http2-max-requests这个配置配置太小，建议的解决方法是参考https://trac.nginx.org/nginx/ticket/1250

链接的的建议When we set http2_max_requests 1000000, the possibility became very low.

这里在nginx-ingress需要设置下controller的全局configmap配置，在配置中加下如下配置即可

http2-max-requests: "1000000"

配置好之后，重启下nginx-ingress controller的pod即可。

7. nginx-ingress获取真实客户端ip

很多业务场景需要获取到真实客户端ip，如果是通过nginx-ingress提供访问，那么在后端的pod内怎么样才能获取到真实客户端ip呢？下面我们以nginx服务为例，当我们通过ingress的域名访问nginx服务，怎么样才能获取真实客户端ip。

如果不配置的话，nginx服务是获取不到真实客户端ip的。现在我们做如下配置

修改nginx-ingress controller的入口service配置，将externalTrafficPolicy改成Local

修改nginx-ingress controller的configmap配置，加上如下配置

  compute-full-forwarded-for: "true"
  forwarded-for-header: "X-Forwarded-For"
  use-forwarded-headers: "true"

修改后，nginx-ingress controller会自动加载configmap配置，我们在访问看看

这时候可以发现日志的客户端ip已经是真实的客户端ip了。

8. nginx-ingress代理tcp和udp服务

具体配置参考文档 https://www.niewx.cn/2021/11/09/2021-11-09-Use-Nginx-Ingress-to-expose-TCP-and-udp-services/

9. nginx-ingress同时支持http和https

大家在使用nginx-ingress的时候，经常会给域名配置证书，让域名走https协议访问，但是有的时候我们希望域名既能http访问，也能https访问，当你在ingress配置了证书后，会发现通过http访问域名默认会跳转到https。

这个时候大家会有一个疑惑，我没配置强制跳转，为什么会http强制跳转到https呢？并且nginx-ingress不像云厂商一样支持混合协议的配置，如果我希望域名能同时通过http和https访问到，该怎么配置呢？下面我来给大家讲解下

首先你在ingress配置了证书，没配置强制跳转，但是http直接强制到了https，这是因为Ingress 启用了 TLS，控制器会使用 308永久重定向响应将HTTP客户端重定向到HTTPS端口 443，Ingress 里配置了 https 证书的话，默认就一定会走 https。

那么我们要如何关闭这个强制跳转呢？其实ingress提供一个注解来进行控制

nginx.ingress.kubernetes.io/ssl-redirect: “false”

如果你希望不进行强制跳转，那你在ingress加上这个注解即可。

从测试可以发现，加上注解后，http和https都可以进行访问了。

10. nginx-ingress实现金丝雀发布

参考tke官网文档配置即可 https://cloud.tencent.com/document/product/457/48907

11. nginx-ingress实现grpc转发

参考下面文档配置

https://cloud.tencent.com/developer/article/1970910

12. nginx-ingress如何开启tls 1.0和1.1

官网文档说明：https://kubernetes.github.io/ingress-nginx/user-guide/tls/

修改nginx ingress controller的配置文件，加上如下配置即可，tke控制台可以直接组件里面进行修改

ssl-ciphers: "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA"
ssl-protocols: "TLSv1 TLSv1.1 TLSv1.2 TLSv1.3"

13. nginx-ingress重试机制

问题现象：通过ingress的域名访问后端，调用一次，服务日志里显示调用了三次，nginx-ingress的请求日志出现了3次返回码

 - - [2022-09-14T11:40:27+00:00] 1663155627.705 "GET 
/api HTTP/1.1" 502 150 "-" "curl/7.29.0" 121 
3.073 [background-sino-xray-9091] [] [10.160.253.235:9091, 
10.160.253.235:9091, 10.160.253.235:9091] [0, 0, 0] [1.028, 1.024, 
1.024] [502, 502, 502] 7d407c28f5d7d2aae3afd0c368fbe9e5

问题原因：从访问日志看，nginx ingress controller出现3次调后端rs，为什么会调用了3次呢？这里是因为nginx本身的重试机制，具体参数可以参考文档

http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream

但是为什么是重试3次呢，这里可以到nginx-ingress controller看下配置文件

proxy_next_upstream                     error timeout;  #哪些情况下应将请求传递到下一台服务器
proxy_next_upstream_timeout             0; # 限制可以将请求传递到下一台服务器的时间，0表示将关闭此限制
proxy_next_upstream_tries               3; # 限制将请求传递到下一台服务器的可能尝试次数。0表示关闭此限制

可以看出tke的nginx ingress controller默认是重试3次，因此日志里会显示调用了3次。如果要关闭重试，直接在全局configmap里proxy_next_upstream_tries设置为0

14. nginx-ingress无法reload全局配置

https://cloud.tencent.com/developer/article/2040556

API Nginx TCP/IP 网络安全

TKE集群中nginx-ingress使用实践