kubernetes 入口 502 错误网关

问题描述

我在具有以下节点的裸机（使用 VMware 虚拟机）上安装了 Kubernetes 集群

master-01   Ready    control-plane,master   5d3h   v1.21.3
master-02   Ready    control-plane,master   5d3h   v1.21.3
master-03   Ready    control-plane,master   5d3h   v1.21.3
worker-01   Ready    <none>                 5d2h   v1.21.3
worker-02   Ready    <none>                 5d2h   v1.21.3
worker-03   Ready    <none>                 5d2h   v1.21.3

Metallb 安装为集群负载均衡器，calico 安装为 CNI

我还用 helm 安装了 nginx-ingress-controller

$ helm repo add nginx-stable https://helm.nginx.com/stable
$ helm repo update
$ helm install ingress-controller nginx-stable/nginx-ingress

我部署了一个简单的 nginx 服务器用于测试

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx-app
  template:
    metadata:
      labels:
        app: nginx-app
    spec:
      containers:
        - name: nginx
          image: nginx
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  selector:
    app: nginx-app
  #type: LoadBalancer
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-myapp
  annotations:
    # use the shared ingress-nginx
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: myapp.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx-service
            port:
              number: 80

我的负载均衡器类型的部署从 metallb 获取它们的 IP 并且工作正常，但是当我添加入口时虽然分配了 IP，但我收到错误 502 bad gateway，如下所示：

防火墙已启用，但所需的端口已打开

6443/tcp 2379-2380/tcp 10250-10252/tcp 179/tcp 7946/tcp 7946/udp 8443/tcp on master nodes
10250/tcp 30000-32767/tcp 7946/tcp 7946/udp 8443/tcp 179/tcp on worker nodes

我的服务和 Pod 运行良好

 kubectl get svc -o wide
NAME                               TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)                      AGE   SELECTOR
ingress-controller-nginx-ingress   LoadBalancer   10.101.17.180   10.1.210.100   80:31509/TCP,443:30004/TCP   33m   app=ingress-controller-nginx-ingress
kubernetes                         ClusterIP      10.96.0.1       <none>         443/TCP                      6d    <none>
nginx-service                      ClusterIP      10.101.48.198   <none>         80/TCP                       31m   app=nginx-app

我的入口日志给我错误，没有到内部 IP 的路由

2021/07/29 07:46:24 [error] 42#42: *8 connect() failed (113: No route to host) while connecting to upstream,client: 10.1.210.5,server: myapp.com,request: "GET / HTTP/1.1",upstream: "http://192.168.171.17:80/",host: "myapp.com"
10.1.210.5 - - [29/Jul/2021:07:46:24 +0000] "GET / HTTP/1.1" 502 157 "-" "curl/7.68.0" "-"
W0729 07:50:16.416830       1 warnings.go:70] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+,unavailable in v1.22+; use networking.k8s.io/v1 Ingress
192.168.2.131 - - [29/Jul/2021:07:51:03 +0000] "GET / HTTP/1.1" 404 555 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/92.0.4515.107 Safari/537.36" "-"
192.168.2.131 - - [29/Jul/2021:07:51:03 +0000] "GET /favicon.ico HTTP/1.1" 404 555 "http://10.1.210.100/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/92.0.4515.107 Safari/537.36" "-"
W0729 07:56:43.420282       1 warnings.go:70] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+,unavailable in v1.22+; use networking.k8s.io/v1 Ingress
W0729 08:05:28.422594       1 warnings.go:70] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+,unavailable in v1.22+; use networking.k8s.io/v1 Ingress
W0729 08:10:45.425329       1 warnings.go:70] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+,unavailable in v1.22+; use networking.k8s.io/v1 Ingress
2021/07/29 08:13:59 [error] 42#42: *12 connect() failed (113: No route to host) while connecting to upstream,host: "myapp.com"
10.1.210.5 - - [29/Jul/2021:08:13:59 +0000] "GET / HTTP/1.1" 502 157 "-" "curl/7.68.0" "-"
2021/07/29 08:14:09 [error] 42#42: *14 connect() failed (113: No route to host) while connecting to upstream,host: "myapp.com"
10.1.210.5 - - [29/Jul/2021:08:14:09 +0000] "GET / HTTP/1.1" 502 157 "-" "curl/7.68.0" "-"

有什么想法吗？

编辑：如这里询问的服务和 Pod 描述

$ kubectl describe pod nginx-deployment-6f7d8d4d55-sncdr
Name:         nginx-deployment-6f7d8d4d55-sncdr
Namespace:    default
Priority:     0
Node:         worker-01/10.1.210.63
Start Time:   Thu,29 Jul 2021 08:43:59 +0100
Labels:       app=nginx-app
              pod-template-hash=6f7d8d4d55
Annotations:  cni.projectcalico.org/podIP: 192.168.171.17/32
              cni.projectcalico.org/podIPs: 192.168.171.17/32
Status:       Running
IP:           192.168.171.17
IPs:
  IP:           192.168.171.17
Controlled By:  ReplicaSet/nginx-deployment-6f7d8d4d55
Containers:
  nginx:
    Container ID:   docker://fc61b73f8a833ad13b8956d8ce151b221b75a58a9a2fbae928464f3b0a77cca2
    Image:          nginx
    Image ID:       docker-pullable://nginx@sha256:8f335768880da6baf72b70c701002b45f4932acae8d574dedfddaf967fc3ac90
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Thu,29 Jul 2021 08:44:01 +0100
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wkc48 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  kube-api-access-wkc48:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  16m   default-scheduler  Successfully assigned default/nginx-deployment-6f7d8d4d55-sncdr to worker-01
  Normal  Pulling    16m   kubelet            Pulling image "nginx"
  Normal  Pulled     16m   kubelet            Successfully pulled image "nginx" in 1.51808376s
  Normal  Created    16m   kubelet            Created container nginx
  Normal  Started    16m   kubelet            Started container nginx

$ kubectl describe svc ingress-controller-nginx-ingress
Name:                     ingress-controller-nginx-ingress
Namespace:                default
Labels:                   app.kubernetes.io/instance=ingress-controller
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=ingress-controller-nginx-ingress
                          helm.sh/chart=nginx-ingress-0.10.0
Annotations:              meta.helm.sh/release-name: ingress-controller
                          meta.helm.sh/release-namespace: default
Selector:                 app=ingress-controller-nginx-ingress
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.101.17.180
IPs:                      10.101.17.180
LoadBalancer Ingress:     10.1.210.100
Port:                     http  80/TCP
TargetPort:               80/TCP
NodePort:                 http  31509/TCP
Endpoints:                192.168.37.202:80
Port:                     https  443/TCP
TargetPort:               443/TCP
NodePort:                 https  30004/TCP
Endpoints:                192.168.37.202:443
Session Affinity:         None
External Traffic Policy:  Local
HealthCheck NodePort:     31108
Events:
  Type    Reason        Age                    From                Message
  ----    ------        ----                   ----                -------
  Normal  IPAllocated   18m                    metallb-controller  Assigned IP "10.1.210.100"
  Normal  nodeAssigned  3m21s (x182 over 18m)  metallb-speaker     announcing from node "worker-02"

$ kubectl describe svc nginx-service
Name:              nginx-service
Namespace:         default
Labels:            <none>
Annotations:       <none>
Selector:          app=nginx-app
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.101.48.198
IPs:               10.101.48.198
Port:              <unset>  80/TCP
TargetPort:        80/TCP
Endpoints:         192.168.171.17:80
Session Affinity:  None
Events:            <none>

$ kubectl exec -it ingress-controller-nginx-ingress-dd5db86dc-gqdpm -- /bin/bash
nginx@ingress-controller-nginx-ingress-dd5db86dc-gqdpm:/$ curl 192.168.171.17:80
curl: (7) Failed to connect to 192.168.171.17 port 80: No route to host
nginx@ingress-controller-nginx-ingress-dd5db86dc-gqdpm:/$ curl 192.168.171.17
curl: (7) Failed to connect to 192.168.171.17 port 80: No route to host
nginx@ingress-controller-nginx-ingress-dd5db86dc-gqdpm:/$ curl 10.101.48.198
curl: (7) Failed to connect to 10.101.48.198 port 80: Connection timed out
nginx@ingress-controller-nginx-ingress-dd5db86dc-gqdpm:/$ curl nginx-deployment-6f7d8d4d55-sncdr
curl: (6) Could not resolve host: nginx-deployment-6f7d8d4d55-sncdr
nginx@ingress-controller-nginx-ingress-dd5db86dc-gqdpm:/$

老实说，我不明白为什么 curl svcip 不再起作用；昨天成功了。

解决方法

问题是防火墙问题，我禁用了 firewalld，现在可以使用了，我认为必须打开端口 8443，但如果有人能告诉我哪个端口，它似乎是另一个端口

谢谢