将指标推送到Prometheus Pushgateway经常失败

问题描述

我正在使用Prometheus Client library将指标推送到Pushgateway。推送指标时，我经常遇到错误。如何找到此问题的根本原因？

 push_to_gateway(
  File "/usr/local/lib/python3.8/dist-packages/prometheus_client/exposition.py",line 285,in push_to_gateway
    _use_gateway('PUT',gateway,job,registry,grouping_key,timeout,handler)
  File "/usr/local/lib/python3.8/dist-packages/prometheus_client/exposition.py",line 358,in _use_gateway
    handler(
  File "/usr/local/lib/python3.8/dist-packages/prometheus_client/exposition.py",line 217,in handle
    resp = build_opener(HTTPHandler).open(request,timeout=timeout)
  File "/usr/lib/python3.8/urllib/request.py",line 525,in open
    response = self._open(req,data)
  File "/usr/lib/python3.8/urllib/request.py",line 542,in _open
    result = self._call_chain(self.handle_open,protocol,protocol +
  File "/usr/lib/python3.8/urllib/request.py",line 502,in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py",line 1369,in https_open
    return self.do_open(http.client.HTTPSConnection,req,File "/usr/lib/python3.8/urllib/request.py",line 1330,in do_open
    r = h.getresponse()
  File "/usr/lib/python3.8/http/client.py",line 1332,in getresponse
    response.begin()
  File "/usr/lib/python3.8/http/client.py",line 303,in begin
    version,status,reason = self._read_status()
  File "/usr/lib/python3.8/http/client.py",line 272,in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without respons

解决方法

我们在内部使用Kubernetes来部署服务，在这里我试图通过推送度量标准来使用其入口推送网关。将其更改为使用Kubernetes服务名称而不是Ingress可以显着减少这些错误，但是如果将服务重定位到其他集群，则这不是可移植的解决方案。对我有用的解决方案是使用python装饰器函数重试