GKE：将文件日志从 docker 容器发送到 Google Cloud Logging

问题描述

我正在尝试在 GKE 上的应用程序 pod 中捕获一些基于文件的日志，并从 Google Cloud Logging 中查看它们。

由于各种原因，这些应用日志不会发送到 STDOUT 或 STDERR（因为这些日志会自动发送到 Cloud Logging）。有人建议我实施一个脚本解决方案，该解决方案跟踪日志并将它们发送到 STDOUT。但是，我希望采用 Fluentd（或 Fluentbit）日志记录代理的辅助方法，该代理将跟踪日志并将其输出到 Cloud Logging。

使用 sidecar 映像 "k8s.gcr.io/fluentd-gcp:1.30"，我尝试了以下 YAML 文件（包含 fluentd configmap 和部署）：

---
apiVersion: v1
kind: ConfigMap
Metadata:
  name: app-log-config
data:
  fluentd.conf: |
    <source>
      type tail
      format none
      path /var/log/execution*.log
      pos_file /var/log/execution.pos
      tag app.*
    </source>

    <match **>
      type google_cloud
    </match>
---
apiVersion: apps/v1
kind: Deployment
Metadata:
  name: app
  labels:
    app.kubernetes.io/name: app
    app.kubernetes.io/instance: app
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: app
      app.kubernetes.io/instance: app
  template:
    Metadata:
      labels:
        app.kubernetes.io/name: app
        app.kubernetes.io/instance: app
    spec:
      serviceAccountName: app
      volumes:
        - name: executionlogs
          emptyDir: {}
        - name: fluentdconfig
          configMap:
            name: app-log-config
      containers:
        - name: app
          image: appimage:version
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: executionlogs
              mountPath: /tmp/executionLogs
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
        - name: log-agent
          image: "k8s.gcr.io/fluentd-gcp:1.30"
          imagePullPolicy: IfNotPresent
          env:
            - name: FLUENTD_ARGS
              value: "-c /etc/fluentd-config/fluentd.conf"
          volumeMounts:
            - name: executionlogs
              mountPath: /var/log
            - name: fluentdconfig
              mountPath: /etc/fluentd-config

最初，sidecar 日志抛出 403 错误，因为我没有为服务帐户授予必要的权限（我使用的是 GKE 工作负载身份，并且相应的 GCP IAM 服务帐户需要添加 logWriter 权限）。修复错误后，我得到以下日志：

2021-06-27 12:49:09 +0000 [info]: fluent/supervisor.rb:471:read_config: reading config file path="/etc/fluentd-config/fluentd.conf"
2021-06-27 12:49:09 +0000 [info]: fluent/supervisor.rb:337:supervise: starting fluentd-0.12.29
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-google-cloud' version '0.5.2'
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-kafka' version '0.3.1'
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-mongo' version '0.7.15'
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-record-reformer' version '0.8.2'
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.5'
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-s3' version '0.7.1'
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-scribe' version '0.10.14'
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-systemd' version '0.0.5'
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-td' version '0.10.29'
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-td-monitoring' version '0.2.2'
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-webhdfs' version '0.4.2'
2021-06-27 12:49:09 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluentd' version '0.12.29'
2021-06-27 12:49:09 +0000 [info]: fluent/agent.rb:129:add_match: adding match pattern="**" type="google_cloud"
2021-06-27 12:49:10 +0000 [info]: plugin/out_google_cloud.rb:519:block in detect_platform: Detected GCE platform
2021-06-27 12:49:10 +0000 [info]: plugin/out_google_cloud.rb:290:configure: Logs viewer address: https://console.developers.google.com/
project/projectname/logs?service=compute.googleapis.com&key1=instance&key2=9071465168741286442
2021-06-27 12:49:10 +0000 [info]: fluent/root_agent.rb:147:add_source: adding source type="tail"
2021-06-27 12:49:10 +0000 [info]: fluent/engine.rb:133:configure: using configuration file: <ROOT>
  <source>
    type tail
    format none
    path /var/log/execution*.log
    pos_file /var/log/execution.pos
    tag app.*
  </source>
  <match **>
    type google_cloud
  </match>
</ROOT>
2021-06-27 12:52:10 +0000 [info]: plugin/in_tail.rb:557:initialize: following tail of /var/log/execution1.log
2021-06-27 12:53:10 +0000 [info]: plugin/out_google_cloud.rb:451:block in write: Successfully sent to Google Cloud Logging API.

尽管消息成功，但我在 Cloud Logging 端看不到任何内容。

那么，这是我的问题：

对于我的用例，是否有更好的解决方案？
我应该使用边车图像吗？我找不到任何其他流畅的图像，而我使用的图像已经 3 岁了。我更喜欢使用 Google 推荐的东西，而不是自己创建。
我还需要做什么才能查看 Cloud Logging 上的日志？我该如何进一步调试？

谢谢！

解决方法

我没有看到过滤器、解析器、输入或输出的 conf。应该有输出 conf 数据，如 - [输出] 名称标准输出匹配 * 在此处查看更多详细信息 - https://docs.fluentd.org/output/stdout https://docs.fluentd.org/input/tail

我已尝试实施您实施的配置，但遇到了同样的问题。然后，我将所有源配置为将输出流式传输到 STDOUT，并且能够在 Cloud Logging 仪表板上查看日志。

以下是我使用过的示例配置。

Sample_map-config.yaml：

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
data:
  fluentd.conf: |
    <source>
      type tail
      format none
      path /var/log/1.log
      pos_file /var/log/1.log.pos
      tag count.format1
    </source>

    <source>
      type tail
      format none
      path /var/log/2.log
      pos_file /var/log/2.log.pos
      tag count.format2
    </source>

    <match **>
      type stdout
    </match>

示例 pod.yaml：

apiVersion: v1
kind: Pod
metadata:
  name: counter
spec:
  containers:
  - name: count
    image: busybox
    args:
    - /bin/sh
    - -c
    - >
      i=0;
      while true;
      do
        echo "$i: $(date)" >> /var/log/1.log;
        echo "$(date) INFO $i" >> /var/log/2.log;
        i=$((i+1));
        sleep 1;
      done      
    volumeMounts:
    - name: varlog
      mountPath: /var/log
  - name: count-agent
    image: k8s.gcr.io/fluentd-gcp:1.30
    env:
    - name: FLUENTD_ARGS
      value: -c /etc/fluentd-config/fluentd.conf
    volumeMounts:
    - name: varlog
      mountPath: /var/log
    - name: config-volume
      mountPath: /etc/fluentd-config
  volumes:
  - name: varlog
    emptyDir: {}
  - name: config-volume
    configMap:
      name: fluentd-config

fluentd google-cloud-logging google-kubernetes-engine kubernetes sidecar

GKE：将文件日志从 docker 容器发送到 Google Cloud Logging

问题描述

解决方法

相关问答