如何在Google Cloudbuild步骤中保留变量？

问题描述

我有一个cloudbuild.json，用于将管道上传到gcp kubeflow。现在，我想添加另一个步骤，以获取最新的管道ID，然后将其作为实验运行。 所以我的主要问题是如何在后续步骤中获取管道ID。我已经编写了一个小脚本来获取最新的管道ID，并将其添加为从docker运行的步骤，但是现在我不确定如何获取此管道ID。

这是我的coudbuild.json

{
    "steps": [
        {
            "name": "gcr.io/cloud-builders/docker","args": [
                "build","-t","trainer_image","."
            ],"dir": "./trainer_image/"
        },{
            "name": "gcr.io/cloud-builders/docker","base_image","dir": "./base_image/"
        },{
            "name": "gcr.io/dmgcp-pkg-internal-poc-oct-04/kfp-cli","args": [
                "-c","dsl-compile --py covertype_training_pipeline.py --output covertype_training_pipeline.yaml"
            ],"env": [
                "BASE_IMAGE=gcr.io/dmgcp-pkg-internal-poc-oct-04/base_image:test","TRAINER_IMAGE=gcr.io/dmgcp-pkg-internal-poc-oct-04/trainer_image:test","RUNTIME_VERSION=1.15","PYTHON_VERSION=3.7","COMPONENT_URL_SEARCH_PREFIX=https://raw.githubusercontent.com/kubeflow/pipelines/0.2.5/components/gcp/","USE_KFP_SA=False"
            ],"dir": "./pipeline/"
        },"kfp --endpoint 66df1d31e46e6510-dot-us-central2.pipelines.googleusercontent.com pipeline upload -p credit_fraud_training_test covertype_training_pipeline.yaml"
            ],"id_image","dir": "./id_image/"
        }
    ],"images": [
        "gcr.io/dmgcp-pkg-internal-poc-oct-04/trainer_image:test","gcr.io/dmgcp-pkg-internal-poc-oct-04/base_image:test"
    ]
}

这是我的python脚本，用于获取最新的管道ID

import kfp
client = kfp.Client(host='66df1d31e46e6510-dot-us-central2.pipelines.googleusercontent.com')
pipelines = client.list_pipelines()
total_pipeline = len(pipelines.pipelines)
latest_pipeline_id = pipelines.pipelines[total_pipeline-1].id
with open('/workspace/pipeline.txt') as file:
    file.write(latest_pipeline_id)

解决方法

您不能在步骤之间保留变量。您只能保留文件。但是，在一个步骤中，您可以重用局部变量。

- name: "gcr.io/cloud-builders/docker",entrypoint: "bash"
  args: 
   - "-c"
   - |
      export DOCKER_VERSION=$(docker version)
      echo $$DOCKER_VERSION #Reuse local variable
      echo $$DOCKER_VERSION > DOCKER_VERSION.txt #Save the variable for the next step
- name: "gcr.io/cloud-builders/docker",entrypoint: "bash"
  args: 
   - "-c"
   - |
      export DOCKER_VERSION=$(cat DOCKER_VERSION.txt) #Get the value saved in file
      echo $$DOCKER_VERSION #Reuse local variable