问题描述
我正在尝试通过Google Cloud Build在jfrog存储库中pip安装私有软件包。我可以通过https:<USER>:<API_KEY>@<artifactory_url>访问jfrog存储库，也可以进行pip install <package_name_and_version> https:<USER>:<API_KEY>@<artifactory_url>
我想将此步骤集成到使用Cloud KMS的云构建中，以在pip安装期间解密API_KEY。 我已经使用以下命令对API_KEY字符串进行了加密
# Create a local file with the secret
echo "MyAPIKEY" > plain_pwd.txt
# To encrypt a secret using KMS
gcloud kms encrypt \
--plaintext-file=plain_pwd.txt \
--ciphertext-file=cipher_pwd.enc.txt \
--location=global \
--keyring=<keyring> \
--key=<key>
# Encode the binary encoded secret as base64 string
base64 cipher_pwd.enc.txt -w 0 > cipher_pw.enc.64.txt
cloudbuild.yaml中指定的机密为：
secrets:
- kmsKeyName: projects/<project_id>/locations/global/keyRings/<keyring>/cryptoKeys/<key>
secretEnv:
APIKEY: <base64 encrypted string from cloud kms encrypt command>
这是我的pip安装的cloudbuild.yaml步骤：
-
args:
- "-m"
- pip
- install
- "-t"
- /workspace/lib
- "schema-registry-client==0.8.14.dev0"
- "--extra-index-url"
- 'https://adminuser:$$APIKEY@<artifactory_url>'
entrypoint: python3
secretEnv: ['APIKEY']
id: INSTALL_SCHEMA_REGISTRY
name: "python:3.7"
在执行此步骤时，我在云构建中收到“ EOFError：读取行时EOF”：
查找索引：https://pypi.org/simple，
https://onpat:****@artifactory.build.****.****.com/artifactory/api/pypi/dpfw-pypi-dev-local/simple
ERROR: Exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/pip/_internal/cli/base_command.py",line 216,in _main
status = self.run(options,args)
File "/usr/local/lib/python3.7/site-packages/pip/_internal/cli/req_command.py",line 182,in wrapper
return func(self,options,args)
File "/usr/local/lib/python3.7/site-packages/pip/_internal/commands/install.py",line 325,in run
reqs,check_supported_wheels=not options.target_dir
File "/usr/local/lib/python3.7/site-packages/pip/_internal/resolution/legacy/resolver.py",line 183,in resolve
discovered_reqs.extend(self._resolve_one(requirement_set,req))
File "/usr/local/lib/python3.7/site-packages/pip/_internal/resolution/legacy/resolver.py",line 388,in _resolve_one
abstract_dist = self._get_abstract_dist_for(req_to_install)
File "/usr/local/lib/python3.7/site-packages/pip/_internal/resolution/legacy/resolver.py",line 339,in _get_abstract_dist_for
self._populate_link(req)
File "/usr/local/lib/python3.7/site-packages/pip/_internal/resolution/legacy/resolver.py",line 305,in _populate_link
req.link = self._find_requirement_link(req)
File "/usr/local/lib/python3.7/site-packages/pip/_internal/resolution/legacy/resolver.py",line 270,in _find_requirement_link
best_candidate = self.finder.find_requirement(req,upgrade)
File "/usr/local/lib/python3.7/site-packages/pip/_internal/index/package_finder.py",line 899,in find_requirement
req.name,specifier=req.specifier,hashes=hashes,File "/usr/local/lib/python3.7/site-packages/pip/_internal/index/package_finder.py",line 881,in find_best_candidate
candidates = self.find_all_candidates(project_name)
File "/usr/local/lib/python3.7/site-packages/pip/_internal/index/package_finder.py",line 826,in find_all_candidates
project_url,link_evaluator=link_evaluator,line 790,in process_project_url
html_page = self._link_collector.fetch_page(project_url)
File "/usr/local/lib/python3.7/site-packages/pip/_internal/index/collector.py",line 643,in fetch_page
return _get_html_page(location,session=self.session)
File "/usr/local/lib/python3.7/site-packages/pip/_internal/index/collector.py",line 455,in _get_html_page
resp = _get_html_response(url,session=session)
File "/usr/local/lib/python3.7/site-packages/pip/_internal/index/collector.py",line 169,in _get_html_response
"Cache-Control": "max-age=0",File "/usr/local/lib/python3.7/site-packages/pip/_vendor/requests/sessions.py",line 543,in get
return self.request('GET',url,**kwargs)
File "/usr/local/lib/python3.7/site-packages/pip/_internal/network/session.py",line 421,in request
return super(PipSession,self).request(method,*args,**kwargs)
File "/usr/local/lib/python3.7/site-packages/pip/_vendor/requests/sessions.py",line 530,in request
resp = self.send(prep,**send_kwargs)
File "/usr/local/lib/python3.7/site-packages/pip/_vendor/requests/sessions.py",line 650,in send
r = dispatch_hook('response',hooks,r,**kwargs)
File "/usr/local/lib/python3.7/site-packages/pip/_vendor/requests/hooks.py",line 31,in dispatch_hook
_hook_data = hook(hook_data,**kwargs)
File "/usr/local/lib/python3.7/site-packages/pip/_internal/network/auth.py",line 256,in handle_401
username,password,save = self._prompt_for_password(parsed.netloc)
File "/usr/local/lib/python3.7/site-packages/pip/_internal/network/auth.py",line 226,in _prompt_for_password
username = ask_input("User for {}: ".format(netloc))
File "/usr/local/lib/python3.7/site-packages/pip/_internal/utils/misc.py",line 259,in ask_input
return input(message)
EOFError: EOF when reading a line
Finished Step #2 - "INSTALL_SCHEMA_REGISTRY"
ERROR
ERROR: build step 2 "python:3.7" Failed: step exited with non-zero status: 2
此外，我尝试了在相同密文上进行gcloud kms解密，并获得了原始API KEY。因此，我认为加密/解密不是这里的问题。我还授予了访问Cloud kms的云构建服务代理的必要权限。
有关如何解决此问题的任何建议/帮助？
解决方法
我找到了解决此问题的另一种方法，显然，pip cli在扩展参数和env变量时存在问题。您可以使用Google Cloud Secret管理器来存储凭据（在这种情况下为JFrog API_KEY）。使用gcloud secrets versions access <version_id> --secret=<secret_key>在运行时检索密码。在构建步骤中使用~/.pip/pip.conf在--extra-index-url=https://user:<api_key_secret_value>@<artifactory_url>处创建pip.conf文件，然后在后续构建步骤中执行pip install <package_name>。唯一的缺点是它在构建过程中增加了一个额外的步骤。我创建了bash脚本，以在构建步骤中将密钥读取到文件，并在下一步构建中使用该密钥值创建pip.conf值。使用云构建卷在步骤之间传递文件。
#read_secret.sh
#!/bin/bash
gcloud secrets versions access 1 --secret=artifactory-api-key > "/data/key.txt"
使用以下bash脚本在下一步中安装软件包
#install_module.sh
#!/bin/bash
export API_KEY=$(cat "/data/key.txt")
echo $API_KEY
mkdir -p ~/.pip
echo -e "[global]\n--extra-index-url=https://myuser:$API_KEY@<artifactory_url>" > ~/.pip/pip.conf
pip install -t "/workspace/lib" private-package-lib==0.1.2