问题描述
我正在尝试重现 this tutorial 以在 Dataflow 上运行 Flex 模板。
Output from execution of subprocess: b'Collecting apache-beam[gcp]==2.26.0
Downloading apache-beam-2.26.0.zip (2.4 MB)
Saved /tmp/dataflow-requirements-cache/apache-beam-2.26.0.zip
Collecting avro-python3!=1.9.2,<1.10.0,>=1.8.1
Downloading avro-python3-1.9.2.1.tar.gz (37 kB)
Saved /tmp/dataflow-requirements-cache/avro-python3-1.9.2.1.tar.gz
Collecting crcmod<2.0,>=1.7
Downloading crcmod-1.7.tar.gz (89 kB)
Saved /tmp/dataflow-requirements-cache/crcmod-1.7.tar.gz
Collecting dill<0.3.2,>=0.3.1.1
Downloading dill-0.3.1.1.tar.gz (151 kB)
Saved /tmp/dataflow-requirements-cache/dill-0.3.1.1.tar.gz
Collecting fastavro<2,>=0.21.4
Downloading fastavro-1.2.2.tar.gz (663 kB)
Saved /tmp/dataflow-requirements-cache/fastavro-1.2.2.tar.gz
Collecting future<1.0.0,>=0.18.2
Downloading future-0.18.2.tar.gz (829 kB)
Saved /tmp/dataflow-requirements-cache/future-0.18.2.tar.gz
Collecting grpcio<2,>=1.29.0
Downloading grpcio-1.34.0.tar.gz (21.0 MB)
Saved /tmp/dataflow-requirements-cache/grpcio-1.34.0.tar.gz
Collecting hdfs<3.0.0,>=2.1.0
Downloading hdfs-2.5.8.tar.gz (41 kB)
Saved /tmp/dataflow-requirements-cache/hdfs-2.5.8.tar.gz
Collecting httplib2<0.18.0,>=0.8
Downloading httplib2-0.17.4.tar.gz (256 kB)
Saved /tmp/dataflow-requirements-cache/httplib2-0.17.4.tar.gz
Collecting mock<3.0.0,>=1.0.1
Downloading mock-2.0.0.tar.gz (73 kB)
Saved /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting numpy<2,>=1.14.3
Downloading numpy-1.19.4.zip (7.3 MB)
Installing build dependencies: started
Installing build dependencies: still running...
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing wheel Metadata: started
Preparing wheel Metadata: finished with status 'done'
Saved /tmp/dataflow-requirements-cache/numpy-1.19.4.zip
Collecting pymongo<4.0.0,>=3.8.0
Downloading pymongo-3.11.2.tar.gz (770 kB)
Saved /tmp/dataflow-requirements-cache/pymongo-3.11.2.tar.gz
Collecting oauth2client<5,>=2.0.1
Downloading oauth2client-4.1.3.tar.gz (155 kB)
Saved /tmp/dataflow-requirements-cache/oauth2client-4.1.3.tar.gz
Collecting protobuf<4,>=3.12.2
Downloading protobuf-3.14.0.tar.gz (227 kB)
Saved /tmp/dataflow-requirements-cache/protobuf-3.14.0.tar.gz
Collecting pyarrow<0.18.0,>=0.15.1
Downloading pyarrow-0.17.1.tar.gz (2.6 MB)
Installing build dependencies: started
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing wheel Metadata: started
Preparing wheel Metadata: finished with status 'error'
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_Metadata_for_build_wheel /tmp/tmp_p9la_sd
cwd: /tmp/pip-download-uramyun1/pyarrow
Complete output (42 lines):
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py",line 280,in <module>
main()
File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py",line 263,in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py",line 133,in prepare_Metadata_for_build_wheel
return hook(Metadata_directory,config_settings)
File "/tmp/pip-build-env-5__v4cuc/overlay/lib/python3.7/site-packages/setuptools/build_Meta.py",line 161,in prepare_Metadata_for_build_wheel
self.run_setup()
File "/tmp/pip-build-env-5__v4cuc/overlay/lib/python3.7/site-packages/setuptools/build_Meta.py",line 254,in run_setup
self).run_setup(setup_script=setup_script)
File "/tmp/pip-build-env-5__v4cuc/overlay/lib/python3.7/site-packages/setuptools/build_Meta.py",line 145,in run_setup
exec(compile(code,__file__,'exec'),locals())
File "setup.py",line 605,in <module>
url='https://arrow.apache.org/'
File "/tmp/pip-build-env-5__v4cuc/overlay/lib/python3.7/site-packages/setuptools/__init__.py",line 153,in setup
return distutils.core.setup(**attrs)
File "/usr/local/lib/python3.7/distutils/core.py",line 108,in setup
_setup_distribution = dist = klass(attrs)
File "/tmp/pip-build-env-5__v4cuc/overlay/lib/python3.7/site-packages/setuptools/dist.py",line 424,in __init__
k: v for k,v in attrs.items()
File "/usr/local/lib/python3.7/distutils/dist.py",line 292,in __init__
self.finalize_options()
File "/tmp/pip-build-env-5__v4cuc/overlay/lib/python3.7/site-packages/setuptools/dist.py",line 695,in finalize_options
ep(self)
File "/tmp/pip-build-env-5__v4cuc/overlay/lib/python3.7/site-packages/setuptools/dist.py",line 702,in _finalize_setup_keywords
ep.load()(self,ep.name,value)
File "/tmp/pip-build-env-5__v4cuc/overlay/lib/python3.7/site-packages/setuptools_scm/integration.py",line 26,in version_keyword
dist.Metadata.version = _get_version(config)
File "/tmp/pip-build-env-5__v4cuc/overlay/lib/python3.7/site-packages/setuptools_scm/__init__.py",line 173,in _get_version
parsed_version = _do_parse(config)
File "/tmp/pip-build-env-5__v4cuc/overlay/lib/python3.7/site-packages/setuptools_scm/__init__.py",line 119,in _do_parse
parse_result = _call_entrypoint_fn(config.absolute_root,config,config.parse)
File "/tmp/pip-build-env-5__v4cuc/overlay/lib/python3.7/site-packages/setuptools_scm/__init__.py",line 54,in _call_entrypoint_fn
return fn(root)
File "setup.py",line 539,in parse_git
return parse(root,**kwargs)
File "/tmp/pip-build-env-5__v4cuc/overlay/lib/python3.7/site-packages/setuptools_scm/git.py",line 95,in parse
require_command("git")
File "/tmp/pip-build-env-5__v4cuc/overlay/lib/python3.7/site-packages/setuptools_scm/utils.py",line 151,in require_command
raise EnvironmentError("%r was not found" % name)
OSError: 'git' was not found
----------------------------------------
ERROR: Command errored out with exit status 1: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_Metadata_for_build_wheel /tmp/tmp_p9la_sd Check the logs for full command output.
WARNING: You are using pip version 20.2.3; however,version 20.3.3 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
其他人有同样的问题吗？
如果有帮助，这里是镜像的 Dockerfile：
# copyright 2020 Google LLC
#
# Licensed under the Apache License,Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,software
# distributed under the License is distributed on an "AS IS" BASIS,# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
FROM gcr.io/dataflow-templates-base/python3-template-launcher-base
ARG workdir=/dataflow/template
RUN mkdir -p ${workdir}
workdir ${workdir}
# Due to a change in the Beam base image in version 2.24,we need to install
# libffi-dev manually as a dependency. For more @R_568_4045@ion:
# https://github.com/GoogleCloudplatform/python-docs-samples/issues/4891
RUN apt-get update && apt-get install -y libffi-dev && rm -rf /var/lib/apt/lists/*
copY requirements.txt .
copY streaming_beam.py .
ENV FLEX_TEMPLATE_PYTHON_REQUIREMENTS_FILE="${workdir}/requirements.txt"
ENV FLEX_TEMPLATE_PYTHON_PY_FILE="${workdir}/streaming_beam.py"
RUN pip install -U -r ./requirements.txt
解决方法
看起来你需要在你的 dockerfile 中安装 git。添加这一行：
RUN apt-get install git
这看起来像是一个错误，其中某些上游依赖项开始需要 git 才能构建，但基础映像当前不包含它。我在这里提交了一个问题：https://issuetracker.google.com/issues/176570473