问题描述
我正在尝试在 Linux 机器上将 pyspark 安装到我的虚拟环境中。使用了以下代码，它以前工作正常。突然之间，我们遇到了一个问题。
conda install -q -y -c conda-forge pyspark==2.3.1
DEBUG conda.gateways.logging:set_verbosity(148): verbosity set to 2
Solving environment: ...working... DEBUG conda.core.solve:solve_final_state(142): solving prefix /***jenkins/workspace/utilization-build-branch-pipeline/miniconda/envs/utilization_venv
specs_to_remove: frozenset()
specs_to_add: frozenset({MatchSpec(name=LowerStrMatch('pyspark'),version=VersionSpec('2.3.1.*'))})
prune: False
DEBUG conda.core.package_cache_data:_check_writable(237): package cache directory '/***jenkins/workspace/utilization-build-branch-pipeline/miniconda/pkgs' writable: True
DEBUG conda.core.package_cache_data:_check_writable(237): package cache directory '/home/olgaci/.conda/pkgs' writable: True
DEBUG conda.core.subdir_data:_load(216): Using cached repodata for https://repo.anaconda.com/pkgs/free/noarch at /***jenkins/workspace/utilization-build-branch-pipeline/miniconda/pkgs/cache/d77296e0.json. Timeout in 25 sec
DEBUG conda.core.subdir_data:_read_pickled(293): found pickle file /***jenkins/workspace/utilization-build-branch-pipeline/miniconda/pkgs/cache/d77296e0.q
DEBUG conda.core.subdir_data:_load(216): Using cached repodata for https://repo.anaconda.com/pkgs/main/noarch at /***jenkins/workspace/utilization-build-branch-pipeline/miniconda/pkgs/cache/3e39a7aa.json. Timeout in 25 sec
DEBUG conda.core.subdir_data:_read_pickled(293): found pickle file /***jenkins/workspace/utilization-build-branch-pipeline/miniconda/pkgs/cache/3e39a7aa.q
DEBUG conda.core.subdir_data:_load(216): Using cached repodata for https://repo.anaconda.com/pkgs/pro/linux-64 at /***jenkins/workspace/utilization-build-branch-pipeline/miniconda/pkgs/cache/7b043413.json. Timeout in 23 sec
DEBUG conda.core.subdir_data:_read_pickled(293): found pickle file /***jenkins/workspace/utilization-build-branch-pipeline/miniconda/pkgs/cache/7b043413.q
DEBUG conda.core.subdir_data:_load(216): Using cached repodata for https://repo.anaconda.com/pkgs/main/linux-64 at /***jenkins/workspace/utilization-build-branch-pipeline/miniconda/pkgs/cache/47929eba.json. Timeout in 23 sec
DEBUG conda.core.subdir_data:_read_pickled(293): found pickle file /***jenkins/workspace/utilization-build-branch-pipeline/miniconda/pkgs/cache/47929eba.q
DEBUG conda.core.subdir_data:_load(216): Using cached repodata for https://repo.anaconda.com/pkgs/pro/noarch at /***jenkins/workspace/utilization-build-branch-pipeline/miniconda/pkgs/cache/5fa16146.json. Timeout in 23 sec
DEBUG conda.core.subdir_data:_read_pickled(293): found pickle file /***jenkins/workspace/utilization-build-branch-pipeline/miniconda/pkgs/cache/5fa16146.q
DEBUG conda.core.subdir_data:_load(216): Using cached repodata for https://conda.anaconda.org/conda-forge/noarch at /***jenkins/workspace/utilization-build-branch-pipeline/miniconda/pkgs/cache/09cdf8bf.json. Timeout in 251 sec
DEBUG conda.core.subdir_data:_read_pickled(293): found pickle file /***jenkins/workspace/utilization-build-branch-pipeline/miniconda/pkgs/cache/09cdf8bf.q
DEBUG conda.core.subdir_data:_load(216): Using cached repodata for https://repo.anaconda.com/pkgs/r/linux-64 at /***jenkins/workspace/utilization-build-branch-pipeline/miniconda/pkgs/cache/2ce54b42.json. Timeout in 22 sec
DEBUG conda.core.subdir_data:_read_pickled(293): found pickle file /***jenkins/workspace/utilization-build-branch-pipeline/miniconda/pkgs/cache/2ce54b42.q
DEBUG conda.core.subdir_data:_load(216): Using cached repodata for https://repo.anaconda.com/pkgs/free/linux-64 at /***jenkins/workspace/utilization-build-branch-pipeline/miniconda/pkgs/cache/98cb6fa3.json. Timeout in 22 sec
DEBUG conda.core.subdir_data:_read_pickled(293): found pickle file /***jenkins/workspace/utilization-build-branch-pipeline/miniconda/pkgs/cache/98cb6fa3.q
DEBUG conda.core.subdir_data:_load(216): Using cached repodata for https://conda.anaconda.org/conda-forge/linux-64 at /***jenkins/workspace/utilization-build-branch-pipeline/miniconda/pkgs/cache/497deca9.json. Timeout in 264 sec
DEBUG conda.core.subdir_data:_read_pickled(293): found pickle file /***jenkins/workspace/utilization-build-branch-pipeline/miniconda/pkgs/cache/497deca9.q
DEBUG conda.core.subdir_data:_load(216): Using cached repodata for https://repo.anaconda.com/pkgs/r/noarch at /***jenkins/workspace/utilization-build-branch-pipeline/miniconda/pkgs/cache/4ea078d6.json. Timeout in 20 sec
DEBUG conda.core.subdir_data:_read_pickled(293): found pickle file /***jenkins/workspace/utilization-build-branch-pipeline/miniconda/pkgs/cache/4ea078d6.q
Failed
Traceback (most recent call last):
File "/***jenkins/workspace/utilization-build-branch-pipeline/miniconda/lib/python3.7/site-packages/conda/exceptions.py",line 819,in __call__
return func(*args,**kwargs)
File "/***jenkins/workspace/utilization-build-branch-pipeline/miniconda/lib/python3.7/site-packages/conda/cli/main.py",line 78,in _main
exit_code = do_call(args,p)
File "/***jenkins/workspace/utilization-build-branch-pipeline/miniconda/lib/python3.7/site-packages/conda/cli/conda_argparse.py",line 77,in do_call
exit_code = getattr(module,func_name)(args,parser)
File "/***jenkins/workspace/utilization-build-branch-pipeline/miniconda/lib/python3.7/site-packages/conda/cli/main_install.py",line 11,in execute
install(args,parser,'install')
File "/***jenkins/workspace/utilization-build-branch-pipeline/miniconda/lib/python3.7/site-packages/conda/cli/install.py",line 235,in install
force_reinstall=context.force,File "/***jenkins/workspace/utilization-build-branch-pipeline/miniconda/lib/python3.7/site-packages/conda/core/solve.py",line 518,in solve_for_transaction
force_remove,force_reinstall)
File "/***jenkins/workspace/utilization-build-branch-pipeline/miniconda/lib/python3.7/site-packages/conda/core/solve.py",line 451,in solve_for_diff
final_precs = self.solve_final_state(deps_modifier,prune,ignore_pinned,force_remove)
File "/***jenkins/workspace/utilization-build-branch-pipeline/miniconda/lib/python3.7/site-packages/conda/core/solve.py",line 180,in solve_final_state
index,r = self._prepare(prepared_specs)
File "/***jenkins/workspace/utilization-build-branch-pipeline/miniconda/lib/python3.7/site-packages/conda/core/solve.py",line 592,in _prepare
self.subdirs,prepared_specs)
File "/***jenkins/workspace/utilization-build-branch-pipeline/miniconda/lib/python3.7/site-packages/conda/core/index.py",line 217,in get_reduced_index
push_record(record)
File "/***jenkins/workspace/utilization-build-branch-pipeline/miniconda/lib/python3.7/site-packages/conda/core/index.py",line 198,in push_record
for _spec in record.combined_depends:
File "/***jenkins/workspace/utilization-build-branch-pipeline/miniconda/lib/python3.7/site-packages/conda/models/records.py",line 273,in combined_depends
result = {ms.name: ms for ms in MatchSpec.merge(self.depends)}
File "/***jenkins/workspace/utilization-build-branch-pipeline/miniconda/lib/python3.7/site-packages/conda/models/match_spec.py",line 411,in merge
match_specs = tuple(cls(s) for s in match_specs)
File "/***jenkins/workspace/utilization-build-branch-pipeline/miniconda/lib/python3.7/site-packages/conda/models/match_spec.py",in <genexpr>
match_specs = tuple(cls(s) for s in match_specs)
File "/***jenkins/workspace/utilization-build-branch-pipeline/miniconda/lib/python3.7/site-packages/conda/models/match_spec.py",line 43,in __call__
return super(MatchSpecType,cls).__call__(**parsed)
File "/***jenkins/workspace/utilization-build-branch-pipeline/miniconda/lib/python3.7/site-packages/conda/models/match_spec.py",in __init__
self._match_components = self._build_components(**kwargs)
File "/***jenkins/workspace/utilization-build-branch-pipeline/miniconda/lib/python3.7/site-packages/conda/models/match_spec.py",line 367,in _build_components
return frozendict(_make(key,value) for key,value in iteritems(kwargs))
File "/***jenkins/workspace/utilization-build-branch-pipeline/miniconda/lib/python3.7/site-packages/conda/models/match_spec.py",in <genexpr>
return frozendict(_make(key,line 361,in _make
matcher = _implementors[field_name](value)
File "/***jenkins/workspace/utilization-build-branch-pipeline/miniconda/lib/python3.7/site-packages/conda/models/version.py",line 429,in __new__
self.tup = tup = tuple(VersionSpec(s) for s in spec[1:])
File "/***jenkins/workspace/utilization-build-branch-pipeline/miniconda/lib/python3.7/site-packages/conda/models/version.py",in <genexpr>
self.tup = tup = tuple(VersionSpec(s) for s in spec[1:])
File "/***jenkins/workspace/utilization-build-branch-pipeline/miniconda/lib/python3.7/site-packages/conda/models/version.py",line 445,in __new__
raise InvalidVersionSpecError(spec)
conda.exceptions.InvalidVersionSpecError: Invalid version spec: =2.7
请帮忙。
Conda Info
**********
active environment : utilization_venv
active env location : ***/jenkins/workspace/branch-pipeline/miniconda/envs/utilization_venv
shell level : 1
user config file : /home/olgaci/.condarc
populated config files :
conda version : 4.5.12
conda-build version : not installed
python version : 3.7.1.final.0
base environment : ***/jenkins/workspace/branch-pipeline/miniconda (writable)
channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/free/linux-64
https://repo.anaconda.com/pkgs/free/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
https://repo.anaconda.com/pkgs/pro/linux-64
https://repo.anaconda.com/pkgs/pro/noarch
package cache : ***/jenkins/workspace/branch-pipeline/miniconda/pkgs
/home/olgaci/.conda/pkgs
envs directories : ***/jenkins/workspace/branch-pipeline/miniconda/envs
/home/olgaci/.conda/envs
platform : linux-64
user-agent : conda/4.5.12 requests/2.21.0 cpython/3.7.1 Linux/3.10.0-1160.11.1.el7.x86_64 rhel/7.9 glibc/2.17
UID:GID : 27650:26686
netrc file : None
offline mode : False
Solving environment: ...working... done
这是我尝试安装的软件包列表。
conda install -y -q bokeh==1.0.0 flask arrow==0.12.1 PyYAML==3.12 cryptography==2.2.2
--if os type is linux
conda install -y -q flask-cors gunicorn
conda install -q -c conda-forge -y findspark
conda install -y -q pytest coverage pytest-cov snakeviz nose=1.3.7 Flake8==3.7.9 ipython future
conda install -y -c conda-forge -q lapack==3.6.1 cvxopt==1.2.0
conda install -q -y numpy==1.14.3
conda install -q -y scipy==1.1.0
conda install -q -y ecos==2.0.5
conda install -q -y -c cvxgrp cvxpy==1.0.6
conda install -q -y -c conda-forge pyspark==2.3.1
conda install -q -y -c anaconda mysql-connector-python==8.0.13
尝试安装pyspark时出现问题。当我尝试首先安装 pyspark 然后安装其他软件包时，我遇到了 numpy 和 scipy 无法满足的错误。需要帮助
解决方法
我设法在全新的 3.7 环境中使用相同的命令安装了它。我认为您应该检查您的 conda 版本并使用更新的版本（4.9.2 是可用的最新版本）。
conda --version
# conda 4.9.2
对于现有环境，您可以尝试 clean 缓存，因为我在日志中看到 conda 正在使用可能包含不正确信息的索引缓存。
# clean just the index as suggested my @merv
conda clean --index-cache
# clean all the cache if needed
conda clean --all
你也可以选择为pyspark新建一个全新的环境，这里有一个例子。
# create a new env
conda create -y -n pyspark pyspark=2.3.1
# activate it
conda activate pyspark
# check if pyspark is installed
conda list pyspark
# packages in environment at /opt/conda/envs/pyspark:
#
# Name Version Build Channel
# pyspark 2.3.1 py37_1001 conda-forge