问题描述
我正在尝试对数据块使用cudf。
我开始关注https://medium.com/rapids-ai/rapids-can-now-be-accessed-on-databricks-unified-analytics-platform-666e42284bd1。但是初始化脚本链接已损坏。
然后,我点击了此链接(https://github.com/rapidsai/spark-examples/blob/master/getting-started-guides/csp/databricks/databricks.md#start-a-databricks-cluster),该链接将cudf jar安装在群集上。仍然我无法import cudf
。
我也尝试过:
%sh conda install -c rapidsai -c nvidia -c numba -c conda-forge cudf=0.13 python=3.7 cudatoolkit=10.1
也失败了,但出现了一个长错误,结尾为:
active environment : /databricks/python
active env location : /databricks/python
shell level : 2
user config file : /root/.condarc
populated config files : /databricks/conda/.condarc
conda version : 4.8.2
conda-build version : not installed
python version : 3.7.6.final.0
virtual packages : __cuda=10.2
__glibc=2.27
base environment : /databricks/conda (writable)
channel URLs : https://conda.anaconda.org/nvidia/linux-64
https://conda.anaconda.org/nvidia/noarch
https://conda.anaconda.org/rapidsai/linux-64
https://conda.anaconda.org/rapidsai/noarch
https://conda.anaconda.org/numba/linux-64
https://conda.anaconda.org/numba/noarch
https://conda.anaconda.org/conda-forge/linux-64
https://conda.anaconda.org/conda-forge/noarch
https://conda.anaconda.org/pytorch/linux-64
https://conda.anaconda.org/pytorch/noarch
https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
package cache : /databricks/python/pkgs
/local_disk0/conda/pkgs
envs directories : /databricks/conda/envs
/root/.conda/envs
platform : linux-64
user-agent : conda/4.8.2 requests/2.22.0 CPython/3.7.6 Linux/4.4.0-1114-aws ubuntu/18.04.5 glibc/2.27
UID:GID : 0:0
netrc file : None
offline mode : False
An unexpected error has occurred. Conda has prepared the above report.
Upload successful.
有什么想法如何在数据砖集群上使用cudf
吗?
解决方法
我记得前一段时间曾帮助写过该博客:)。现在已经过时了。
自spark-rapids
起,Karthik和团队进行了一些重大更新。这是带有数据砖的RAPID的最新实现:https://nvidia.github.io/spark-rapids/docs/get-started/getting-started-databricks.html。这样可以使您使用最新版本的Cudf。
我会要求某人在该特定博客上添加免责声明,这样其他人也不会感到困惑。感谢您通过这个问题提醒我们!
,也许您需要cudatoolkit=10.2
?您在该报告中有virtual packages : __cuda=10.2
。
我正在研究数据块GPU群集上的安装问题(尽管存在其他问题),并指出CUDA的版本是10.2,而不是我期望的10.1。
,我认为 OP 想将 python 与 cudf 一起使用。 如果是这样,则文档中未涵盖该内容。
但我尝试将以下内容添加到 generate-init-script.ipynb 中以使其工作:
#Use mamba to install packages to speed up conda resolve time
base=$(conda info --base)
conda create -y -n mamba -c conda-forge mamba
pip uninstall -y pyarrow
${base}/envs/mamba/bin/mamba remove -y c-ares zstd libprotobuf pandas
${base}/envs/mamba/bin/mamba install -y "pyarrow=1.0.1" -c "conda-forge"
${base}/envs/mamba/bin/mamba install -y -c "rapidsai" -c "nvidia" -c "conda-forge" -c "defaults" "cudf=0.18" "cudatoolkit=10.1"
conda env remove -n mamba
注意:根据您的环境更改cudf版本和cudatoolkit。