如何使用PBS Scheduler将python用作提交脚本？

问题描述

我过去使用python作为提交脚本，而不是bash等。在slurm conder中，我通常在顶部指示编译器的路径。我在qsub中尝试过，但是没有用。知道如何使它起作用吗？

代码：

#/homes/miranda9/.conda/envs/myenv/lib/python3.7
#PBS -V
#PBS -M [e-mail]
#PBS -m abe

import sys

for p in sys.path:
    print(p)

那我做了：

qsub test_qsub.py

错误消息：

$ cat test_qsub.py.e381299 
/homes/miranda9/.profile: line 2: /opt/intel/compilers_and_libraries_2017/linux/mpi/intel64/bin/mpivars.sh: No such file or directory
/homes/miranda9/.profile: line 3: /opt/intel/compilers_and_libraries_2017/linux/bin/compilervars.sh: No such file or directory
import: unable to open X server `' @ error/import.c/ImportImageCommand/369.
/var/spool/pbs/mom_priv/jobs/381299.iam-pbs.SC: line 9: syntax error near unexpected token `print'
/var/spool/pbs/mom_priv/jobs/381299.iam-pbs.SC: line 9: `    print(p)'

---

在我添加!并在顶部制作了“＃！/ homes / miranda9 / .conda / envs / automl-meta-learning / lib / python3.7”之后，我得到了一个新错误：

$ cat test_qsub.py.e381301 
/homes/miranda9/.profile: line 2: /opt/intel/compilers_and_libraries_2017/linux/mpi/intel64/bin/mpivars.sh: No such file or directory
/homes/miranda9/.profile: line 3: /opt/intel/compilers_and_libraries_2017/linux/bin/compilervars.sh: No such file or directory
-bash: /var/spool/pbs/mom_priv/jobs/381301.iam-pbs.SC: /homes/miranda9/.conda/envs/automl-meta-learning/lib/python3.7: bad interpreter: Permission denied

---

交叉张贴：

• https://community.openpbs.org/t/how-does-one-use-python-as-a-submission-script-using-the-pbs-scheduler/2271
• How does one use python as a submission script using the PBS scheduler?
• https://qr.ae/pNFW6P
• https://www.reddit.com/r/HPC/comments/issqbu/how_does_one_use_python_as_a_submission_script/

解决方法

首先不要忘记顶部的!。

诀窍是弄清楚python的二进制文件在哪里（例如，使用conda环境），并在顶部告诉PBS您的提交脚本。我尝试使用sys.path，但这会导致PBS无法使用的文件夹的路径。我必须找出sys.executable才能找到它。然后我把它放在顶部。

演示会话以找到它：

$ python
Python 3.7.7 (default,Mar 26 2020,15:48:22) 
[GCC 7.3.0] :: Anaconda,Inc. on linux
Type "help","copyright","credits" or "license" for more information.
>>> import sys
>>> sys.executable
'/homes/miranda9/.conda/envs/myenv/bin/python'

然后用感叹号正确放置：

#!/homes/miranda9/.conda/envs/YOURENV/bin/python
#PBS -V
#PBS -M [e-mail]
#PBS -m abe
#PBS -lselect=1:ncpus=112

import sys
import os

for p in sys.path:
    print(p)

print(os.environ)

现在可以工作！