运行Python作业时，如何让condor自动导入conda环境？

问题描述

我正在将作业提交给神鹰，但它说未安装tensorboard，这是错误的，因为我在一个交互式作业中遇到了，所以它被安装了。

我如何让秃鹰使用我当前的活动conda环境？

我的秃鹰提交脚本：

####################
#
# Experiments script
# Simple HTCondor submit description file
#
# reference: https://gitlab.engr.illinois.edu/Vision/vision-gpu-servers/-/wikis/HTCondor-user-guide#submit-jobs
#
# chmod a+x test_condor.py
# chmod a+x experiments_meta_model_optimization.py
# chmod a+x meta_learning_experiments_submission.py
# chmod a+x download_miniImagenet.py
#
# condor_submit -i
# condor_submit job.sub
#
####################

# Executable   = meta_learning_experiments_submission.py
# Executable = automl-proj/experiments/meta_learning/meta_learning_experiments_submission.py
# Executable = ~/automl-meta-learning/automl-proj/experiments/meta_learning/meta_learning_experiments_submission.py
Executable = /home/miranda9/automl-meta-learning/automl-proj/experiments/meta_learning/meta_learning_experiments_submission.py

## Output Files
Log          = condor_job.$(CLUSTER).log.out
Output       = condor_job.$(CLUSTER).stdout.out
Error        = condor_job.$(CLUSTER).err.out

# Use this to make sure 1 gpu is available. The key words are case insensitive.
REquest_gpus = 1
# requirements = ((CUDADeviceName = "Tesla K40m")) && (TARGET.Arch == "X86_64") && (TARGET.OpSys == "LINUX") && (TARGET.Disk >= RequestDisk) && (TARGET.Memory >= RequestMemory) && (TARGET.Cpus >= RequestCpus) && (TARGET.gpus >= Requestgpus) && ((TARGET.FileSystemDomain == MY.FileSystemDomain) || (TARGET.HasFileTransfer))
# requirements = (CUDADeviceName == "Tesla K40m")
# requirements = (CUDADeviceName == "Quadro RTX 6000")
requirements = (CUDADeviceName != "Tesla K40m")

# Note: to use multiple CPUs instead of the default (one CPU),use request_cpus as well
Request_cpus = 8

# E-mail option
Notify_user = [e-mail]
Notification = always

Environment = MY_CONDOR_JOB_ID= $(CLUSTER)

# "Queue" means add the setup until this line to the queue (needs to be at the end of script).
Queue

我的提交脚本的前几行，直到失败行：

#!/home/miranda9/.conda/bin/python3.7

import torch
import torch.nn as nn
import torch.optim as optim
# import torch.functional as F
from torch.utils.tensorboard import SummaryWriter

---

相关评论：

我确实看到了这个问题how to run a python program on Condor?和这个http://chtc.cs.wisc.edu/python-jobs.shtml，但我不敢相信我们必须这样做。集群中的其他所有人都没有做任何复杂的事情，我已经运行了我的脚本，而不必做任何复杂的事情，我非常对此表示怀疑。 >

解决方法

HTCondor在交互式和批处理作业中使用不同的默认环境。交互式作业复制与登录会话相同的外壳环境（包括激活的conda环境）。批处理作业从非常精简的环境开始（要查看实际情况，请尝试以/usr/bin/env作为可执行文件运行测试作业）；激活的conda环境将不会继续进入批处理作业环境。

此行为和潜在的提交文件解决方案在HTCondor手册中进行了描述：https://htcondor.readthedocs.io/en/latest/users-manual/services-for-jobs.html?highlight=environment#environment-variables

,

我真的不明白condor是如何工作的，但是似乎一旦我将python的正确路径放在当前环境的顶部，它便开始工作。因此，请检查您的python命令在哪里：

(automl-meta-learning) miranda9~/automl-meta-learning $ which python
~/miniconda3/envs/automl-meta-learning/bin/python

然后将其复制粘贴到python提交脚本的顶部：

#!/home/miranda9/miniconda3/envs/automl-meta-learning/bin/python

我希望我可以将所有这些内容都包含在job.sub中。如果您知道如何，请告诉我。

---

参考解决方案：https://stackoverflow.com/a/64484025/1601580