问题描述
我的gpucompute节点上有一个down状态，无法在GPU节点上发送作业。 遵循网络上的所有解决方案后，我无法返回“ down GPU”节点。在发生此问题之前，我通过运行“ NVIDIA-Linux-x86_64-410.79.run --no”解决了该错误，从而导致Nvidia驱动程序配置出现错误，导致无法通过“ nvidia-smi”检测到GPU。 -drm'我遇到此错误是由于节点的关闭状态。非常感谢您的帮助！
command: sbatch md1.s
sbatch: error: Batch job submission Failed: Requested node configuration is not available
command: sinfo
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
gpucompute* up infinite 1 down* fwb-lab-tesla1
command: sinfo -R
REASON USER TIMESTAMP NODELIST
Not responding slurm 2020-09-25T13:13:19 fwb-lab-tesla1
command: sinfo -Nl
Fri Sep 25 16:35:25 2020
NODELIST NODES PARTITION STATE cpuS S:C:T MEMORY TMP_disK WEIGHT AVAIL_FE REASON
fwb-lab-tesla1 1 gpucompute* down* 32 32:1:1 64000 0 1 (null)Not responding
command: vim /etc/slurm/slurm.conf
# slurm.conf file generated by configurator easy.html.
# Put this file on all nodes of your cluster.
# See the slurm.conf man page for more @R_328_4045@ion.
#
ControlMachine=FWB-Lab-Tesla
#ControlAddr=137.72.38.102
#
MailProg=/bin/mail
MpiDefault=none
#MpiParams=ports=#-#
ProctrackType=proctrack/cgroup
ReturnToService=1
SlurmctldPidFile=/var/run/slurmctld.pid
#SlurmctldPort=6817
SlurmdPidFile=/var/run/slurmd.pid
#SlurmdPort=6818
SlurmdspoolDir=/var/spool/slurmd
#SlurmUser=slurm
SlurmdUser=root
StateSaveLocation=/var/spool/slurm/StateSave
SwitchType=switch/none
TaskPlugin=task/cgroup
#
#
# TIMERS
#KillWait=30
command: ls /etc/init.d
functions livesys livesys-late netconsole network README
command: nvidia-smi
Fri Sep 25 16:35:01 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.79 Driver Version: 410.79 CUDA Version: 10.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 TITAN V Off | 00000000:02:00.0 Off | N/A |
| 24% 32C P8 N/A / N/A | 0MiB / 12036MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 TITAN V Off | 00000000:03:00.0 Off | N/A |
| 23% 35C P8 N/A / N/A | 0MiB / 12036MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 2 TITAN V Off | 00000000:83:00.0 Off | N/A |
| 30% 44C P8 N/A / N/A | 0MiB / 12036MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 3 TITAN V Off | 00000000:84:00.0 Off | N/A |
| 31% 42C P8 N/A / N/A | 0MiB / 12036MiB | 6% Default |
---------------------------------------------------------------------------+
----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
解决方法
您提到的问题可能阻止了slurmd上的gpucompute守护程序启动。您应该能够通过运行systemctl status slurmd或Linux发行版的等效命令来确认这一点。
slurmd日志可能包含类似于
slurmd[1234]: fatal: can't stat gres.conf file /dev/nvidia0: No such file or directory
尝试使用重新启动
systemctl start slurmd
一旦您确定nvidia-smi正确响应。
以下说明解决了我的问题。请记住，重新启动系统后，您需要在重新启动后输入命令。感谢琼·布莱恩（Joan Bryan）解决了这个问题！
var hashStr = "INDiTF0Nq8hU6d4fHqu1dQ==";
var hash = Convert.FromBase64String( hashStr );
var sb = new StringBuilder();
foreach( var b in hash )
{
sb.Append( b.ToString("X2") );
}
sb.ToString().Dump();