循环中使用的srun：srun：作业步骤中止：等待32秒以完成作业步骤

问题描述

我有一个{.1}运行的.sh文件，因为我想查看脚本的动态打印输出。但是通过运行srun，我总是在2个主循环之后出现错误（由于时间限制错误而超时）...这是.sh文件中的主要代码。顺便说一句，我想运行一个12个月的模型并将其循环20次（所谓的加速20次）。但是错误发生在第二个循环的11月（加速）中... 这是job_spinup.sh中的代码：

srun job_spinup.sh southfr_exp 1 &

输出如下：

#!/bin/bash
#SBATCH -J spinup
#SBATCH -p knl_cache
#SBATCH -N 1
#SBATCH -n 1
#SBATCH -t 10:00:00
#SBATCH -o spinup.log
#SBATCH -e spinup.log
#=========================================================================
# USAGE
#   nohup ./job_spinup DOM[:EXP] nodes[:tasks_per_node:tasks_for_trip N START_ID:START_MM] &
#
# by default: EXP=spinup,N=20,START_ID=0,START_MM=1
#=========================================================================
#set -x
#
if [ $# -lt 2 ]; then
  echo "Usage: $0 DOM[:EXP:VERSION] nodes[:tasks_per_node:tasks_for_trip N START_ID:START_MM]"
  echo "DOM            = the name of a domain"
  echo "EXP            = the name of an experiment"
  echo "N              = the number of runnings"
  echo "START_ID       = start id of a running"
  echo "START_MM       = start month of a running"
  exit
fi

DOM=`echo $1 | awk '{split($1,f,":"); print f[1]}'`
EXP=`echo $1 | awk '{split($1,":"); print f[2]}'`
EXP=${EXP:-spinup}
VERSION=`echo $1 | awk '{split($1,":"); print f[3]}'`
VERSION=${VERSION:--X0}
num_nodes=`echo ${2} | awk '{split($1,":"); print f[1]}'`
tasks_per_node=`echo ${2} | awk '{split($1,":"); print f[2]}'`
tasks_per_node=${tasks_per_node:-40}
tasks_for_trip=`echo ${2} | awk '{split($1,":"); print f[3]}'`
tasks_for_trip=${tasks_for_trip:-1}
SPINUP_N=${3:-20}
START_ID=`echo $4 | awk '{split($1,":"); print f[1]}'`
START_ID=${START_ID:-0}
START_MM=`echo $4 | awk '{split($1,":"); print f[2]}'`
START_MM=${START_MM:-1}

# source ~/anaconda3/etc/profile.d/conda.sh
source $(conda info --base)/etc/profile.d/conda.sh
conda activate myenv
echo "***************************************"
echo " CONDA ENV ACTIVATED FOR NCO COMMAND"
echo "***************************************"
echo $SPINUP_N
#
# check if TRIP is used
LTRIP=`grep "Loasis *= *T" OPTIONS/OPTIONS.nam | wc -l`
#
ulimit -s unlimited
ulimit -n 500000
ulimit -u 64000
unset I_MPI_PMI_LIBRARY
export OMP_NUM_THREADS=1
export DR_HOOK=0
export DR_HOOK_OPT=prof

...

YYYY=${YYYYMMDDHH::4}
MM=${YYYYMMDDHH:4:2}
j=$start_ID
while [ $j -lt $SPINUP_N ] ; do

  echo " "
  echo "------------------"
  echo "SPINUP : $j / $SPINUP_N"

  while [ $MM -le 12 ] ; do
    if [ $LTRIP -eq 1 ]; then
      mpirun -np $((SLURM_NTASKS - tasks_for_trip)) offline.exe : -np $tasks_for_trip trip.exe &> offline
    else
      #echo ${SLURM_NTASKS}
      #mpirun -np ${SLURM_NTASKS} offline.exe &> offline
      #srun -n 1 offline.exe &> offline
      offline.exe &> offline
    fi
....

# Change dates to start again
    if [ $MM -eq 12 ]; then
      ncap2 -O -s "'DTCUR-YEAR'=$YYYY;'DTCUR-MONTH'=1;'DTCUR-DAY'=1;'DTCUR-TIME'=0" PREP.nc PREP.nc
      [ $LTRIP -eq 1 ] && ncap2 -O -s "date(:)={$YYYY,1,0}" TRIP_PREP.nc TRIP_PREP.nc
    fi

...


  done

  echo '------------------'
  echo ' '

  MM=01
  j=$(( j+1 ))

done
...
# end simulation
date >> date_$EXP
echo "***************************************"
echo "   SPINUP ENDS CORRECTLY"
echo "***************************************"

conda deactivate
echo "***************************************"
echo "   CONDA ENV DEACTIVATED"
echo "***************************************"

有人可以帮助我吗？非常感谢！我是初学者.....是因为我激活了conda环境吗？并通过挤压，我看到队列仅持续5分钟...不知道为什么...是因为offline.exe吗？

解决方法

srun不能像sbatch那样读取作业脚本。这意味着您所有的#SBATCH选项都将被忽略，包括为作业设置的时间限制。因此，您的工作将转到具有默认时间限制的默认分区，这似乎只够两个循环的时间。

有多种解决方法：

1. 使用sbatch并查看您的输出文件（tail -f spinup.log）
2. 使用sbatch并通过sattach附加到作业
3. 将#SBATCH选项作为参数添加到srun