I think the problem might be that you are not requesting memory, so by default, all memory on a node is allocated to the job and "cons_res" will not allocate a second job to any node. That comes up quite often.
Gareth -----Original Message----- From: slurm-users <slurm-users-boun...@lists.schedmd.com> On Behalf Of Guillaume De Nayer Sent: Thursday, 16 June 2022 1:37 AM To: slurm-users@lists.schedmd.com Subject: Re: [slurm-users] Multiple Program Runs using srun in one Slurm batch Job on one node On 06/15/2022 05:25 PM, Ward Poelmans wrote: > Hi Guillaume, > > On 15/06/2022 16:59, Guillaume De Nayer wrote: >> >> Perhaps I missunderstand the Slurm documentation... >> >> As thought that the --exclusive option used in combination with >> sbatch will reserve the whole node (40 cores) for the job (submitted >> with sbatch). This part is working fine. I can check it with sacct. >> >> Then, this job starts subtasks on the reserved 40 cores with srun. >> Therefore I'm using "-n1 -c1" in combination with "srun". I thought >> that it was possible to use the reserved cores inside this job using srun. > > You're correct. --exclusive will give you all cores on the nodes but > only as much memory as requested. > > >> The following slightly modified job without --exclusive and with >> --ntasks=2 leads to a similar problem: Only one srun is running at a >> time. The second starts directly after the first one finished. >> >> #!/bin/bash >> #SBATCH --job-name=test_multi_prog_srun #SBATCH --ntasks=2 #SBATCH >> --partition=short #SBATCH --time=02:00:00 >> >> srun -vvv --exact -n1 -c1 sleep 20 > srun1.log 2>&1 & srun -vvv >> --exact -n1 -c1 sleep 30 > srun2.log 2>&1 & wait > > This should work... It works on our cluster. Are you sure they don't > run in parallel? > Yes I'm pretty sure that it does not work in parallel: The command sacct show me only on subtask "RUNNING". Then, when this subtask is marked as "COMPLETED", the second one appears and is marked "RUNNING". Moreover, if I directly connect on the node, only one process of "sleep" is running. ok. If it works on your cluster, I have perhaps a problem in my slurm config. Which version of slurm are you using on your cluster? And can you share your slurm.conf? > We usually recommend to use gnu parallel or xargs like: > > xargs -P $SLURM_NTASKS srun -N 1 -n 1 -c 1 --exact sleep 30 > ok. I will install "gnu parallel" and also test your xargs command. Thx a lot! Guillaume
