Re: [slurm-users] slurm.conf syntax checker?

[Paul Edmon]

Sadly no.  There is a feature request for one though: 
https://bugs.schedmd.com/show_bug.cgi?id=3435


What we've done in the meantime is put together a gitlab runner which 
basically starts up a mini instance of the scheduler and runs slurmctld 
on the slurm.conf we want to put in place.  We then have it reject any 
changes that cause failure.  It's not perfect but it works.  A real 
syntax checker would be better.


-Paul Edmon-

On 10/12/2021 4:08 PM, bbenede...@goodyear.com wrote:

Is there any sort of syntax checker that we could run our slurm.conf file
through before committing it?  (And sometimes crashing slurmctld in the
process...)

Thanks!

Re: [slurm-users] job is pending but resources are available

[Adam Xu]

在 10/13/21 16:30, Ole Holm Nielsen 写道:

On 10/13/21 9:59 AM, Adam Xu wrote:


在 2021/10/13 9:22, Brian Andrus 写道:


Something is very odd when you have the node reporting:

RealMemory=1 AllocMem=0 FreeMem=47563 Sockets=2 Boards=1

What do you get when you run ‘slurmd -C’ on the node?


# slurmd -C
NodeName=apollo CPUs=36 Boards=1 SocketsPerBoard=2 CoresPerSocket=18 
ThreadsPerCore=1 RealMemory=128306

UpTime=22-16:14:48


Maybe try to replace "Boards=1 SocketsPerBoard=2" by "Sockets=2". The 
"Boards" used to give problems in Slurm 20.02, see 
https://wiki.fysik.dtu.dk/niflheim/Slurm_configuration#compute-node-configuration

but should be fixed in 20.11.
"NodeName=apollo Sockets=2 CoresPerSocket=18 ThreadsPerCore=1 
Gres=gpu:v100:8,mps:800 State=UNKNOWN" in my slurm.conf file.


Your node apollo has CPUAlloc=28 CPUTot=36 which is OK, since there 
are 8 available cores.


Maybe your 8 GPUs are all in use by the 7 jobs?  Can you send your 
gres.conf file defining GPUs?

# sinfo -O GresUsed
GRES_USED
gpu:0,mps:0

gpu:v100:7(IDX:0-6),

cat gres.conf

Name=gpu Type=v100 File=/dev/nvidia0
Name=gpu Type=v100 File=/dev/nvidia1
Name=gpu Type=v100 File=/dev/nvidia2
Name=gpu Type=v100 File=/dev/nvidia3
Name=gpu Type=v100 File=/dev/nvidia4
Name=gpu Type=v100 File=/dev/nvidia5
Name=gpu Type=v100 File=/dev/nvidia6
Name=gpu Type=v100 File=/dev/nvidia7
Name=mps Count=100 File=/dev/nvidia0
Name=mps Count=100 File=/dev/nvidia1
Name=mps Count=100 File=/dev/nvidia2
Name=mps Count=100 File=/dev/nvidia3
Name=mps Count=100 File=/dev/nvidia4
Name=mps Count=100 File=/dev/nvidia5
Name=mps Count=100 File=/dev/nvidia6
Name=mps Count=100 File=/dev/nvidia7



To view the node apollo jobs, I recommend to download the "pestat" 
command from 
https://github.com/OleHolmNielsen/Slurm_tools/tree/master/pestat


Just type "pestat -G" to view the GRES resources for each job on all 
nodes.

Ok, I will try it later.


Best regards,
Ole




*From: *Adam Xu 
*Sent: *Tuesday, October 12, 2021 6:07 PM
*To: *slurm-users@lists.schedmd.com
*Subject: *Re: [slurm-users] job is pending but resources are available

在 2021/10/12 21:21, Adam Xu 写道:

    Hi All,

    OS: Rocky Linux 8.4

    slurm version: 20.11.7

    the partition's name is apollo. the node's name is apollo too. the
    node has 36 cpu cores and 8GPUs in it.

    partition info

    $ scontrol show partition apollo
    PartitionName=apollo
       AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
       AllocNodes=ALL Default=NO QoS=N/A
       DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0
    Hidden=NO
       MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO
    MaxCPUsPerNode=UNLIMITED
       Nodes=apollo
       PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO
    OverSubscribe=YES:36
       OverTimeLimit=NONE PreemptMode=OFF
       State=UP TotalCPUs=36 TotalNodes=1 SelectTypeParameters=NONE
       JobDefaults=(null)
       DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED

    node info

    $ scontrol show node apollo
    NodeName=apollo Arch=x86_64 CoresPerSocket=18
       CPUAlloc=28 CPUTot=36 CPULoad=7.02
       AvailableFeatures=(null)
       ActiveFeatures=(null)
       Gres=gpu:v100:8,mps:v100:800
       NodeAddr=apollo NodeHostName=apollo Version=20.11.7
       OS=Linux 4.18.0-305.19.1.el8_4.x86_64 #1 SMP Wed Sep 15 19:12:32
    UTC 2021
       RealMemory=1 AllocMem=0 FreeMem=47563 Sockets=2 Boards=1
       State=MIXED ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A
    MCS_label=N/A
       Partitions=apollo
       BootTime=2021-09-20T23:43:49 SlurmdStartTime=2021-10-12T16:55:44
       CfgTRES=cpu=36,mem=1M,billing=36
       AllocTRES=cpu=28
       CapWatts=n/a
       CurrentWatts=0 AveWatts=0
       ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s
       Comment=(null)

    Now I have 7 jobs running but when I submit 8th job, the status of
    the job is pending beacuse Resources.

    $ squeue
     JOBID PARTITION NAME USER ST TIME NODES
    NODELIST(REASON)
       879    apollo    do.sh zhining_ PD 0:00 1
    (Resources)
       489    apollo    do.sh zhining_  R 13-12:50:45 1 
apollo
       490    apollo    do.sh zhining_  R 13-12:41:00 1 
apollo
       592    apollo runme-gp junwen_f  R 4-12:42:31 1 
apollo
       751    apollo runme-gp junwen_f  R 1-12:48:20 1 
apollo
       752    apollo runme-gp junwen_f  R 1-12:48:10 1 
apollo

       871    apollo runme-gp junwen_f  R 7:13:45 1 apollo
       872    apollo runme-gp junwen_f  R 7:12:42 1 apollo

    $ scontrol show job 879
    JobId=879 JobName=do.sh
       UserId=zhining_wan(1001) GroupId=zhining_wan(1001) MCS_label=N/A
       Priority=4294900882 Nice=0 Account=(null) QOS=(null)
       JobState=PENDING Reason=Resources Dependency=(null)
       Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
       RunTime=00:00:00 TimeLimit=UNLIMITED 

Re: [slurm-users] job is pending but resources are available

[Ole Holm Nielsen]

On 10/13/21 9:59 AM, Adam Xu wrote:


在 2021/10/13 9:22, Brian Andrus 写道:


Something is very odd when you have the node reporting:

RealMemory=1 AllocMem=0 FreeMem=47563 Sockets=2 Boards=1

What do you get when you run ‘slurmd -C’ on the node?


# slurmd -C
NodeName=apollo CPUs=36 Boards=1 SocketsPerBoard=2 CoresPerSocket=18 
ThreadsPerCore=1 RealMemory=128306

UpTime=22-16:14:48


Maybe try to replace "Boards=1 SocketsPerBoard=2" by "Sockets=2".  The 
"Boards" used to give problems in Slurm 20.02, see 
https://wiki.fysik.dtu.dk/niflheim/Slurm_configuration#compute-node-configuration

but should be fixed in 20.11.

Your node apollo has CPUAlloc=28 CPUTot=36 which is OK, since there are 8 
available cores.


Maybe your 8 GPUs are all in use by the 7 jobs?  Can you send your 
gres.conf file defining GPUs?


To view the node apollo jobs, I recommend to download the "pestat" command 
from https://github.com/OleHolmNielsen/Slurm_tools/tree/master/pestat


Just type "pestat -G" to view the GRES resources for each job on all nodes.

Best regards,
Ole




*From: *Adam Xu 
*Sent: *Tuesday, October 12, 2021 6:07 PM
*To: *slurm-users@lists.schedmd.com
*Subject: *Re: [slurm-users] job is pending but resources are available

在 2021/10/12 21:21, Adam Xu 写道:

Hi All,

OS: Rocky Linux 8.4

slurm version: 20.11.7

the partition's name is apollo. the node's name is apollo too. the
node has 36 cpu cores and 8GPUs in it.

partition info

$ scontrol show partition apollo
PartitionName=apollo
   AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
   AllocNodes=ALL Default=NO QoS=N/A
   DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0
Hidden=NO
   MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO
MaxCPUsPerNode=UNLIMITED
   Nodes=apollo
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO
OverSubscribe=YES:36
   OverTimeLimit=NONE PreemptMode=OFF
   State=UP TotalCPUs=36 TotalNodes=1 SelectTypeParameters=NONE
   JobDefaults=(null)
   DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED

node info

$ scontrol show node apollo
NodeName=apollo Arch=x86_64 CoresPerSocket=18
   CPUAlloc=28 CPUTot=36 CPULoad=7.02
   AvailableFeatures=(null)
   ActiveFeatures=(null)
   Gres=gpu:v100:8,mps:v100:800
   NodeAddr=apollo NodeHostName=apollo Version=20.11.7
   OS=Linux 4.18.0-305.19.1.el8_4.x86_64 #1 SMP Wed Sep 15 19:12:32
UTC 2021
   RealMemory=1 AllocMem=0 FreeMem=47563 Sockets=2 Boards=1
   State=MIXED ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A
MCS_label=N/A
   Partitions=apollo
   BootTime=2021-09-20T23:43:49 SlurmdStartTime=2021-10-12T16:55:44
   CfgTRES=cpu=36,mem=1M,billing=36
   AllocTRES=cpu=28
   CapWatts=n/a
   CurrentWatts=0 AveWatts=0
   ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s
   Comment=(null)

Now I have 7 jobs running but when I submit 8th job, the status of
the job is pending beacuse Resources.

$ squeue
 JOBID PARTITION NAME USER ST   TIME NODES
NODELIST(REASON)
   879    apollo    do.sh zhining_ PD   0:00 1
(Resources)
   489    apollo    do.sh zhining_  R 13-12:50:45 1 apollo
   490    apollo    do.sh zhining_  R 13-12:41:00 1 apollo
   592    apollo runme-gp junwen_f  R 4-12:42:31 1 apollo
   751    apollo runme-gp junwen_f  R 1-12:48:20 1 apollo
   752    apollo runme-gp junwen_f  R 1-12:48:10 1 apollo
   871    apollo runme-gp junwen_f  R    7:13:45 1 apollo
   872    apollo runme-gp junwen_f  R    7:12:42 1 apollo

$ scontrol show job 879
JobId=879 JobName=do.sh
   UserId=zhining_wan(1001) GroupId=zhining_wan(1001) MCS_label=N/A
   Priority=4294900882 Nice=0 Account=(null) QOS=(null)
   JobState=PENDING Reason=Resources Dependency=(null)
   Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
   RunTime=00:00:00 TimeLimit=UNLIMITED TimeMin=N/A
   SubmitTime=2021-10-12T16:29:29 EligibleTime=2021-10-12T16:29:29
   AccrueTime=2021-10-12T16:29:29
   StartTime=2021-10-12T21:17:41 EndTime=Unknown Deadline=N/A
   SuspendTime=None SecsPreSuspend=0 LastSchedEval=2021-10-12T21:17:39
   Partition=apollo AllocNode:Sid=sms:1281191
   ReqNodeList=(null) ExcNodeList=(null)
   NodeList=(null) SchedNodeList=apollo
   NumNodes=1-1 NumCPUs=4 NumTasks=4 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
   TRES=cpu=4,node=1,billing=4
   Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
   MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0
   Features=(null) DelayBoot=00:00:00
   OverSubscribe=YES Contiguous=0 Licenses=(null) Network=(null)

Command=/home/zhining_wan/job/2021/20210603_ctla4_double_bilayer/final_pdb_minimize/amber/nolipid/test/do.sh

Re: [slurm-users] job is pending but resources are available

[Adam Xu]

在 2021/10/13 9:22, Brian Andrus 写道:


Something is very odd when you have the node reporting:

RealMemory=1 AllocMem=0 FreeMem=47563 Sockets=2 Boards=1

What do you get when you run ‘slurmd -C’ on the node?


# slurmd -C
NodeName=apollo CPUs=36 Boards=1 SocketsPerBoard=2 CoresPerSocket=18 
ThreadsPerCore=1 RealMemory=128306

UpTime=22-16:14:48


Brian Andrus

*From: *Adam Xu 
*Sent: *Tuesday, October 12, 2021 6:07 PM
*To: *slurm-users@lists.schedmd.com
*Subject: *Re: [slurm-users] job is pending but resources are available

在 2021/10/12 21:21, Adam Xu 写道:

Hi All,

OS: Rocky Linux 8.4

slurm version: 20.11.7

the partition's name is apollo. the node's name is apollo too. the
node has 36 cpu cores and 8GPUs in it.

partition info

$ scontrol show partition apollo
PartitionName=apollo
   AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
   AllocNodes=ALL Default=NO QoS=N/A
   DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO
GraceTime=0 Hidden=NO
   MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO
MaxCPUsPerNode=UNLIMITED
   Nodes=apollo
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO
OverSubscribe=YES:36
   OverTimeLimit=NONE PreemptMode=OFF
   State=UP TotalCPUs=36 TotalNodes=1 SelectTypeParameters=NONE
   JobDefaults=(null)
   DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED

node info

$ scontrol show node apollo
NodeName=apollo Arch=x86_64 CoresPerSocket=18
   CPUAlloc=28 CPUTot=36 CPULoad=7.02
   AvailableFeatures=(null)
   ActiveFeatures=(null)
   Gres=gpu:v100:8,mps:v100:800
   NodeAddr=apollo NodeHostName=apollo Version=20.11.7
   OS=Linux 4.18.0-305.19.1.el8_4.x86_64 #1 SMP Wed Sep 15
19:12:32 UTC 2021
   RealMemory=1 AllocMem=0 FreeMem=47563 Sockets=2 Boards=1
   State=MIXED ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A
MCS_label=N/A
   Partitions=apollo
   BootTime=2021-09-20T23:43:49 SlurmdStartTime=2021-10-12T16:55:44
   CfgTRES=cpu=36,mem=1M,billing=36
   AllocTRES=cpu=28
   CapWatts=n/a
   CurrentWatts=0 AveWatts=0
   ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s
   Comment=(null)

Now I have 7 jobs running but when I submit 8th job, the status of
the job is pending beacuse Resources.

$ squeue
 JOBID PARTITION NAME USER ST   TIME NODES
NODELIST(REASON)
   879    apollo    do.sh zhining_ PD   0:00 1
(Resources)
   489    apollo    do.sh zhining_  R 13-12:50:45 1
apollo
   490    apollo    do.sh zhining_  R 13-12:41:00 1
apollo
   592    apollo runme-gp junwen_f  R 4-12:42:31 1 apollo
   751    apollo runme-gp junwen_f  R 1-12:48:20 1 apollo
   752    apollo runme-gp junwen_f  R 1-12:48:10 1 apollo
   871    apollo runme-gp junwen_f  R    7:13:45 1 apollo
   872    apollo runme-gp junwen_f  R    7:12:42 1 apollo

$ scontrol show job 879
JobId=879 JobName=do.sh
   UserId=zhining_wan(1001) GroupId=zhining_wan(1001) MCS_label=N/A
   Priority=4294900882 Nice=0 Account=(null) QOS=(null)
   JobState=PENDING Reason=Resources Dependency=(null)
   Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
   RunTime=00:00:00 TimeLimit=UNLIMITED TimeMin=N/A
   SubmitTime=2021-10-12T16:29:29 EligibleTime=2021-10-12T16:29:29
   AccrueTime=2021-10-12T16:29:29
   StartTime=2021-10-12T21:17:41 EndTime=Unknown Deadline=N/A
   SuspendTime=None SecsPreSuspend=0
LastSchedEval=2021-10-12T21:17:39
   Partition=apollo AllocNode:Sid=sms:1281191
   ReqNodeList=(null) ExcNodeList=(null)
   NodeList=(null) SchedNodeList=apollo
   NumNodes=1-1 NumCPUs=4 NumTasks=4 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
   TRES=cpu=4,node=1,billing=4
   Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
   MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0
   Features=(null) DelayBoot=00:00:00
   OverSubscribe=YES Contiguous=0 Licenses=(null) Network=(null)

Command=/home/zhining_wan/job/2021/20210603_ctla4_double_bilayer/final_pdb_minimize/amber/nolipid/test/do.sh


WorkDir=/home/zhining_wan/job/2021/20210603_ctla4_double_bilayer/final_pdb_minimize/amber/nolipid/test


StdErr=/home/zhining_wan/job/2021/20210603_ctla4_double_bilayer/final_pdb_minimize/amber/nolipid/test/slurm-879.out

   StdIn=/dev/null

StdOut=/home/zhining_wan/job/2021/20210603_ctla4_double_bilayer/final_pdb_minimize/amber/nolipid/test/slurm-879.out

   Power=
   TresPerNode=gpu:1
   NtasksPerTRES:0

After running 7 jobs, the node has 8 cpu cores and 1 gpu left, so
I can be sure that the remaining resources are sufficient. but why
the job is pending with reason "Resources"?

Some information to add：

I have killed some jobs with kill instead of