The Partition QOS will override the job's QOS, but slurm doesn't make a differrence between partition or user QOS. The REASON for a pending job shows QOSMaxNodePerUserLimit . I am running slurm 16.05.10-2. Partition preempt is set up with QOS=unlimited set with a max of 960 nodes. The user's jobs run on 877 nodes so the next job would not fit in the partition limit. A reason of QOSMaxNodePerPartitionLimit would clarify why the isn't started and not mislead to check user QOS double for problems. The partiton QOS does not show up in the squeue command. JOBID EXEC_HOST DEPEND ACCOUNT PARTITION QOS ST USER RESERVATION TIME_LEFT NODE NODELIST(REASON) 5397 n/a hpc preempt double PD dummy (null) 7-00:00:00 128 (QOSMaxNodePerUserLimit ) PartitionName=preempt AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL AllocNodes=ALL Default=NO QoS=unlimit User QOS: Name Priority GraceTime Preempt PreemptMode Flags UsageThres UsageFactor GrpTRES GrpTRESMins GrpTRESRunMin GrpJobs GrpSubmit GrpWall MaxTRES MaxTRESPerNode MaxTRESMins MaxWall MaxTRESPU MaxJobsPU MaxSubmitPU MaxTRESPA MaxJobsPA MaxSubmitPA MinTRES ---------- ---------- ---------- ---------- ----------- ---------------------------------------- ---------- ----------- ------------- ------------- ------------- ------- --------- ----------- ------------- -------------- ------------- ----------- ------------- --------- ----------- ------------- --------- ----------- ------------- unlimit 0 00:00:00 cluster 1.000000 node=960 double 0 00:00:00 cluster 1.000000 node=512 Is this fixed in a newer version of slurm?ThanksFred
