[slurm-users] cpu-bind=MASK at output files
Hello,
Running this simple script:
#!/bin/bash
#
#SBATCH --job-name=mega_job
#SBATCH --output=mega_job.out
#SBATCH --tasks=3
#SBATCH --array=0-5
#SBATCH --partition=cuda.q
echo "STARTING"
srun echo "hello world" >> file_${SLURM_ARRAY_TASK_ID}.out
echo "ENDING"
I always get this output:
STARTING
STARTING
cpu-bind=MASK - aoclsd, task 0 0 [14072]: mask 0xc00c00 set
STARTING
cpu-bind=MASK - aoclsd, task 0 0 [14080]: mask 0xc00c set
cpu-bind=MASK - aoclsd, task 0 0 [14081]: mask 0x30030 set
cpu-bind=MASK - aoclsd, task 1 1 [14136]: mask 0xc00c0 set
STARTING
cpu-bind=MASK - aoclsd, task 0 0 [14144]: mask 0x3003 set
cpu-bind=MASK - aoclsd, task 1 1 [14145]: mask 0x3003 set
cpu-bind=MASK - aoclsd, task 2 2 [14150]: mask 0x3003 set
cpu-bind=MASK - aoclsd, task 2 2 [14158]: mask 0x300300 set
cpu-bind=MASK - aoclsd, task 2 2 [14137]: mask 0xc00c0 set
cpu-bind=MASK - aoclsd, task 0 0 [14135]: mask 0xc00c0 set
STARTING
STARTING
cpu-bind=MASK - aoclsd, task 0 0 [14156]: mask 0x300300 set
cpu-bind=MASK - aoclsd, task 1 1 [14157]: mask 0x300300 set
cpu-bind=MASK - aoclsd, task 2 2 [14175]: mask 0xc00c00 set
cpu-bind=MASK - aoclsd, task 1 1 [14174]: mask 0xc00c00 set
cpu-bind=MASK - aoclsd, task 0 0 [14173]: mask 0xc00c00 set
cpu-bind=MASK - aoclsd, task 1 1 [14197]: mask 0xc00c set
cpu-bind=MASK - aoclsd, task 0 0 [14196]: mask 0xc00c set
cpu-bind=MASK - aoclsd, task 2 2 [14198]: mask 0xc00c set
cpu-bind=MASK - aoclsd, task 0 0 [14206]: mask 0x30030 set
cpu-bind=MASK - aoclsd, task 1 1 [14207]: mask 0x30030 set
cpu-bind=MASK - aoclsd, task 2 2 [14208]: mask 0x30030 set
ENDING
ENDING
ENDING
ENDING
ENDING
ENDING
As you can see, it appears some "cpu-bind=MASK" that I would like to prevent
them from appearing in the output file. I have reviewed configuration files and
now I'm going to copy here main lines from my slurmd.conf file:
ControlMachine=my_server
ProctrackType=proctrack/linuxproc
AuthType=auth/munge
SwitchType=switch/none
TaskPlugin=task/none,task/affinity,task/cgroup
TaskPluginParam=none
DebugFlags=NO_CONF_HASH,BackfillMap,SelectType,Steps,TraceJobs
PropagateResourceLimits=ALL
FastSchedule=1
SchedulerType=sched/backfill
SelectType=select/cons_res
SelectTypeParameters=CR_Core
AccountingStorageType=accounting_storage/slurmdbd
JobCompType=jobcomp/filetxt
JobCompLoc=/var/log/slurm/job_completions
JobAcctGatherType=jobacct_gather/linux
SlurmctldDebug=info
SlurmctldLogFile=/var/log/slurmctld.log
SlurmdDebug=info
SlurmdLogFile=/var/log/slurmd.log
AccountingStorageEnforce=associations,limits,qos
AccountingStorageHost=my_server
AccountingStorageLoc=/var/log/slurm/slurm_job_accounting.txt
GresTypes=gpu
KillOnBadExit=1
OverTimeLimit=2
TCPTimeout=5
I would appreciate any help in the future.
Thanks.
[slurm-users] Unconfigured GPUs being allocated
Hi, I manually configure the GPUs in our Slurm configuration (AutoDetect=off in gres.conf) and everything works fine when all the GPUs in a node are configured in gres.conf and available to Slurm. But we have some nodes where a GPU is reserved for running the display and is specifically not configured in gres.conf. In these cases, Slurm includes this unconfigured GPU and makes it available to Slurm jobs. Using a simple Slurm job that executes "nvidia-smi -L", it will display the unconfigured GPU along with as many configured GPUs as requested by the job. For example, in a node configured with this line in slurm.conf: NodeName=oryx CoreSpecCount=2 CPUs=8 RealMemory=64000 Gres=gpu:RTX2080TI:1 and this line in gres.conf: Nodename=oryx Name=gpu Type=RTX2080TI File=/dev/nvidia1 I will get the following results from a job running "nvidia-smi -L" that requested a single GPU: GPU 0: NVIDIA GeForce GT 710 (UUID: GPU-21fe15f0-d8b9-b39e-8ada-8c1c8fba8a1e) GPU 1: NVIDIA GeForce RTX 2080 Ti (UUID: GPU-0dc4da58-5026-6173-1156-c4559a268bf5) But in another node that has all GPUs configured in Slurm like this in slurm.conf: NodeName=beluga CoreSpecCount=1 CPUs=16 RealMemory=128500 Gres=gpu:TITANX:2 and this line in gres.conf: Nodename=beluga Name=gpu Type=TITANX File=/dev/nvidia[0-1] I get the expected results from the job running "nvidia-smi -L" that requested a single GPU: GPU 0: NVIDIA RTX A5500 (UUID: GPU-3754c069-799e-2027-9fbb-ff90e2e8e459) I'm running Slurm 22.05.5. Thanks in advance for any suggestions to help correct this problem! Steve
Re: [slurm-users] Backfill Scheduling
> On Jun 27, 2023, at 1:10 AM, Loris Bennett wrote: > > Hi Reed, > > Reed Dier mailto:reed.d...@focusvq.com>> writes: > >> Is this an issue with the relative FIFO nature of the priority scheduling >> currently with all of the other factors disabled, >> or since my queue is fairly deep, is this due to bf_max_job_test being >> the default 100, and it can’t look deep enough into the queue to find >> a job that will fit into what is unoccupied? > > It could be that bf_max_job_test is too low. On our system some users > think it is a good idea to submit lots of jobs with identical resource > requirements by writing a loop around sbatch. Such jobs will exhaust > the bf_max_job_test very quickly. Thus we increased the limit to 1000 > and try to persuade users to use job arrays instead of home-grown loops. > This seem to work OK[1]. > > Cheers, > > Loris > > -- > Dr. Loris Bennett (Herr/Mr) > ZEDAT, Freie Universität Berlin Thanks Loris, I think this will be the next knob to turn and gives a bit more confidence to that, as we too have many such identical jobs. > On Jun 26, 2023, at 9:10 PM, Brian Andrus wrote: > > Reed, > > You may want to look at the timelimit aspect of the job(s). > > For one to 'squeeze in', it needs to be able to finish before the resources > in use are expected to become available. > > Consider: > Job A is running on 2 nodes of a 3 node cluster. It will finish in 1 hour. > Pending job B will run for 2 hours needs 2 nodes, but only 1 is free, it > waits. > Pending job C (with a lower priority) needs 1 node for 2 hours. Hmm, well it > won't finish before the time job B is expected to start, so it waits. > Pending job D (with even lower priority) needs 1 node for 30 minutes. That > can squeeze in before the additional node for Job B is expected to be > available, so it runs on the idle node. > > Brian Andrus Thanks Brian, Our layout is a bit less exciting, in that none of these are >1 node per job. So the blocking out nodes for job:node Tetris isn’t really at play here. The timing however is something I may turn an eye towards. Most jobs have a “sanity” time limit applied, in that it is not so much an expected time limit, but rather an “if it goes this long, something obviously went awry and we shouldn’t keep holding on to resources” limit. So its a bit hard to quantify the timing portion, but I haven’t looked into the slurm guesses of when it thinks the next task will start, etc. The pretty simplistic example at play here is that there are nodes that are ~50-60% loaded for CPU and memory. The next job up is a “whale” job that wants a ton of resources, cpu and/or memory, but down the line there is a job with 2 cpu’s and 2 gb of memory that can easily slot in to the unused resources. So my thinking was that the job_test list may be too short to actually get that far down the queue to see that it could shove that job into some holes. I’ll report back any findings after testing Loris’s suggestions. Appreciate everyone’s help and suggestions, Reed smime.p7s Description: S/MIME cryptographic signature
Re: [slurm-users] Backfill Scheduling
Hi Reed, Reed Dier writes: > On Jun 27, 2023, at 1:10 AM, Loris Bennett > wrote: > > Hi Reed, > > Reed Dier writes: > > Is this an issue with the relative FIFO nature of the priority scheduling > currently with all of the other factors disabled, > or since my queue is fairly deep, is this due to bf_max_job_test being > the default 100, and it can’t look deep enough into the queue to find > a job that will fit into what is unoccupied? > > It could be that bf_max_job_test is too low. On our system some users > think it is a good idea to submit lots of jobs with identical resource > requirements by writing a loop around sbatch. Such jobs will exhaust > the bf_max_job_test very quickly. Thus we increased the limit to 1000 > and try to persuade users to use job arrays instead of home-grown loops. > This seem to work OK[1]. > > Cheers, > > Loris > > -- > Dr. Loris Bennett (Herr/Mr) > ZEDAT, Freie Universität Berlin > > Thanks Loris, > I think this will be the next knob to turn and gives a bit more confidence to > that, as we too have many such identical jobs. > > On Jun 26, 2023, at 9:10 PM, Brian Andrus wrote: > > Reed, > > You may want to look at the timelimit aspect of the job(s). > > For one to 'squeeze in', it needs to be able to finish before the resources > in use are expected to become available. > > Consider: > Job A is running on 2 nodes of a 3 node cluster. It will finish in 1 hour. > Pending job B will run for 2 hours needs 2 nodes, but only 1 is free, it > waits. > Pending job C (with a lower priority) needs 1 node for 2 hours. Hmm, well it > won't finish before the time job B is expected to start, so it waits. > Pending job D (with even lower priority) needs 1 node for 30 minutes. That > can squeeze in before the additional node for Job B is expected to be > available, so it runs on the idle node. > > Brian Andrus > > Thanks Brian, > > Our layout is a bit less exciting, in that none of these are >1 node per job. > So the blocking out nodes for job:node Tetris isn’t really at play here. > The timing however is something I may turn an eye towards. > Most jobs have a “sanity” time limit applied, in that it is not so much an > expected time limit, but rather an “if it goes this long, something obviously > went > awry and we shouldn’t keep holding on to resources” limit. > So its a bit hard to quantify the timing portion, but I haven’t looked into > the slurm guesses of when it thinks the next task will start, etc. > > The pretty simplistic example at play here is that there are nodes that are > ~50-60% loaded for CPU and memory. > The next job up is a “whale” job that wants a ton of resources, cpu and/or > memory, but down the line there is a job with 2 cpu’s and 2 gb of memory > that can easily slot in to the unused resources. > > So my thinking was that the job_test list may be too short to actually get > that far down the queue to see that it could shove that job into some holes. You might also want to look at increasing bf_window to the maximum time limit, as suggested in 'man slurm.conf'. If backfill is not looking far enough into the future to know whether starting a job early will negatively impact a 'whale', then that 'whale' could potentially wait indefinitely. This is what happened on our system when we had a maximum runtime of 14 days but the 1 day default for bf_window. With both set to 14 days the problem was solved. Cheers, Loris > I’ll report back any findings after testing Loris’s suggestions. > > Appreciate everyone’s help and suggestions, > Reed > -- Dr. Loris Bennett (Herr/Mr) ZEDAT, Freie Universität Berlin
