Thanks Lyn – that was exactly the problem. Jake
From: slurm-users <slurm-users-boun...@lists.schedmd.com> On Behalf Of Lyn Gerner Sent: 03 June 2022 01:51 To: Slurm User Community List <slurm-users@lists.schedmd.com> Subject: Re: [slurm-users] New slurm configuration - multiple jobs per host Jake, my hunch is that your jobs are getting hung up on mem allocation, such that Slurm is assigning all of memory to each job as it runs; you can verify w/scontrol show job. If that's what's happening, try setting a DefMemPerCPU value for your partition(s). Best of luck, Lyn On Thu, May 26, 2022 at 1:39 PM Jake Jellinek <jakejelli...@outlook.com<mailto:jakejelli...@outlook.com>> wrote: Hi Ole I only added the oversubscribe option because without it, it didn’t work - so in fact, it appears not to have made any difference I though the RealMemory option just said not to offer any jobs to the node that didn’t have AT LEAST that amount of RAM My large node has more than 64GB RAM (and more will be allocated later) but I have yet to get to a memory issue…still working on cores jake@compute001:~$ slurmd -C NodeName=compute001 CPUs=32 Boards=1 SocketsPerBoard=2 CoresPerSocket=8 ThreadsPerCore=2 RealMemory=64359 UpTime=0-06:58:54 Thanks Jake > On 26 May 2022, at 21:11, Ole Holm Nielsen > <ole.h.niel...@fysik.dtu.dk<mailto:ole.h.niel...@fysik.dtu.dk>> wrote: > > Hi Jake, > > Firstly, which Slurm version and which OS do you use? > > Next, try simplifying by removing the oversubscribe configuration. Read the > slurm.conf manual page about oversubscribe, it looks a bit tricky. > > The RealMemory=1000 is extremely low and might prevent jobs from starting! > Run "slurmd -C" on the nodes to read appropriate node parameters for > slurm.conf. > > I hope this helps. > > /Ole > > >> On 26-05-2022 21:12, Jake Jellinek wrote: >> Hi >> I am just building my first Slurm setup and have got everything running – >> well, almost. >> I have a two node configuration. All of my setup exists on a single HyperV >> server and I have divided up the resources to create my VMs >> One node I will use for heavy duty work; this is called compute001 >> One node I will use for normal work; this is called compute002 >> My compute node specification in slurm.conf is >> NodeName=DEFAULT CPUs=1 RealMemory=1000 State=UNKNOWN >> NodeName=compute001 CPUs=32 >> NodeName=compute002 CPUs=2 >> The partition specification is >> PartitionName=DEFAULT State=UP >> PartitionName=interactive Nodes=compute002 MaxTime=INFINITE >> OverSubscribe=FORCE >> PartitionName=simulation Nodes=compute001 MaxTime=30 OverSubscribe=FORCE >> I have added the OverSubscribe=FORCE option as I want more than one job to >> be able to land on my interactive/simulation queues. >> All of the nodes and cluster master start up fine and they all talk to each >> other but no matter what I do, I cannot get my cluster to accept more than >> one job per node. >> Can you help me determine where I am going wrong? >> Thanks a lot >> Jake >> The entire slurm.conf is pasted below >> # slurm.conf file generated by configurator.html. >> ClusterName=pm-slurm >> SlurmctldHost=slurm-master >> MpiDefault=none >> ProctrackType=proctrack/cgroup >> ReturnToService=2 >> SlurmctldPidFile=/var/run/slurmctld.pid >> SlurmctldPort=6817 >> SlurmdPidFile=/var/run/slurmd.pid >> SlurmdPort=6818 >> SlurmdSpoolDir=/var/spool/slurmd >> SlurmUser=slurm >> StateSaveLocation=/home/slurm/var/spool/slurmctld >> SwitchType=switch/none >> TaskPlugin=task/cgroup >> # >> # TIMERS >> InactiveLimit=0 >> KillWait=30 >> MinJobAge=300 >> SlurmctldTimeout=120 >> SlurmdTimeout=300 >> Waittime=0 >> # >> # SCHEDULING >> SchedulerType=sched/backfill >> SelectType=select/cons_tres >> SelectTypeParameters=CR_Core_Memory >> # >> # LOGGING AND ACCOUNTING >> JobAcctGatherFrequency=30 >> JobAcctGatherType=jobacct_gather/cgroup >> SlurmctldDebug=info >> SlurmctldLogFile=/var/log/slurmctld.log >> SlurmdDebug=info >> SlurmdLogFile=/var/log/slurmd.log >> # COMPUTE NODES >> NodeName=DEFAULT CPUs=1 RealMemory=1000 State=UNKNOWN >> NodeName=compute001 CPUs=32 >> NodeName=compute002 CPUs=2 >> PartitionName=DEFAULT State=UP >> PartitionName=interactive Nodes=compute002 MaxTime=INFINITE >> OverSubscribe=FORCE >> PartitionName=simulation Nodes=compute001 MaxTime=30 OverSubscribe=FORCE > >
