Hi Slurm-Dev,
I'm currently designing and testing what will ultimately be a small Slurm cluster of about 60 heterogeneous nodes (five different generations of hardware). Our user-base is also diverse, with need for fast turnover of small, sequential jobs and for long-duration parallel codes (e.g., 16 cores for several months).
In the past we limited users by how many cores they could allocate at any one time. This has the drawback that no distinction is made between, say, 128 cores for 2 hours and 128 cores for 2 months. We want users to be able to run on a large portion of the cluster when it is available while ensuring that they cannot take advantage of an idle period to start jobs which will monopolize it for weeks.
Limiting by GrpCPURunMins seems like a good answer. I think of it as allocating computational area (i.e., cores*minutes) and not just width (cores). I'd love to know if anyone has any experience or thoughts on imposing limits in this way. Also, is anyone aware of a simple way to calculate remaining "area"? I can use squeue or sacct to ultimately derive how much of a limit is in use by looking at remaining wall-time and core count, but if there's something built in - or pre-existing - it would be nice to know.
It's worth noting that the cluster is divided into several partitions with most nodes existing in several. This is partially political (to give groups increased priority on nodes they helped pay for) and partially practical (to ensure users explicitly requesting slow nodes instead of just dumping them on ancient Opterons). Also, each user gets their own Account, so the QoS Grp limits apply to each human separately. Accounts would also have absolute core limits.
Thank you for your thoughts! Corey -- Corey Keasling Software Manager JILA Computing Group University of Colorado-Boulder 440 UCB Room S244 Boulder, CO 80309-0440 303-492-9643
