Hi, My problem is that less than half the capacity of our nodes in being used.
We have two 64-core nodes. At any one time <~50 jobs are scheduled. Looking at a job that's queued I get: $ checkjob 22360 checking job 22360 State: Idle Creds: user:bckhouse group:minos class:minos qos:DEFAULT WallTime: 00:00:00 of INFINITY SubmitTime: Tue May 14 06:11:10 (Time Queued Total: 5:24:43 Eligible: 5:24:43) Total Tasks: 1 Req[0] TaskCount: 1 Partition: ALL Network: [NONE] Memory >= 0 Disk >= 0 Swap >= 0 Opsys: [NONE] Arch: [NONE] Features: [NONE] Dedicated Resources Per Task: PROCS: 1 MEM: 2000M IWD: [NONE] Executable: [NONE] Bypass: 0 StartCount: 0 PartitionMask: [ALL] Flags: RESTARTABLE Reservation '22360' (00:00:00 -> INFINITY Duration: INFINITY) PE: 1.00 StartPriority: 324 job cannot run in partition DEFAULT (insufficient idle procs available: 0 < 1) So, the problem is there are no idle procs. Checking one of the nodes: $ checknode node078 checking node node078 State: Busy (in current state for 00:32:08) Expected State: Running SyncDeadline: Tue May 14 11:37:10 Configured Resources: PROCS: 64 MEM: 252G SWAP: 252G DISK: 558G Utilized Resources: PROCS: 64 SWAP: 17G DISK: 32M Dedicated Resources: PROCS: 24 MEM: 46G Opsys: linux Arch: [NONE] Speed: 1.00 Load: 23.180 Network: [DEFAULT] Features: [amd][MEM256G] Attributes: [Batch] Classes: [batch 64:64][minos 40:64] Total Time: INFINITY Up: INFINITY (99.66%) Active: 27:15:36:09 (20.67%) <snip a bunch of reservations corresponding to the running jobs> So it knows it's only scheduled ("dedicated") 24 jobs, but it thinks the utilization is the full 64. This obviously isn't true. node078:~$ uptime 11:38:13 up 16:45, 2 users, load average: 22.97, 23.24, 23.97 So the load is only the 24 jobs that are running, not 64. Likewise, the node looks half-loaded in ganglia. So, how is the "utilized" number determined? The docs make it sound like it's just the load average, but that doesn't seem to be the case. I set "NODEAVAILABILITYPOLICY DEDICATED:PROCS" in maui.cfg on the head node, in an attempt to get maui to ignore the "utilized" number, but it doesn't seem to have had any effect. Additional information: I can get the nodes full if I "surprise" them by submitting 128 jobs simultaneously. Those jobs run happily. That's the state I want to be able to obtain in the general case. Watching the load on ganglia, it looks like the jobs drain out of the system as they complete, and then every 30 minutes they're topped back up to the ~half utilization level. My guess is driven by the "defer" mechanism. Sorry if this problem has been answered before. I found several instances of what sound like the same thing in the mail archives, but no solutions :( Thanks - Chris _______________________________________________ mauiusers mailing list mauiusers@supercluster.org http://www.supercluster.org/mailman/listinfo/mauiusers