Hallo William,
many thanks for your quick reply! Okay, I need to specify:
From the scheduler's point of view the jobs are identical. They have no
"-l" resource requirements. And yes, "load" is the only criteria which
restricts access to a node. No nodes in alarm state. Any other ideas?
I'm almost sure it has to do with the "job_load_adjustment", I just cannot
prove it yet.... :-)
Best regards
Erik Soyez.
On Tue, 23 Jun 2015, William Hay wrote:
On Tue, 23 Jun 2015 12:33:27 +0000 Erik Soyez <[email protected]>
wrote:
Hello,
we have a very peculiar array job scheduling problem.
Cluster: Very heterogeneous SGE cluster with different types of
workstations.
Problem: One array job can use more job slots than two array jobs
together.
Example: User A submitted an array job, 64 array tasks were running
at the same time. Then user B submitted a similar array job and
after a while each of the jobs was running with approx 28 array
tasks. According to the users the problem gets worse with each array
job. Terminating one of the jobs immediately leads to the normal
usage again.
Question: Does anybody know if the "job_load_adjustment" of array
jobs depends only on number of tasks or if the number of jobs is
taken into account as well?
It should be just array tasks. Is load the only criteria which
restricts access to a node in your cluster?
Is there any clue from looking at the actual nodes. Do certain nodes
go into alarm with fewer jobs?
From my point of view limits, quotas, etc. cannot be the cause of the
problem because then the total of job slots being used by two jobs
could not be less then the job slots being used by one single job.
The scheduler configuration though was not made for short array jobs
but for longer running parallel jobs on interactively used
workstations.
You claim the jobs are similar but that isn't the same as identical
This might explain the difference. With identical jobs grid engine will
obtain optimal packing of jobs onto nodes simply by fitting as many
jobs onto each node as it can. For jobs with differing requirements
and nodes with different resources there is no simple strategy for
doing this (indeed it smells like an NP-hard problem to me)
Imagine two nodes with consumables as shown:
nodeA
h_vmem=4G diskspace=30G
nodeB
h_vmem=6G diskspace=20G
Array job 1 requests 2G h_vmem and 15G of diskspace
Array job 2 requests 3G of h_vmem and 10G of diskspace
If a task from array job 1 is scheduled to nodeB there isn't room for
another task of either job on that node(diskspace shortage). If a task
from array job 2 is scheduled to nodeA there isn't room for another
task of either job on that node(h_vmem shortage). On the other hand you
can fit two tasks of array job 1 on node A and two tasks of array job 2
on nodeB.
--
--
Vorstandsvorsitzender/Chairman of the board of management:
Gerd-Lothar Leonhart
Vorstand/Board of Management:
Dr. Bernd Finkbeiner, Dr. Arno Steitz
Vorsitzender des Aufsichtsrats/
Chairman of the Supervisory Board:
Philippe Miltin
Sitz/Registered Office: Tuebingen
Registergericht/Registration Court: Stuttgart
Registernummer/Commercial Register No.: HRB 382196
_______________________________________________
users mailing list
[email protected]
https://gridengine.org/mailman/listinfo/users
