Am 10.08.2016 um 21:35 schrieb Jaime Huerta Cepas: > Thanks Reuti, > the problem I have is that the queue system is used to handle jobs submitted > through a web service. Each job is an array job, and if 10 web users submit a > job, the last one in submitting has to wait until the rest is done for seeing > at least some of their tasks processed.
This is due to the way array jobs are handled. Once an array job was granted to start, it won't be put back to waiting state - only the tasks may have to wait until resources are free again. > As the web users are not registered in the host system, GE see all jobs as > from the same user (now thinking: is there any way to use fake users in GE?). Not fake users, but what about fake groups? In the web interface you can distinguish the users by their login or any other way? Before the submit jobs one could wrap this in a `newgrp` resp. `sg ...` and set SGE to honor the group. But I fear this won't help much due to the above mentioned fact. $ qconf -sconf ... execd_params USE_QSUB_GID=TRUE Only a co-sheduler could help, applying a hold to the waiting array tasks (therefore the -t option to `qalter` (resp. `qhold` / `qrls`, it won't change the range but addresses a specific range of instances). The syntax might be unusual though: $ qalter -h u 1082575 -t 50-200 -- Reuti > The max_aj_instances would mitigate the problem, but if the number of slots > is lower than the number of webusers, some array jobs will have to wait > anyways. > > cheers, > -jaime > > On 10 August 2016 at 20:58, Reuti <re...@staff.uni-marburg.de> wrote: > Hi, > > Am 10.08.2016 um 20:04 schrieb Jaime Huerta Cepas: > > > Hi there, > > I've been struggling for a while to find a configuration that allows me to > > balance multiple arrays from the same user evenly. > > > > I tried adjusting all priorities to the same level, but array jobs are > > always processed in submission order. I also tried assigning jobs to > > different projects. Although this works for normal jobs, array jobs seem to > > ignore the config (once an array job start being executed, all their tasks > > are processed until the next array job is executed). > > Correct. > > > > am I missing something obvious? Is there any way to get that behaviour? I > > would just need that the tasks from multiple array jobs from the same user > > are distributed evenly, not following the submission order. > > AFAICS this won't speed up the overall workflow as it's the same user. The > last instance of the last array job will finish at the same time, even if two > ore more jobs are interleaved. > > Nevertheless you can limit the number of instances which are executed of each > particular array job to avoid that one job fills all empty slots in the > cluster. > > a) `qsub -tc <max_running_tasks> ...` on a per job basis (could even be put > in a JSV) > > b) the setting "max_aj_instances" in SGE's configuration `qconf -mconf` > > HTH -- Reuti > > > > thanks a lot in advance, > > -jaime > > _______________________________________________ > > users mailing list > > users@gridengine.org > > https://gridengine.org/mailman/listinfo/users > > _______________________________________________ users mailing list users@gridengine.org https://gridengine.org/mailman/listinfo/users
