[
https://issues.apache.org/jira/browse/HADOOP-5187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12671753#action_12671753
]
Matei Zaharia commented on HADOOP-5187:
---------------------------------------
Right, this is what I was proposing. For fair sharing mode, the global ordering
by deficit works fine because it makes sure that each pool gets its own share.
However, for FIFO, it means that the scheduler as a whole runs jobs in FIFO
order, regardless of what pool they're in, so things like per-pool minimum
shares don't work. I imagine it may be useful to support having multiple pools
each of which runs FIFO - this would emulate the capabilities of the capacity
scheduler for example. Does this make sense?
Doing this would currently require tricky logic if we wanted to integrate it
with the deficit model to allow FIFO and fair-share pools to coexist. However,
if instead we remove the concept of deficits and sort jobs by how far they are
below their fair share (or how long they've waited if they are below their min
share), I think it should be possible to define a comparator that takes into
account both FIFO and fair-sharing pools. This is why I'm proposing to remove
the deficit model in HADOOP-4803. It seemed like a good idea initially,
especially when the fair scheduler had no preemption, but it does the wrong
thing at times (as pointed out in that JIRA) and it complicates the code,
making it hard to add features such as this.
> Provide an option to turn off priorities in jobs
> ------------------------------------------------
>
> Key: HADOOP-5187
> URL: https://issues.apache.org/jira/browse/HADOOP-5187
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/fair-share
> Reporter: Hemanth Yamijala
> Priority: Minor
>
> The fairshare scheduler can define pools mapping to queues (as defined in the
> capacity scheduler - HADOOP-3445). When used in this manner, one can imagine
> queues set up to be used by users who come from disparate teams or
> organizations (say a default queue). For such a queue, it makes sense to
> ignore job priorities and consider the queue as strict FIFO, as it is
> difficult to compare priorities of jobs from different users.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
