Support for FIFO pools in the fair scheduler
--------------------------------------------
Key: MAPREDUCE-706
URL: https://issues.apache.org/jira/browse/MAPREDUCE-706
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: contrib/fair-share
Reporter: Matei Zaharia
The fair scheduler should support making the internal scheduling algorithm for
some pools be FIFO instead of fair sharing in order to work better for batch
workloads. FIFO pools will behave exactly like the current default scheduler,
sorting jobs by priority and then submission time. Pools will have their
scheduling algorithm set through the pools config file, and it will be
changeable at runtime.
To support this feature, I'm also changing the internal logic of the fair
scheduler to no longer use deficits. Instead, for fair sharing, we will assign
tasks to the job farthest below its share as a ratio of its share. This is
easier to combine with other scheduling algorithms and leads to a more stable
sharing situation, avoiding unfairness issues brought up in MAPREDUCE-543 and
MAPREDUCE-544 that happen when some jobs have long tasks. The new preemption
(MAPREDUCE-551) will ensure that critical jobs can gain their fair share within
a bounded amount of time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
