[ https://issues.apache.org/jira/browse/MAPREDUCE-706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matei Zaharia updated MAPREDUCE-706: ------------------------------------ Attachment: fsdesigndoc.tex fsdesigndoc.pdf I've attached a design document explaining the new organization of the code and the fair scheduler design in general. > Support for FIFO pools in the fair scheduler > -------------------------------------------- > > Key: MAPREDUCE-706 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-706 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/fair-share > Reporter: Matei Zaharia > Assignee: Matei Zaharia > Attachments: fsdesigndoc.pdf, fsdesigndoc.tex, mapreduce-706.patch > > > The fair scheduler should support making the internal scheduling algorithm > for some pools be FIFO instead of fair sharing in order to work better for > batch workloads. FIFO pools will behave exactly like the current default > scheduler, sorting jobs by priority and then submission time. Pools will have > their scheduling algorithm set through the pools config file, and it will be > changeable at runtime. > To support this feature, I'm also changing the internal logic of the fair > scheduler to no longer use deficits. Instead, for fair sharing, we will > assign tasks to the job farthest below its share as a ratio of its share. > This is easier to combine with other scheduling algorithms and leads to a > more stable sharing situation, avoiding unfairness issues brought up in > MAPREDUCE-543 and MAPREDUCE-544 that happen when some jobs have long tasks. > The new preemption (MAPREDUCE-551) will ensure that critical jobs can gain > their fair share within a bounded amount of time. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.