[
https://issues.apache.org/jira/browse/MAPREDUCE-706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743468#action_12743468
]
Tsz Wo (Nicholas), SZE commented on MAPREDUCE-706:
--------------------------------------------------
The design doc is very nice (especially, it was typeset by tex)!
Some suggestions for future works:
- In hadoop, fs usually refers to FileSystem. "fsdesigndoc" sounds like
FileSystem design doc. I think we should prevent overloading the term "fs".
- The tex file needs a license header.
- We do not have pdf files under ./src before. "fsdesigndoc.pdf" is the first.
I think the correct approach is to generate the pdf file by "ant docs".
However, it may not be easy to do so.
> Support for FIFO pools in the fair scheduler
> --------------------------------------------
>
> Key: MAPREDUCE-706
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-706
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: contrib/fair-share
> Reporter: Matei Zaharia
> Assignee: Matei Zaharia
> Fix For: 0.21.0
>
> Attachments: fsdesigndoc.pdf, fsdesigndoc.tex, mapreduce-706.patch,
> mapreduce-706.v1.patch, mapreduce-706.v2.patch, mapreduce-706.v3.patch,
> mapreduce-706.v4.patch, mapreduce-706.v5.patch
>
>
> The fair scheduler should support making the internal scheduling algorithm
> for some pools be FIFO instead of fair sharing in order to work better for
> batch workloads. FIFO pools will behave exactly like the current default
> scheduler, sorting jobs by priority and then submission time. Pools will have
> their scheduling algorithm set through the pools config file, and it will be
> changeable at runtime.
> To support this feature, I'm also changing the internal logic of the fair
> scheduler to no longer use deficits. Instead, for fair sharing, we will
> assign tasks to the job farthest below its share as a ratio of its share.
> This is easier to combine with other scheduling algorithms and leads to a
> more stable sharing situation, avoiding unfairness issues brought up in
> MAPREDUCE-543 and MAPREDUCE-544 that happen when some jobs have long tasks.
> The new preemption (MAPREDUCE-551) will ensure that critical jobs can gain
> their fair share within a bounded amount of time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
