[
https://issues.apache.org/jira/browse/MAPREDUCE-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12935577#action_12935577
]
Matei Zaharia commented on MAPREDUCE-2198:
------------------------------------------
One other minor comment: if the fair scheduler is over-scheduling task
trackers, maybe we should consider that a bug. I don't think it was intended to
do that, although in trunk at least, it looks like it may do it if mapAssignCap
and reduceAssignCap are set to something less than infinity. (Otherwise, it
looks at the number of slots free on the TT and does not assign more than
that.) To deal with any sort of race condition that occurs if you lower a slot
count while a heartbeat is in progress, I'd suggest making the TT report
over-scheduled tasks as killed and drop them.
> Allow FairScheduler to control the number of slots on each TaskTracker
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-2198
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2198
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: contrib/fair-share
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
>
> We can set the number of slots on the TaskTracker to be high and let
> FairScheduler handles the slots.
> This approach allows us to change the number of slots on each node
> dynamically.
> The administrator can change the number of slots with a CLI tool.
> One use case of this is for upgrading the MapReduce.
> Instead of restarting the cluster, we can run the new MapReduce on the same
> cluster.
> And use the CLI tool to gradually migrate the slots.
> This way we don't lost the progress fo the jobs that's already executed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
