[
https://issues.apache.org/jira/browse/MAPREDUCE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115255#comment-13115255
]
Jeff Bean commented on MAPREDUCE-2905:
--------------------------------------
I was mistaken. CapBasedLoadManager isn't working.
The problem is this:
public boolean canAssignMap(TaskTrackerStatus tracker,
int totalRunnableMaps, int totalMapSlots) {
return tracker.countMapTasks() < getCap(totalRunnableMaps,
tracker.getMaxMapSlots(), totalMapSlots);
}
The calls to tracker.countMapTasks() and tracker.countReduceTasks() always
return 0 when called from the loop inside assignTasks. Even though tasks get
assigned by the scheduler, countMapTasks() and countReduceTasks don't change
what they return.
Need to either fix countMapTasks() or determine another way to pass in the
information about the tasks which were just assigned by the scheduler.
> Allow mapred.fairscheduler.assignmultple to be set per job
> ----------------------------------------------------------
>
> Key: MAPREDUCE-2905
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2905
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
> Affects Versions: 0.20.2
> Reporter: Jeff Bean
>
> We encountered a situation where in the same cluster, large jobs benefit from
> mapred.fairscheduler.assignmultiple, but small jobs with small numbers of
> mappers do not: the mappers all clump to fully occupy just a few nodes, which
> causes those nodes to saturate and bottleneck. The desired behavior is to
> spread the job across more nodes so that a relatively small job doesn't
> saturate any node in the cluster.
> Testing has shown that setting mapred.fairscheduler.assignmultiple to false
> gives the desired behavior for small jobs, but is unnecessary for large jobs.
> However, since this is a cluster-wide setting, we can't properly tune.
> It'd be nice if jobs can set a param similar to
> mapred.fairscheduler.assignmultiple on submission to better control the task
> distribution of a particular job.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
