[jira] [Commented] (MAPREDUCE-2905) CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155363#comment-13155363 ] Todd Lipcon commented on MAPREDUCE-2905: Thanks Matei. Will fix that on commit. > CapBasedLoadManager incorrectly allows assignment when assignMultiple is true > (was: assignmultiple per job) > --- > > Key: MAPREDUCE-2905 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2905 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/fair-share >Affects Versions: 0.20.2 >Reporter: Jeff Bean >Assignee: Jeff Bean > Attachments: MR-2905.10-13-2011, MR-2905.patch, MR-2905.patch.2, > mr-2905.txt, mr-2905.txt, screenshot-1.jpg > > > We encountered a situation where in the same cluster, large jobs benefit from > mapred.fairscheduler.assignmultiple, but small jobs with small numbers of > mappers do not: the mappers all clump to fully occupy just a few nodes, which > causes those nodes to saturate and bottleneck. The desired behavior is to > spread the job across more nodes so that a relatively small job doesn't > saturate any node in the cluster. > Testing has shown that setting mapred.fairscheduler.assignmultiple to false > gives the desired behavior for small jobs, but is unnecessary for large jobs. > However, since this is a cluster-wide setting, we can't properly tune. > It'd be nice if jobs can set a param similar to > mapred.fairscheduler.assignmultiple on submission to better control the task > distribution of a particular job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2905) CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job)
[
https://issues.apache.org/jira/browse/MAPREDUCE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155358#comment-13155358
]
Matei Zaharia commented on MAPREDUCE-2905:
--
The patch looks good, except that there's an extra space on a line in
TestFairScheduler:
{code:java}
List tasks = scheduler.assignTasks(tracker(taskTrackerName))
{code}
Other than that, I think it's good to go.
> CapBasedLoadManager incorrectly allows assignment when assignMultiple is true
> (was: assignmultiple per job)
> ---
>
> Key: MAPREDUCE-2905
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2905
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/fair-share
>Affects Versions: 0.20.2
>Reporter: Jeff Bean
>Assignee: Jeff Bean
> Attachments: MR-2905.10-13-2011, MR-2905.patch, MR-2905.patch.2,
> mr-2905.txt, mr-2905.txt, screenshot-1.jpg
>
>
> We encountered a situation where in the same cluster, large jobs benefit from
> mapred.fairscheduler.assignmultiple, but small jobs with small numbers of
> mappers do not: the mappers all clump to fully occupy just a few nodes, which
> causes those nodes to saturate and bottleneck. The desired behavior is to
> spread the job across more nodes so that a relatively small job doesn't
> saturate any node in the cluster.
> Testing has shown that setting mapred.fairscheduler.assignmultiple to false
> gives the desired behavior for small jobs, but is unnecessary for large jobs.
> However, since this is a cluster-wide setting, we can't properly tune.
> It'd be nice if jobs can set a param similar to
> mapred.fairscheduler.assignmultiple on submission to better control the task
> distribution of a particular job.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2905) CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151533#comment-13151533 ] Todd Lipcon commented on MAPREDUCE-2905: So, I disagree with some of what you said above. The test was broken, but not quite in the way you described. The issue is that the test was asserting correct behavior, but the mocking didn't accurately reflect the way the true scheduler interacts with JobInProgress, etc. In the mocks, as soon as "obtainNewMapTask" was called, the new task was inserted into the TaskTrackerStatus's taskReports structure, so that the "countMapTasks" and "countReduceTasks" functions included the just-scheduled tasks. So, the old code in LoadManager actually did the right thing as far as the test/mock setup was concerned. Once we fixed the LoadManager to work correctly with the real code (which doesn't insert anything into TaskTrackerStatus when the tasks are allocated), it ended up basically double-counting each assigned task when running against the mocks. So, only half as many tasks were scheduled as were supposed to. The fix was to change the mock to obtain all of the scheduled tasks, and only then add them to the task report structure. I also had to change the code in the assignment loop to add mapsAssigned and reducesAssigned around line 476 of FairScheduler.java. Otherwise the "flip flopping" back and forth between map and reduce task assignment broke. > CapBasedLoadManager incorrectly allows assignment when assignMultiple is true > (was: assignmultiple per job) > --- > > Key: MAPREDUCE-2905 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2905 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/fair-share >Affects Versions: 0.20.2 >Reporter: Jeff Bean >Assignee: Jeff Bean > Attachments: MR-2905.10-13-2011, MR-2905.patch, MR-2905.patch.2, > mr-2905.txt, mr-2905.txt, screenshot-1.jpg > > > We encountered a situation where in the same cluster, large jobs benefit from > mapred.fairscheduler.assignmultiple, but small jobs with small numbers of > mappers do not: the mappers all clump to fully occupy just a few nodes, which > causes those nodes to saturate and bottleneck. The desired behavior is to > spread the job across more nodes so that a relatively small job doesn't > saturate any node in the cluster. > Testing has shown that setting mapred.fairscheduler.assignmultiple to false > gives the desired behavior for small jobs, but is unnecessary for large jobs. > However, since this is a cluster-wide setting, we can't properly tune. > It'd be nice if jobs can set a param similar to > mapred.fairscheduler.assignmultiple on submission to better control the task > distribution of a particular job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2905) CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151008#comment-13151008 ] Jeff Bean commented on MAPREDUCE-2905: -- just a bunch of tuesday. no big deal. > CapBasedLoadManager incorrectly allows assignment when assignMultiple is true > (was: assignmultiple per job) > --- > > Key: MAPREDUCE-2905 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2905 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/fair-share >Affects Versions: 0.20.2 >Reporter: Jeff Bean >Assignee: Jeff Bean > Attachments: MR-2905.10-13-2011, MR-2905.patch, MR-2905.patch.2, > mr-2905.txt, screenshot-1.jpg > > > We encountered a situation where in the same cluster, large jobs benefit from > mapred.fairscheduler.assignmultiple, but small jobs with small numbers of > mappers do not: the mappers all clump to fully occupy just a few nodes, which > causes those nodes to saturate and bottleneck. The desired behavior is to > spread the job across more nodes so that a relatively small job doesn't > saturate any node in the cluster. > Testing has shown that setting mapred.fairscheduler.assignmultiple to false > gives the desired behavior for small jobs, but is unnecessary for large jobs. > However, since this is a cluster-wide setting, we can't properly tune. > It'd be nice if jobs can set a param similar to > mapred.fairscheduler.assignmultiple on submission to better control the task > distribution of a particular job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2905) CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150994#comment-13150994 ] Todd Lipcon commented on MAPREDUCE-2905: Hey Jeff - hope you didnt spend much time on it. After my earlier comment I looked into it and came to similar conclusions... i'll post a new patch here soon. > CapBasedLoadManager incorrectly allows assignment when assignMultiple is true > (was: assignmultiple per job) > --- > > Key: MAPREDUCE-2905 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2905 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/fair-share >Affects Versions: 0.20.2 >Reporter: Jeff Bean >Assignee: Jeff Bean > Attachments: MR-2905.10-13-2011, MR-2905.patch, MR-2905.patch.2, > mr-2905.txt, screenshot-1.jpg > > > We encountered a situation where in the same cluster, large jobs benefit from > mapred.fairscheduler.assignmultiple, but small jobs with small numbers of > mappers do not: the mappers all clump to fully occupy just a few nodes, which > causes those nodes to saturate and bottleneck. The desired behavior is to > spread the job across more nodes so that a relatively small job doesn't > saturate any node in the cluster. > Testing has shown that setting mapred.fairscheduler.assignmultiple to false > gives the desired behavior for small jobs, but is unnecessary for large jobs. > However, since this is a cluster-wide setting, we can't properly tune. > It'd be nice if jobs can set a param similar to > mapred.fairscheduler.assignmultiple on submission to better control the task > distribution of a particular job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2905) CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150969#comment-13150969 ] Jeff Bean commented on MAPREDUCE-2905: -- trying to think about how to fix the test, though. can we increase the slot count per tracker of the test cluster so as not to hit the clumping? Or maybe just comment out those particular tests? > CapBasedLoadManager incorrectly allows assignment when assignMultiple is true > (was: assignmultiple per job) > --- > > Key: MAPREDUCE-2905 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2905 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/fair-share >Affects Versions: 0.20.2 >Reporter: Jeff Bean >Assignee: Jeff Bean > Attachments: MR-2905.10-13-2011, MR-2905.patch, MR-2905.patch.2, > mr-2905.txt, screenshot-1.jpg > > > We encountered a situation where in the same cluster, large jobs benefit from > mapred.fairscheduler.assignmultiple, but small jobs with small numbers of > mappers do not: the mappers all clump to fully occupy just a few nodes, which > causes those nodes to saturate and bottleneck. The desired behavior is to > spread the job across more nodes so that a relatively small job doesn't > saturate any node in the cluster. > Testing has shown that setting mapred.fairscheduler.assignmultiple to false > gives the desired behavior for small jobs, but is unnecessary for large jobs. > However, since this is a cluster-wide setting, we can't properly tune. > It'd be nice if jobs can set a param similar to > mapred.fairscheduler.assignmultiple on submission to better control the task > distribution of a particular job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2905) CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150968#comment-13150968 ] Jeff Bean commented on MAPREDUCE-2905: -- (strike the last incomplete sentence of the last comment.) > CapBasedLoadManager incorrectly allows assignment when assignMultiple is true > (was: assignmultiple per job) > --- > > Key: MAPREDUCE-2905 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2905 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/fair-share >Affects Versions: 0.20.2 >Reporter: Jeff Bean >Assignee: Jeff Bean > Attachments: MR-2905.10-13-2011, MR-2905.patch, MR-2905.patch.2, > mr-2905.txt, screenshot-1.jpg > > > We encountered a situation where in the same cluster, large jobs benefit from > mapred.fairscheduler.assignmultiple, but small jobs with small numbers of > mappers do not: the mappers all clump to fully occupy just a few nodes, which > causes those nodes to saturate and bottleneck. The desired behavior is to > spread the job across more nodes so that a relatively small job doesn't > saturate any node in the cluster. > Testing has shown that setting mapred.fairscheduler.assignmultiple to false > gives the desired behavior for small jobs, but is unnecessary for large jobs. > However, since this is a cluster-wide setting, we can't properly tune. > It'd be nice if jobs can set a param similar to > mapred.fairscheduler.assignmultiple on submission to better control the task > distribution of a particular job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2905) CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job)
[
https://issues.apache.org/jira/browse/MAPREDUCE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150964#comment-13150964
]
Jeff Bean commented on MAPREDUCE-2905:
--
testFairScheduler is buggy. In one instance it actually looks for the bug
condition and passes it by asserting that tt1 fills to capacity on small jobs
that shouldn't clump together:
checkAssignment("tt1", "attempt_test_0001_m_00_0 on tt1",
"attempt_test_0001_r_00_0 on tt1",
"attempt_test_0002_m_00_0 on tt1",
"attempt_test_0002_r_00_0 on tt1");
checkAssignment("tt2", "attempt_test_0001_m_01_0 on tt2",
"attempt_test_0002_r_01_0 on tt2");
If tt1 has a capacity of 2 mappers and 2 reducers, I expect 1 mapper and 1
reducer to be assigned each as to assign all 4 slots would be the clumping
condition that this patch is trying to address.
On every instance of checkAssignment that gets multiple tasks, the test fails
on the length of the task list. The assignMultiple tests are broken for the
reason I describe above. The other tests that fail are the
testDelaySchedulingAtNodeLevel, testDelaySchedulingAtRackLevel, and
testDelaySchedulingOffRack. In all three of those cases a task list length of 2
of the same type of is expected and a task list length of 1 is returned. Again,
with a default capacity of 2 slots per task type, we'd be over capacity and
actually looking for the condition this patch fixes.
Relevant output from failure:
Testcase: testSmallJobsWithAssignMultiple took 0.58 sec
FAILED
expected:<4> but was:<2>
junit.framework.AssertionFailedError: expected:<4> but was:<2>
at
org.apache.hadoop.mapred.TestFairScheduler.checkAssignment(TestFairScheduler.java:2810)
at
org.apache.hadoop.mapred.TestFairScheduler.testSmallJobsWithAssignMultiple(TestFairScheduler.java:784)
Testcase: testLargeJobs took 0.432 sec
Testcase: testLargeJobsWithAssignMultiple took 0.435 sec
FAILED
expected:<4> but was:<2>
junit.framework.AssertionFailedError: expected:<4> but was:<2>
at
org.apache.hadoop.mapred.TestFairScheduler.checkAssignment(TestFairScheduler.java:2810)
at
org.apache.hadoop.mapred.TestFairScheduler.testLargeJobsWithAssignMultiple(TestFairScheduler.java:954)
Testcase: testJobsWithPriorities took 0.432 sec
Testcase: testLargeJobsWithPools took 0.672 sec
Testcase: testLargeJobsWithExcessCapacity took 0.598 sec
Testcase: testLargeJobsWithExcessCapacityAndAssignMultiple took 0.594 sec
FAILED
expected:<4> but was:<2>
junit.framework.AssertionFailedError: expected:<4> but was:<2>
at
org.apache.hadoop.mapred.TestFairScheduler.checkAssignment(TestFairScheduler.java:2810)
at
org.apache.hadoop.mapred.TestFairScheduler.testLargeJobsWithExcessCapacityAndAssignMultiple(TestFairScheduler.java:1301)
Testcase: testSmallJobInLargePool took 0.517 sec
Testcase: testPoolMaxJobs took 0.842 sec
Testcase: testUserMaxJobs took 0.73 sec
Testcase: testComplexJobLimits took 2.053 sec
Testcase: testSizeBasedWeight took 0.387 sec
Testcase: testPoolWeights took 0.851 sec
Testcase: testPoolWeightsWhenNoMaps took 0.637 sec
Testcase: testPoolMaxMapsReduces took 0.404 sec
Testcase: testCapBasedLoadManager took 0.119 sec
Testcase: testMinSharePreemption took 0.487 sec
Testcase: testMinSharePreemptionWithSmallJob took 0.426 sec
Testcase: testFairSharePreemption took 0.612 sec
Testcase: testFairSharePreemptionFromMultiplePools took 0.602 sec
Testcase: testMinAndFairSharePreemption took 0.42 sec
Testcase: testNoPreemptionIfDisabled took 0.457 sec
Testcase: testNoPreemptionIfOnlyLogging took 0.458 sec
Testcase: testDelaySchedulingAtNodeLevel took 0.366 sec
FAILED
expected:<2> but was:<1>
junit.framework.AssertionFailedError: expected:<2> but was:<1>
at
org.apache.hadoop.mapred.TestFairScheduler.checkAssignment(TestFairScheduler.java:2810)
at
org.apache.hadoop.mapred.TestFairScheduler.testDelaySchedulingAtNodeLevel(TestFairScheduler.java:2301)
Testcase: testDelaySchedulingAtRackLevel took 0.521 sec
FAILED
expected:<2> but was:<1>
junit.framework.AssertionFailedError: expected:<2> but was:<1>
at
org.apache.hadoop.mapred.TestFairScheduler.checkAssignment(TestFairScheduler.java:2810)
at
org.apache.hadoop.mapred.TestFairScheduler.testDelaySchedulingAtRackLevel(TestFairScheduler.java:2349)
Testcase: testDelaySchedulingOffRack took 0.261 sec
FAILED
expected:<2> but was:<1>
junit.framework.AssertionFailedError: expected:<2> but was:<1>
This is every time checkAssignment() gets called with more than one task as a
parameter. This is a buggy test. For example, in one instance it actuall
> CapBasedLoadManager incorrectly allows assignment when assignMultiple is true
> (was: assignmultiple per job)
> -
[jira] [Commented] (MAPREDUCE-2905) CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150716#comment-13150716 ] Jeff Bean commented on MAPREDUCE-2905: -- More info about the failure? > CapBasedLoadManager incorrectly allows assignment when assignMultiple is true > (was: assignmultiple per job) > --- > > Key: MAPREDUCE-2905 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2905 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/fair-share >Affects Versions: 0.20.2 >Reporter: Jeff Bean >Assignee: Jeff Bean > Attachments: MR-2905.10-13-2011, MR-2905.patch, MR-2905.patch.2, > mr-2905.txt, screenshot-1.jpg > > > We encountered a situation where in the same cluster, large jobs benefit from > mapred.fairscheduler.assignmultiple, but small jobs with small numbers of > mappers do not: the mappers all clump to fully occupy just a few nodes, which > causes those nodes to saturate and bottleneck. The desired behavior is to > spread the job across more nodes so that a relatively small job doesn't > saturate any node in the cluster. > Testing has shown that setting mapred.fairscheduler.assignmultiple to false > gives the desired behavior for small jobs, but is unnecessary for large jobs. > However, since this is a cluster-wide setting, we can't properly tune. > It'd be nice if jobs can set a param similar to > mapred.fairscheduler.assignmultiple on submission to better control the task > distribution of a particular job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2905) CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150706#comment-13150706 ] Todd Lipcon commented on MAPREDUCE-2905: TestFairScheduler is failing with this patch - so need to either fix up the test case or figure out what bug it might have introduced. > CapBasedLoadManager incorrectly allows assignment when assignMultiple is true > (was: assignmultiple per job) > --- > > Key: MAPREDUCE-2905 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2905 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/fair-share >Affects Versions: 0.20.2 >Reporter: Jeff Bean >Assignee: Jeff Bean > Attachments: MR-2905.10-13-2011, MR-2905.patch, MR-2905.patch.2, > mr-2905.txt, screenshot-1.jpg > > > We encountered a situation where in the same cluster, large jobs benefit from > mapred.fairscheduler.assignmultiple, but small jobs with small numbers of > mappers do not: the mappers all clump to fully occupy just a few nodes, which > causes those nodes to saturate and bottleneck. The desired behavior is to > spread the job across more nodes so that a relatively small job doesn't > saturate any node in the cluster. > Testing has shown that setting mapred.fairscheduler.assignmultiple to false > gives the desired behavior for small jobs, but is unnecessary for large jobs. > However, since this is a cluster-wide setting, we can't properly tune. > It'd be nice if jobs can set a param similar to > mapred.fairscheduler.assignmultiple on submission to better control the task > distribution of a particular job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2905) CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150690#comment-13150690 ] Matei Zaharia commented on MAPREDUCE-2905: -- Yup, the patch itself also looks good. > CapBasedLoadManager incorrectly allows assignment when assignMultiple is true > (was: assignmultiple per job) > --- > > Key: MAPREDUCE-2905 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2905 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/fair-share >Affects Versions: 0.20.2 >Reporter: Jeff Bean >Assignee: Jeff Bean > Attachments: MR-2905.10-13-2011, MR-2905.patch, MR-2905.patch.2, > mr-2905.txt, screenshot-1.jpg > > > We encountered a situation where in the same cluster, large jobs benefit from > mapred.fairscheduler.assignmultiple, but small jobs with small numbers of > mappers do not: the mappers all clump to fully occupy just a few nodes, which > causes those nodes to saturate and bottleneck. The desired behavior is to > spread the job across more nodes so that a relatively small job doesn't > saturate any node in the cluster. > Testing has shown that setting mapred.fairscheduler.assignmultiple to false > gives the desired behavior for small jobs, but is unnecessary for large jobs. > However, since this is a cluster-wide setting, we can't properly tune. > It'd be nice if jobs can set a param similar to > mapred.fairscheduler.assignmultiple on submission to better control the task > distribution of a particular job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2905) CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150050#comment-13150050 ] Eli Collins commented on MAPREDUCE-2905: Hi Matei, Are you +1 on Todd's patch or just the approach, ie does the patch look good to go? > CapBasedLoadManager incorrectly allows assignment when assignMultiple is true > (was: assignmultiple per job) > --- > > Key: MAPREDUCE-2905 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2905 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/fair-share >Affects Versions: 0.20.2 >Reporter: Jeff Bean >Assignee: Jeff Bean > Attachments: MR-2905.10-13-2011, MR-2905.patch, MR-2905.patch.2, > mr-2905.txt, screenshot-1.jpg > > > We encountered a situation where in the same cluster, large jobs benefit from > mapred.fairscheduler.assignmultiple, but small jobs with small numbers of > mappers do not: the mappers all clump to fully occupy just a few nodes, which > causes those nodes to saturate and bottleneck. The desired behavior is to > spread the job across more nodes so that a relatively small job doesn't > saturate any node in the cluster. > Testing has shown that setting mapred.fairscheduler.assignmultiple to false > gives the desired behavior for small jobs, but is unnecessary for large jobs. > However, since this is a cluster-wide setting, we can't properly tune. > It'd be nice if jobs can set a param similar to > mapred.fairscheduler.assignmultiple on submission to better control the task > distribution of a particular job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2905) CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137787#comment-13137787 ] Matei Zaharia commented on MAPREDUCE-2905: -- Sorry for taking a bit of time to get to this, but I agree with Todd that the approach where we change the LoadManager API a little is better. LoadManager was defined as an extension point but it's a pretty advanced feature that I don't think anyone except Facebook has looked at changing. The benefits in understandability from Todd's approach (from not having to maintain state in the LoadManager object) outweigh the cost. > CapBasedLoadManager incorrectly allows assignment when assignMultiple is true > (was: assignmultiple per job) > --- > > Key: MAPREDUCE-2905 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2905 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/fair-share >Affects Versions: 0.20.2 >Reporter: Jeff Bean >Assignee: Jeff Bean > Attachments: MR-2905.10-13-2011, MR-2905.patch, MR-2905.patch.2, > mr-2905.txt, screenshot-1.jpg > > > We encountered a situation where in the same cluster, large jobs benefit from > mapred.fairscheduler.assignmultiple, but small jobs with small numbers of > mappers do not: the mappers all clump to fully occupy just a few nodes, which > causes those nodes to saturate and bottleneck. The desired behavior is to > spread the job across more nodes so that a relatively small job doesn't > saturate any node in the cluster. > Testing has shown that setting mapred.fairscheduler.assignmultiple to false > gives the desired behavior for small jobs, but is unnecessary for large jobs. > However, since this is a cluster-wide setting, we can't properly tune. > It'd be nice if jobs can set a param similar to > mapred.fairscheduler.assignmultiple on submission to better control the task > distribution of a particular job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2905) CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13129536#comment-13129536 ] Jeff Bean commented on MAPREDUCE-2905: -- Hi Todd, I considered your patch but thought it was illegal to break an API. Anyway, I tested your patch and it's adequate. On a 5 node cluster with 7 slots per node, I run the test: hadoop jar /usr/lib/hadoop/hadoop-examples.jar sleep -m 12 -mt 30 your patch divvies up the tasks 3, 3, 3, 3, 0. My patch divvies up the tasks 2, 2, 2, 2, 2. Mine's a little better, but I'm not complaining: without either patch tasks are distributed as follows: 7, 5, 0, 0, 0. > CapBasedLoadManager incorrectly allows assignment when assignMultiple is true > (was: assignmultiple per job) > --- > > Key: MAPREDUCE-2905 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2905 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/fair-share >Affects Versions: 0.20.2 >Reporter: Jeff Bean > Attachments: MR-2905.10-13-2011, MR-2905.patch, MR-2905.patch.2, > mr-2905.txt, screenshot-1.jpg > > > We encountered a situation where in the same cluster, large jobs benefit from > mapred.fairscheduler.assignmultiple, but small jobs with small numbers of > mappers do not: the mappers all clump to fully occupy just a few nodes, which > causes those nodes to saturate and bottleneck. The desired behavior is to > spread the job across more nodes so that a relatively small job doesn't > saturate any node in the cluster. > Testing has shown that setting mapred.fairscheduler.assignmultiple to false > gives the desired behavior for small jobs, but is unnecessary for large jobs. > However, since this is a cluster-wide setting, we can't properly tune. > It'd be nice if jobs can set a param similar to > mapred.fairscheduler.assignmultiple on submission to better control the task > distribution of a particular job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2905) CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13129453#comment-13129453 ] Jeff Bean commented on MAPREDUCE-2905: -- Oh I'm happy to do it. Is two-space indentation an apache or hadoop standard? > CapBasedLoadManager incorrectly allows assignment when assignMultiple is true > (was: assignmultiple per job) > --- > > Key: MAPREDUCE-2905 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2905 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/fair-share >Affects Versions: 0.20.2 >Reporter: Jeff Bean > Attachments: MR-2905.10-13-2011, MR-2905.patch, MR-2905.patch.2, > screenshot-1.jpg > > > We encountered a situation where in the same cluster, large jobs benefit from > mapred.fairscheduler.assignmultiple, but small jobs with small numbers of > mappers do not: the mappers all clump to fully occupy just a few nodes, which > causes those nodes to saturate and bottleneck. The desired behavior is to > spread the job across more nodes so that a relatively small job doesn't > saturate any node in the cluster. > Testing has shown that setting mapred.fairscheduler.assignmultiple to false > gives the desired behavior for small jobs, but is unnecessary for large jobs. > However, since this is a cluster-wide setting, we can't properly tune. > It'd be nice if jobs can set a param similar to > mapred.fairscheduler.assignmultiple on submission to better control the task > distribution of a particular job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2905) CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13129451#comment-13129451 ] Todd Lipcon commented on MAPREDUCE-2905: The patch seems to reformat a bunch of stuff to 4-space indentation instead of 2-space, making it tough to review. Since I've already made you do several iterations, let me take care of the next one for you... will upload an updated patch soon. > CapBasedLoadManager incorrectly allows assignment when assignMultiple is true > (was: assignmultiple per job) > --- > > Key: MAPREDUCE-2905 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2905 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/fair-share >Affects Versions: 0.20.2 >Reporter: Jeff Bean > Attachments: MR-2905.10-13-2011, MR-2905.patch, MR-2905.patch.2, > screenshot-1.jpg > > > We encountered a situation where in the same cluster, large jobs benefit from > mapred.fairscheduler.assignmultiple, but small jobs with small numbers of > mappers do not: the mappers all clump to fully occupy just a few nodes, which > causes those nodes to saturate and bottleneck. The desired behavior is to > spread the job across more nodes so that a relatively small job doesn't > saturate any node in the cluster. > Testing has shown that setting mapred.fairscheduler.assignmultiple to false > gives the desired behavior for small jobs, but is unnecessary for large jobs. > However, since this is a cluster-wide setting, we can't properly tune. > It'd be nice if jobs can set a param similar to > mapred.fairscheduler.assignmultiple on submission to better control the task > distribution of a particular job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2905) CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13126358#comment-13126358 ] Todd Lipcon commented on MAPREDUCE-2905: The point isn't to test that this code didn't introduce a regression - the point is to add a regression test for this bug so that _future_ code doesn't regress _this_ fix :) > CapBasedLoadManager incorrectly allows assignment when assignMultiple is true > (was: assignmultiple per job) > --- > > Key: MAPREDUCE-2905 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2905 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/fair-share >Affects Versions: 0.20.2 >Reporter: Jeff Bean > Attachments: MR-2905.patch, MR-2905.patch.2 > > > We encountered a situation where in the same cluster, large jobs benefit from > mapred.fairscheduler.assignmultiple, but small jobs with small numbers of > mappers do not: the mappers all clump to fully occupy just a few nodes, which > causes those nodes to saturate and bottleneck. The desired behavior is to > spread the job across more nodes so that a relatively small job doesn't > saturate any node in the cluster. > Testing has shown that setting mapred.fairscheduler.assignmultiple to false > gives the desired behavior for small jobs, but is unnecessary for large jobs. > However, since this is a cluster-wide setting, we can't properly tune. > It'd be nice if jobs can set a param similar to > mapred.fairscheduler.assignmultiple on submission to better control the task > distribution of a particular job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2905) CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13126350#comment-13126350 ] Todd Lipcon commented on MAPREDUCE-2905: That's exactly the point. If the existing test didn't fail, even though there was a bug, then the existing test wasn't good enough. Hence you need to add a new test or improve the existing one in such a way that the test fails without this bug fix and passes with it. > CapBasedLoadManager incorrectly allows assignment when assignMultiple is true > (was: assignmultiple per job) > --- > > Key: MAPREDUCE-2905 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2905 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/fair-share >Affects Versions: 0.20.2 >Reporter: Jeff Bean > Attachments: MR-2905.patch, MR-2905.patch.2 > > > We encountered a situation where in the same cluster, large jobs benefit from > mapred.fairscheduler.assignmultiple, but small jobs with small numbers of > mappers do not: the mappers all clump to fully occupy just a few nodes, which > causes those nodes to saturate and bottleneck. The desired behavior is to > spread the job across more nodes so that a relatively small job doesn't > saturate any node in the cluster. > Testing has shown that setting mapred.fairscheduler.assignmultiple to false > gives the desired behavior for small jobs, but is unnecessary for large jobs. > However, since this is a cluster-wide setting, we can't properly tune. > It'd be nice if jobs can set a param similar to > mapred.fairscheduler.assignmultiple on submission to better control the task > distribution of a particular job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2905) CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13126275#comment-13126275 ] Jeff Bean commented on MAPREDUCE-2905: -- I don't understand why the existing JUnit test for CapBasedLoadManager isn't sufficient to determine that the patch does not regress? It was sufficient before, so if the patched LoadManager still passes the test on the build, shouldn't that be sufficient now? The JUnit test actually doesn't properly catch this bug. When AssignMultiple is on, CanAssignMap is called in a loop from the fair scheduler with the same TaskStatus object, and the junit test reinstantiates the task status object on each call to CanAssignMap. I can patch the existing JUnit test to reproduce the issue, which would make the existing CapBasedLoadManager fail. Is this sufficient? > CapBasedLoadManager incorrectly allows assignment when assignMultiple is true > (was: assignmultiple per job) > --- > > Key: MAPREDUCE-2905 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2905 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/fair-share >Affects Versions: 0.20.2 >Reporter: Jeff Bean > Attachments: MR-2905.patch > > > We encountered a situation where in the same cluster, large jobs benefit from > mapred.fairscheduler.assignmultiple, but small jobs with small numbers of > mappers do not: the mappers all clump to fully occupy just a few nodes, which > causes those nodes to saturate and bottleneck. The desired behavior is to > spread the job across more nodes so that a relatively small job doesn't > saturate any node in the cluster. > Testing has shown that setting mapred.fairscheduler.assignmultiple to false > gives the desired behavior for small jobs, but is unnecessary for large jobs. > However, since this is a cluster-wide setting, we can't properly tune. > It'd be nice if jobs can set a param similar to > mapred.fairscheduler.assignmultiple on submission to better control the task > distribution of a particular job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2905) CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13125963#comment-13125963 ] Todd Lipcon commented on MAPREDUCE-2905: We do need a junit test. Manual tests are generally insufficient since there's little guarantee they won't regress. > CapBasedLoadManager incorrectly allows assignment when assignMultiple is true > (was: assignmultiple per job) > --- > > Key: MAPREDUCE-2905 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2905 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/fair-share >Affects Versions: 0.20.2 >Reporter: Jeff Bean > Attachments: MR-2905.patch > > > We encountered a situation where in the same cluster, large jobs benefit from > mapred.fairscheduler.assignmultiple, but small jobs with small numbers of > mappers do not: the mappers all clump to fully occupy just a few nodes, which > causes those nodes to saturate and bottleneck. The desired behavior is to > spread the job across more nodes so that a relatively small job doesn't > saturate any node in the cluster. > Testing has shown that setting mapred.fairscheduler.assignmultiple to false > gives the desired behavior for small jobs, but is unnecessary for large jobs. > However, since this is a cluster-wide setting, we can't properly tune. > It'd be nice if jobs can set a param similar to > mapred.fairscheduler.assignmultiple on submission to better control the task > distribution of a particular job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2905) CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13125904#comment-13125904 ] Jeff Bean commented on MAPREDUCE-2905: -- The test is a sleep job with slightly more mappers than slots on a node. Without the patch it clumps on one node. With the patch it evenly spreads. I also have a unit test that I used to discover the bug. Its a little messy. Let me know if you need that. > CapBasedLoadManager incorrectly allows assignment when assignMultiple is true > (was: assignmultiple per job) > --- > > Key: MAPREDUCE-2905 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2905 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/fair-share >Affects Versions: 0.20.2 >Reporter: Jeff Bean > Attachments: MR-2905.patch > > > We encountered a situation where in the same cluster, large jobs benefit from > mapred.fairscheduler.assignmultiple, but small jobs with small numbers of > mappers do not: the mappers all clump to fully occupy just a few nodes, which > causes those nodes to saturate and bottleneck. The desired behavior is to > spread the job across more nodes so that a relatively small job doesn't > saturate any node in the cluster. > Testing has shown that setting mapred.fairscheduler.assignmultiple to false > gives the desired behavior for small jobs, but is unnecessary for large jobs. > However, since this is a cluster-wide setting, we can't properly tune. > It'd be nice if jobs can set a param similar to > mapred.fairscheduler.assignmultiple on submission to better control the task > distribution of a particular job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2905) CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13125843#comment-13125843 ] Harsh J commented on MAPREDUCE-2905: Jeff, I'll leave the final review to people better suited to reviewing FairScheduler patches, but am gonna post some notes on getting this patch to an acceptable state: A few nits, hence: - Patch is mixing spaces and tabs. Follow the coding guidelines and use only spaces. 2 spaces per indent instead of hard tab characters which seem present right now. - If you'd like to get this included upstream, you'll have to re-up the patch with permission grants to ASF. This is doable when you attach a file (look for an option at the bottom -- or perhaps you missed it accidentally). If possible, can we somehow have a test for this? Just asking. > CapBasedLoadManager incorrectly allows assignment when assignMultiple is true > (was: assignmultiple per job) > --- > > Key: MAPREDUCE-2905 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2905 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/fair-share >Affects Versions: 0.20.2 >Reporter: Jeff Bean > Attachments: MR-2905.patch > > > We encountered a situation where in the same cluster, large jobs benefit from > mapred.fairscheduler.assignmultiple, but small jobs with small numbers of > mappers do not: the mappers all clump to fully occupy just a few nodes, which > causes those nodes to saturate and bottleneck. The desired behavior is to > spread the job across more nodes so that a relatively small job doesn't > saturate any node in the cluster. > Testing has shown that setting mapred.fairscheduler.assignmultiple to false > gives the desired behavior for small jobs, but is unnecessary for large jobs. > However, since this is a cluster-wide setting, we can't properly tune. > It'd be nice if jobs can set a param similar to > mapred.fairscheduler.assignmultiple on submission to better control the task > distribution of a particular job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2905) CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13124603#comment-13124603 ] Jeff Bean commented on MAPREDUCE-2905: -- Verified that this fix works. Can someone please look at it? > CapBasedLoadManager incorrectly allows assignment when assignMultiple is true > (was: assignmultiple per job) > --- > > Key: MAPREDUCE-2905 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2905 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/fair-share >Affects Versions: 0.20.2 >Reporter: Jeff Bean > Attachments: MR-2905.patch > > > We encountered a situation where in the same cluster, large jobs benefit from > mapred.fairscheduler.assignmultiple, but small jobs with small numbers of > mappers do not: the mappers all clump to fully occupy just a few nodes, which > causes those nodes to saturate and bottleneck. The desired behavior is to > spread the job across more nodes so that a relatively small job doesn't > saturate any node in the cluster. > Testing has shown that setting mapred.fairscheduler.assignmultiple to false > gives the desired behavior for small jobs, but is unnecessary for large jobs. > However, since this is a cluster-wide setting, we can't properly tune. > It'd be nice if jobs can set a param similar to > mapred.fairscheduler.assignmultiple on submission to better control the task > distribution of a particular job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
