[jira] [Commented] (MAPREDUCE-3951) Tasks are not evenly spread throughout cluster in MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-3951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13584668#comment-13584668 ] Todd Lipcon commented on MAPREDUCE-3951: Seems like so long as the schedulers we expect people to use support this feature, we should be OK. > Tasks are not evenly spread throughout cluster in MR2 > - > > Key: MAPREDUCE-3951 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3951 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: scheduler >Affects Versions: 0.23.0, 0.24.0 >Reporter: Todd Lipcon > > In MR1 (at least with the fair and fifo schedulers), if you submit a job that > needs fewer resources than the cluster can provide, the tasks are spread > relatively evenly across the node. For example, submitting a 100-map job to a > 50-node cluster, each with 10 slots, results in 2 tasks on each machine. In > MR2, however, the tasks would pile up on the first 10 nodes of the cluster, > leaving the other nodes unused. This is highly suboptimal for many use cases. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3951) Tasks are not evenly spread throughout cluster in MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-3951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13583495#comment-13583495 ] Sandy Ryza commented on MAPREDUCE-3951: --- As of now, the fair and capacity schedulers support even spreading, but the fifo scheduler does not. The capacity scheduler is the default scheduler. Should we still try to add it to the fifo scheduler as well? > Tasks are not evenly spread throughout cluster in MR2 > - > > Key: MAPREDUCE-3951 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3951 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: scheduler >Affects Versions: 0.23.0, 0.24.0 >Reporter: Todd Lipcon > > In MR1 (at least with the fair and fifo schedulers), if you submit a job that > needs fewer resources than the cluster can provide, the tasks are spread > relatively evenly across the node. For example, submitting a 100-map job to a > 50-node cluster, each with 10 slots, results in 2 tasks on each machine. In > MR2, however, the tasks would pile up on the first 10 nodes of the cluster, > leaving the other nodes unused. This is highly suboptimal for many use cases. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
