[
https://issues.apache.org/jira/browse/HADOOP-4943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12663088#action_12663088
]
Matei Zaharia commented on HADOOP-4943:
---------------------------------------
I just committed this. Thanks Zheng!
I also looked at the default and capacity schedulers, but the default scheduler
already seems to have this logic as part of the patch for HADOOP-3136, and the
capacity scheduler doesn't try to do this kind of load balancing when there are
fewer tasks than slots so I think this should be a separate JIRA.
> fair share scheduler does not utilize all slots if the task trackers are
> configured heterogeneously
> ---------------------------------------------------------------------------------------------------
>
> Key: HADOOP-4943
> URL: https://issues.apache.org/jira/browse/HADOOP-4943
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/fair-share
> Affects Versions: 0.19.0
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Fix For: 0.19.1
>
> Attachments: HADOOP-4943-1.patch, hadoop-4943-2.patch
>
>
> There is some code in the fairshare scheduler that tries to make the load
> across the whole cluster the same.
> That piece of code will break if the task trackers are configured
> differently. Basically, we will stop assigning more tasks to tasks trackers
> that have tasks above the cluster average, but we may still want to do that
> because other task trackers may have less slots.
> We should change the code to maintain a cluster-wide slot usage percentage
> (instead of absolute number of slot usage) to make sure the load is evenly
> distributed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
