[ https://issues.apache.org/jira/browse/MAPREDUCE-2108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12935124#action_12935124 ]
Scott Chen commented on MAPREDUCE-2108: --------------------------------------- Hey Arun, Thanks. I have made the first patch. Like you mentioned, the slot may be used in may places and we need to have a careful check. I will read the code more carefully. I have a feeling that there are still things needs to be changed. At least in this patch will not change the behavior if we didn't do anything to the scheduler. Scott > Allow TaskScheduler manage number slots on TaskTrackers > ------------------------------------------------------- > > Key: MAPREDUCE-2108 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2108 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/capacity-sched, contrib/fair-share > Affects Versions: 0.22.0 > Reporter: Scott Chen > Assignee: Scott Chen > Fix For: 0.22.0 > > Attachments: MAPREDUCE-2108.txt > > > Currently the map slots and reduce slots are managed by TaskTracker > configuration. > To change the task tracker slots, we need to restart the TaskTrackers. > Also, for a non-uniform cluster, we have to deploy different sets of > configuration. > Now JobTracker holds the CPU and memory status of TaskTrackers > (MAPREDUCE-1218). > So it makes sense to just let JobTracker.taskScheduler decided the number of > slots on each node. > This way we can > 1. Change the number of slots dynamically without restarting TaskTracker > 2. Use different number of slots based on the resource of a TaskTracker > To achieve this, we need to change the logic that we use totalMapSlots and > totalReduceSlots in JobTracker. > I think they are used in WebUI and speculativeCap. > We will need to make JobTracker calculate these numbers from TaskScheduler > and TaskTrackerStatus. > TaskScheduler and TaskTracker can both hold their maximum slots. We pick the > smaller one. > Thoughts? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.