[jira] Commented: (MAPREDUCE-944) Extend FairShare scheduler to fair-share memory usage in the cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12753503#action_12753503 ] dhruba borthakur commented on MAPREDUCE-944: This is not really a bug fix. It is mostly a refactoring of the code so that a certain functionality is separated out more cleanly. HADOOP-961 will benefit from this JIRA and it will have quite a few unit tests associated with it. Extend FairShare scheduler to fair-share memory usage in the cluster Key: MAPREDUCE-944 URL: https://issues.apache.org/jira/browse/MAPREDUCE-944 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/fair-share Reporter: dhruba borthakur Assignee: dhruba borthakur Fix For: 0.21.0 Attachments: LoadManager.txt, LoadManager2.txt The FairShare Scheduler has an extensible LoadManager API to regulate allocating new tasks on a particular TaskTracker. In similar lines, it would be nice if the FairShare Scheduler can have a pluggable policy to regulate new tasks from a particular job. This will allow one to skip scheduling tasks of a job that is eating a large percentage of memory in the cluster, i.e. fair-share of memory resources among jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-944) Extend FairShare scheduler to fair-share memory usage in the cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12753516#action_12753516 ] Hudson commented on MAPREDUCE-944: -- Integrated in Hadoop-Mapreduce-trunk-Commit #28 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/28/]) . Extend the LoadManager API of the fair-share scheduler to support regulating tasks for a job based on resources currently in use by that job. (dhruba) Extend FairShare scheduler to fair-share memory usage in the cluster Key: MAPREDUCE-944 URL: https://issues.apache.org/jira/browse/MAPREDUCE-944 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/fair-share Reporter: dhruba borthakur Assignee: dhruba borthakur Fix For: 0.21.0 Attachments: LoadManager.txt, LoadManager2.txt The FairShare Scheduler has an extensible LoadManager API to regulate allocating new tasks on a particular TaskTracker. In similar lines, it would be nice if the FairShare Scheduler can have a pluggable policy to regulate new tasks from a particular job. This will allow one to skip scheduling tasks of a job that is eating a large percentage of memory in the cluster, i.e. fair-share of memory resources among jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-944) Extend FairShare scheduler to fair-share memory usage in the cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12752365#action_12752365 ] Hadoop QA commented on MAPREDUCE-944: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12418880/LoadManager2.txt against trunk revision 812287. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/44/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/44/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/44/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/44/console This message is automatically generated. Extend FairShare scheduler to fair-share memory usage in the cluster Key: MAPREDUCE-944 URL: https://issues.apache.org/jira/browse/MAPREDUCE-944 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/fair-share Reporter: dhruba borthakur Fix For: 0.21.0 Attachments: LoadManager.txt, LoadManager2.txt The FairShare Scheduler has an extensible LoadManager API to regulate allocating new tasks on a particular TaskTracker. In similar lines, it would be nice if the FairShare Scheduler can have a pluggable policy to regulate new tasks from a particular job. This will allow one to skip scheduling tasks of a job that is eating a large percentage of memory in the cluster, i.e. fair-share of memory resources among jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-944) Extend FairShare scheduler to fair-share memory usage in the cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12752720#action_12752720 ] Scott Chen commented on MAPREDUCE-944: -- I wonder where why there are test errors. canLaunchTask only returns true so it shouldn't change anything. I have applied this patch and run ant clean test. It shows no test error. Extend FairShare scheduler to fair-share memory usage in the cluster Key: MAPREDUCE-944 URL: https://issues.apache.org/jira/browse/MAPREDUCE-944 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/fair-share Reporter: dhruba borthakur Assignee: dhruba borthakur Fix For: 0.21.0 Attachments: LoadManager.txt, LoadManager2.txt The FairShare Scheduler has an extensible LoadManager API to regulate allocating new tasks on a particular TaskTracker. In similar lines, it would be nice if the FairShare Scheduler can have a pluggable policy to regulate new tasks from a particular job. This will allow one to skip scheduling tasks of a job that is eating a large percentage of memory in the cluster, i.e. fair-share of memory resources among jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-944) Extend FairShare scheduler to fair-share memory usage in the cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12752105#action_12752105 ] Vinod K V commented on MAPREDUCE-944: - I see in the patch attached that only one concrete implementation CapBasedLoadManager is done for the LoadManager which in turn doesn't take into account any resource usage. I guess you are planning a proper implementation for this feature regarding fair-share of memory usage in another JIRA. Some points still not dealt with in this JIRA. I bring about these points so as to know if you are thinking or have already thought anything about this. - Job configuration about how users specify the resource usage. Some memory related configuration properties are added to the framework while working for memory monitoring on TTs as well as memory usage based scheduling in CapacityTaskScheduler. You may want to reuse some/all of it. - Capturing the scheduling decisions involved when we are not able to find a task from a Schedulable because of lack of resources on a given TaskTasker. Regarding the latter, the current patch just returns null, which is similar to the decision CapacityTaskScheduler used to take in previous versions - i.e. block the TT till it can be given a task from the job at the head of the queue/pool. Sometime back, we investigated how this approach works with FairScheduler and realized some important implications. For e.g, because the order of jobs might change significantly in consecutive iterations of FairScheduler, just returning null may not work at all. Eventually we may end up waiting for a long time if significant number of jobs ask for high amount of resources. Thoughts? Extend FairShare scheduler to fair-share memory usage in the cluster Key: MAPREDUCE-944 URL: https://issues.apache.org/jira/browse/MAPREDUCE-944 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/fair-share Reporter: dhruba borthakur Attachments: LoadManager.txt The FairShare Scheduler has an extensible LoadManager API to regulate allocating new tasks on a particular TaskTracker. In similar lines, it would be nice if the FairShare Scheduler can have a pluggable policy to regulate new tasks from a particular job. This will allow one to skip scheduling tasks of a job that is eating a large percentage of memory in the cluster, i.e. fair-share of memory resources among jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-944) Extend FairShare scheduler to fair-share memory usage in the cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12751777#action_12751777
]
Matei Zaharia commented on MAPREDUCE-944:
-
+1 looks good to me, except for one minor detail: I'd change the JavaDoc on
canLaunchTask from Can a given {...@link TaskTracker} run another new task?
to Can a given {...@link TaskTracker} run another new task from a given job?
This method is provided for use by LoadManagers that take into account jobs'
individual resource needs when placing tasks. This way the purpose of the
method, and its difference from canAssignMap/canAssignReduce, is better
explained. It might also be good to look at the comments for
canAssignMap/Reduce and make them say that these methods are used to check
whether any attempt to find a job with maps/reduces should be made.
Extend FairShare scheduler to fair-share memory usage in the cluster
Key: MAPREDUCE-944
URL: https://issues.apache.org/jira/browse/MAPREDUCE-944
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: contrib/fair-share
Reporter: dhruba borthakur
Attachments: LoadManager.txt
The FairShare Scheduler has an extensible LoadManager API to regulate
allocating new tasks on a particular TaskTracker. In similar lines, it would
be nice if the FairShare Scheduler can have a pluggable policy to regulate
new tasks from a particular job. This will allow one to skip scheduling tasks
of a job that is eating a large percentage of memory in the cluster, i.e.
fair-share of memory resources among jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-944) Extend FairShare scheduler to fair-share memory usage in the cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12750822#action_12750822 ] dhruba borthakur commented on MAPREDUCE-944: +1 to Matie's proposal. Extend FairShare scheduler to fair-share memory usage in the cluster Key: MAPREDUCE-944 URL: https://issues.apache.org/jira/browse/MAPREDUCE-944 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/fair-share Reporter: dhruba borthakur The FairShare Scheduler has an extensible LoadManager API to regulate allocating new tasks on a particular TaskTracker. In similar lines, it would be nice if the FairShare Scheduler can have a pluggable policy to regulate new tasks from a particular job. This will allow one to skip scheduling tasks of a job that is eating a large percentage of memory in the cluster, i.e. fair-share of memory resources among jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
