[jira] [Updated] (MAPREDUCE-1684) ClusterStatus can be cached in CapacityTaskScheduler.assignTasks()
[ https://issues.apache.org/jira/browse/MAPREDUCE-1684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-1684: - Target Version/s: 1.2.0 Affects Version/s: 1.0.2 Fix Version/s: (was: 1.1.0) ClusterStatus can be cached in CapacityTaskScheduler.assignTasks() -- Key: MAPREDUCE-1684 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1684 Project: Hadoop Map/Reduce Issue Type: Bug Components: capacity-sched Affects Versions: 1.0.2 Reporter: Amareshwari Sriramadasu Assignee: Koji Noguchi Attachments: mapreduce-1684-v1.0.2-1.patch Currently, CapacityTaskScheduler.assignTasks() calls getClusterStatus() thrice: once in assignTasks(), once in MapTaskScheduler and once in ReduceTaskScheduler. It can be cached in assignTasks() and re-used. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-1684) ClusterStatus can be cached in CapacityTaskScheduler.assignTasks()
[ https://issues.apache.org/jira/browse/MAPREDUCE-1684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-1684: - Resolution: Fixed Fix Version/s: 1.2.0 Status: Resolved (was: Patch Available) ClusterStatus can be cached in CapacityTaskScheduler.assignTasks() -- Key: MAPREDUCE-1684 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1684 Project: Hadoop Map/Reduce Issue Type: Bug Components: capacity-sched Affects Versions: 1.0.2 Reporter: Amareshwari Sriramadasu Assignee: Koji Noguchi Fix For: 1.2.0 Attachments: mapreduce-1684-v1.0.2-1.patch Currently, CapacityTaskScheduler.assignTasks() calls getClusterStatus() thrice: once in assignTasks(), once in MapTaskScheduler and once in ReduceTaskScheduler. It can be cached in assignTasks() and re-used. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-1684) ClusterStatus can be cached in CapacityTaskScheduler.assignTasks()
[ https://issues.apache.org/jira/browse/MAPREDUCE-1684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi updated MAPREDUCE-1684: Attachment: mapreduce-1684-v1.0.2-1.patch bq. Currently, CapacityTaskScheduler.assignTasks() calls getClusterStatus() thrice I think it calls getClusterStatus calls #jobs times in the worst case. For each heartbeat from TaskTracker with some slots available, {noformat} heartbeat -- assignTasks -- addMap/ReduceTasks -- TaskSchedulingMgr.assignTasks -- For each queue : queuesForAssigningTasks) -- getTaskFromQueue(queue) -- For each j : queue.getRunningJobs() -- obtainNewTask -- **getClusterStatus** {noformat} bq. It can be cached in assignTasks() and re-used. Attaching a patch. Would this work? Motivation is, we see getClusterStatus way too often in our jstack holding the global lock. {noformat} IPC Server handler 15 on 50300 daemon prio=10 tid=0x5fc5d800 nid=0x6828 runnable [0x44847000] java.lang.Thread.State: RUNNABLE at org.apache.hadoop.mapred.JobTracker.getClusterStatus(JobTracker.java:4065) - locked 0x2aab6e638bd8 (a org.apache.hadoop.mapred.JobTracker) at org.apache.hadoop.mapred.CapacityTaskScheduler$MapSchedulingMgr.obtainNewTask(CapacityTaskScheduler.java:503) at org.apache.hadoop.mapred.CapacityTaskScheduler$TaskSchedulingMgr.getTaskFromQueue(CapacityTaskScheduler.java:322) at org.apache.hadoop.mapred.CapacityTaskScheduler$TaskSchedulingMgr.assignTasks(CapacityTaskScheduler.java:419) at org.apache.hadoop.mapred.CapacityTaskScheduler$TaskSchedulingMgr.access$500(CapacityTaskScheduler.java:150) at org.apache.hadoop.mapred.CapacityTaskScheduler.addMapTasks(CapacityTaskScheduler.java:1075) at org.apache.hadoop.mapred.CapacityTaskScheduler.assignTasks(CapacityTaskScheduler.java:1044) - locked 0x2aab6e7ffb10 (a org.apache.hadoop.mapred.CapacityTaskScheduler) at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:3398) - locked 0x2aab6e638bd8 (a org.apache.hadoop.mapred.JobTracker) at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382) {noformat} ClusterStatus can be cached in CapacityTaskScheduler.assignTasks() -- Key: MAPREDUCE-1684 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1684 Project: Hadoop Map/Reduce Issue Type: Bug Components: capacity-sched Reporter: Amareshwari Sriramadasu Attachments: mapreduce-1684-v1.0.2-1.patch Currently, CapacityTaskScheduler.assignTasks() calls getClusterStatus() thrice: once in assignTasks(), once in MapTaskScheduler and once in ReduceTaskScheduler. It can be cached in assignTasks() and re-used. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-1684) ClusterStatus can be cached in CapacityTaskScheduler.assignTasks()
[ https://issues.apache.org/jira/browse/MAPREDUCE-1684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated MAPREDUCE-1684: --- Fix Version/s: 1.1.0 Assignee: Koji Noguchi Status: Patch Available (was: Open) ClusterStatus can be cached in CapacityTaskScheduler.assignTasks() -- Key: MAPREDUCE-1684 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1684 Project: Hadoop Map/Reduce Issue Type: Bug Components: capacity-sched Reporter: Amareshwari Sriramadasu Assignee: Koji Noguchi Fix For: 1.1.0 Attachments: mapreduce-1684-v1.0.2-1.patch Currently, CapacityTaskScheduler.assignTasks() calls getClusterStatus() thrice: once in assignTasks(), once in MapTaskScheduler and once in ReduceTaskScheduler. It can be cached in assignTasks() and re-used. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira