[jira] [Commented] (MAPREDUCE-279) Map-Reduce 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129523#comment-13129523 ] Hudson commented on MAPREDUCE-279: -- Integrated in Hadoop-Mapreduce-0.23-Commit #17 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/17/]) MAPREDUCE-279. Adding a changelog to branch-0.23. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185489 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt Map-Reduce 2.0 -- Key: MAPREDUCE-279 URL: https://issues.apache.org/jira/browse/MAPREDUCE-279 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: Arun C Murthy Fix For: 0.23.0 Attachments: MR-279-script-20110817.sh, MR-279-script-final.sh, MR-279-script.sh, MR-279-script.sh, MR-279.patch, MR-279.patch, MR-279.sh, MR-279_MR_files_to_move-20110817.txt, MR-279_MR_files_to_move.txt, MR-279_MR_files_to_move.txt, MapReduce_NextGen_Architecture.pdf, NodeManager.gv, NodeManager.png, ResourceManager.gv, ResourceManager.png, capacity-scheduler-dark-theme.png, hadoop_contributors_meet_07_01_2011.pdf, multi-column-stable-sort-default-theme.png, post-move-patch-20110817.2.txt, post-move-patch-final.txt, post-move.patch, post-move.patch, post-move.patch, yarn-state-machine.job.dot, yarn-state-machine.job.png, yarn-state-machine.task-attempt.dot, yarn-state-machine.task-attempt.png, yarn-state-machine.task.dot, yarn-state-machine.task.png Re-factor MapReduce into a generic resource scheduler and a per-job, user-defined component that manages the application execution. Check it out by following [the instructions|http://goo.gl/rSJJC]. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2702: - Release Note: Enhance OutputCommitter and FileOutputCommitter to allow for recover of tasks across job restart. Editorial pass over 0.23 content. [MR-279] OutputCommitter changes for MR Application Master recovery --- Key: MAPREDUCE-2702 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, mr2702_v1.patch, mr2702_v2.patch, mr2702_v3.patch, mr2702_v4.patch In MR AM recovers from a crash, it only reruns the non completed tasks. The completed tasks (along with their output, if any) needs to be recovered from the previous life. This would require some changes in OutputCommitter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2708) [MR-279] Design and implement MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sharad Agarwal updated MAPREDUCE-2708: -- Status: Patch Available (was: Open) [MR-279] Design and implement MR Application Master recovery Key: MAPREDUCE-2708 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2708 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Priority: Blocker Fix For: 0.23.0 Attachments: mr2708_v1.patch, mr2708_v2.patch Design recovery of MR AM from crashes/node failures. The running job should recover from the state it left off. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-901) Move Framework Counters into a TaskMetric structure
[ https://issues.apache.org/jira/browse/MAPREDUCE-901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-901: Release Note: Efficient implementation of MapReduce framework counters. Move Framework Counters into a TaskMetric structure --- Key: MAPREDUCE-901 URL: https://issues.apache.org/jira/browse/MAPREDUCE-901 Project: Hadoop Map/Reduce Issue Type: Improvement Components: task Affects Versions: 0.21.0 Reporter: Owen O'Malley Assignee: Luke Lu Fix For: 0.23.0 Attachments: 901_1.patch, 901_1.patch, FrameworkCounterGroup.java, MAPREDUCE-901.patch, MAPREDUCE-901.patch, MAPREDUCE-901.patch, MAPREDUCE-901.patch_2, mr-901-trunk-v1.patch I think we should move all of the Counters that the framework updates into a single class called TaskMetrics. TaskMetrics would have specific fields for each of the metrics like input records, input bytes, output records, etc. It would both reduce the serialized size of the heartbeats (by shrinking the Counters down to just the user's counters) and decrease the latency for updates to the JobTracker (since Counters are sent at most 1/minute instead of 1/heartbeat). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-1220) Implement an in-cluster LocalJobRunner
[ https://issues.apache.org/jira/browse/MAPREDUCE-1220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-1220. -- Resolution: Fixed Fix Version/s: 0.23.0 Release Note: An efficient implementation of small jobs by running all tasks in the same JVM, there-by affecting lower latency. Fixed via MAPREDUCE-279. Implement an in-cluster LocalJobRunner -- Key: MAPREDUCE-1220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1220 Project: Hadoop Map/Reduce Issue Type: New Feature Components: client, jobtracker Reporter: Arun C Murthy Assignee: Greg Roelofs Fix For: 0.23.0 Attachments: MAPREDUCE-1220_yhadoop20.patch, MR-1220.v1.trunk-hadoop-common.Progress-dumper.patch.txt, MR-1220.v10e-v11c-v12b.ytrunk-hadoop-mapreduce.delta.patch.txt, MR-1220.v13.ytrunk-hadoop-mapreduce.delta.patch.txt, MR-1220.v14b.ytrunk-hadoop-mapreduce.delta.patch.txt, MR-1220.v15.ytrunk-hadoop-mapreduce.delta.patch.txt, MR-1220.v1b.sshot-02-jobdetails.jsp.png, MR-1220.v1b.sshot-03-jobdetailshistory.jsp.png, MR-1220.v2.trunk-hadoop-mapreduce.patch.txt, MR-1220.v2.trunk-hadoop-mapreduce.patch.txt, MR-1220.v2b.sshot-01-jobtracker.jsp.png, MR-1220.v6.ytrunk-hadoop-mapreduce.patch.txt, MR-1220.v7.ytrunk-hadoop-mapreduce.delta.patch.txt, MR-1220.v8b.ytrunk-hadoop-mapreduce.delta.patch.txt, MR-1220.v9c.ytrunk-hadoop-mapreduce.delta.patch.txt Currently very small map-reduce jobs suffer from latency issues due to overheads in Hadoop Map-Reduce such as scheduling, jvm startup etc. We've periodically tried to optimize all parts of framework to achieve lower latencies. I'd like to turn the problem around a little bit. I propose we allow very small jobs to run as a single task job with multiple maps and reduces i.e. similar to our current implementation of the LocalJobRunner. Thus, under certain conditions (maybe user-set configuration, or if input data is small i.e. less a DFS blocksize) we could launch a special task which will run all maps in a serial manner, followed by the reduces. This would really help small jobs achieve significantly smaller latencies, thanks to lesser scheduling overhead, jvm startup, lack of shuffle over the network etc. This would be a huge benefit, especially on large clusters, to small Hive/Pig queries. Thoughts? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-1220) Implement an in-cluster LocalJobRunner
[ https://issues.apache.org/jira/browse/MAPREDUCE-1220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reopened MAPREDUCE-1220: -- Implement an in-cluster LocalJobRunner -- Key: MAPREDUCE-1220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1220 Project: Hadoop Map/Reduce Issue Type: New Feature Components: client, jobtracker Reporter: Arun C Murthy Assignee: Greg Roelofs Fix For: 0.23.0 Attachments: MAPREDUCE-1220_yhadoop20.patch, MR-1220.v1.trunk-hadoop-common.Progress-dumper.patch.txt, MR-1220.v10e-v11c-v12b.ytrunk-hadoop-mapreduce.delta.patch.txt, MR-1220.v13.ytrunk-hadoop-mapreduce.delta.patch.txt, MR-1220.v14b.ytrunk-hadoop-mapreduce.delta.patch.txt, MR-1220.v15.ytrunk-hadoop-mapreduce.delta.patch.txt, MR-1220.v1b.sshot-02-jobdetails.jsp.png, MR-1220.v1b.sshot-03-jobdetailshistory.jsp.png, MR-1220.v2.trunk-hadoop-mapreduce.patch.txt, MR-1220.v2.trunk-hadoop-mapreduce.patch.txt, MR-1220.v2b.sshot-01-jobtracker.jsp.png, MR-1220.v6.ytrunk-hadoop-mapreduce.patch.txt, MR-1220.v7.ytrunk-hadoop-mapreduce.delta.patch.txt, MR-1220.v8b.ytrunk-hadoop-mapreduce.delta.patch.txt, MR-1220.v9c.ytrunk-hadoop-mapreduce.delta.patch.txt Currently very small map-reduce jobs suffer from latency issues due to overheads in Hadoop Map-Reduce such as scheduling, jvm startup etc. We've periodically tried to optimize all parts of framework to achieve lower latencies. I'd like to turn the problem around a little bit. I propose we allow very small jobs to run as a single task job with multiple maps and reduces i.e. similar to our current implementation of the LocalJobRunner. Thus, under certain conditions (maybe user-set configuration, or if input data is small i.e. less a DFS blocksize) we could launch a special task which will run all maps in a serial manner, followed by the reduces. This would really help small jobs achieve significantly smaller latencies, thanks to lesser scheduling overhead, jvm startup, lack of shuffle over the network etc. This would be a huge benefit, especially on large clusters, to small Hive/Pig queries. Thoughts? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2708) [MR-279] Design and implement MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129550#comment-13129550 ] Hadoop QA commented on MAPREDUCE-2708: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12499261/mr2708_v2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1052//console This message is automatically generated. [MR-279] Design and implement MR Application Master recovery Key: MAPREDUCE-2708 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2708 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Priority: Blocker Fix For: 0.23.0 Attachments: mr2708_v1.patch, mr2708_v2.patch Design recovery of MR AM from crashes/node failures. The running job should recover from the state it left off. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-1220) Implement an in-cluster LocalJobRunner
[ https://issues.apache.org/jira/browse/MAPREDUCE-1220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-1220. -- Resolution: Duplicate Duplicate of MAPREDUCE-2405. Implement an in-cluster LocalJobRunner -- Key: MAPREDUCE-1220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1220 Project: Hadoop Map/Reduce Issue Type: New Feature Components: client, jobtracker Reporter: Arun C Murthy Assignee: Greg Roelofs Fix For: 0.23.0 Attachments: MAPREDUCE-1220_yhadoop20.patch, MR-1220.v1.trunk-hadoop-common.Progress-dumper.patch.txt, MR-1220.v10e-v11c-v12b.ytrunk-hadoop-mapreduce.delta.patch.txt, MR-1220.v13.ytrunk-hadoop-mapreduce.delta.patch.txt, MR-1220.v14b.ytrunk-hadoop-mapreduce.delta.patch.txt, MR-1220.v15.ytrunk-hadoop-mapreduce.delta.patch.txt, MR-1220.v1b.sshot-02-jobdetails.jsp.png, MR-1220.v1b.sshot-03-jobdetailshistory.jsp.png, MR-1220.v2.trunk-hadoop-mapreduce.patch.txt, MR-1220.v2.trunk-hadoop-mapreduce.patch.txt, MR-1220.v2b.sshot-01-jobtracker.jsp.png, MR-1220.v6.ytrunk-hadoop-mapreduce.patch.txt, MR-1220.v7.ytrunk-hadoop-mapreduce.delta.patch.txt, MR-1220.v8b.ytrunk-hadoop-mapreduce.delta.patch.txt, MR-1220.v9c.ytrunk-hadoop-mapreduce.delta.patch.txt Currently very small map-reduce jobs suffer from latency issues due to overheads in Hadoop Map-Reduce such as scheduling, jvm startup etc. We've periodically tried to optimize all parts of framework to achieve lower latencies. I'd like to turn the problem around a little bit. I propose we allow very small jobs to run as a single task job with multiple maps and reduces i.e. similar to our current implementation of the LocalJobRunner. Thus, under certain conditions (maybe user-set configuration, or if input data is small i.e. less a DFS blocksize) we could launch a special task which will run all maps in a serial manner, followed by the reduces. This would really help small jobs achieve significantly smaller latencies, thanks to lesser scheduling overhead, jvm startup, lack of shuffle over the network etc. This would be a huge benefit, especially on large clusters, to small Hive/Pig queries. Thoughts? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2405) MR-279: Implement uber-AppMaster (in-cluster LocalJobRunner for MRv2)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2405: - Release Note: An efficient implementation of small jobs by running all tasks in the MR ApplicationMaster JVM, there-by affecting lower latency. MR-279: Implement uber-AppMaster (in-cluster LocalJobRunner for MRv2) - Key: MAPREDUCE-2405 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2405 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: Mahadev konar Assignee: Greg Roelofs Fix For: 0.23.0 Attachments: MR-2405-MR-1220-yarn.v10.MR-279-hadoop-yarn.patch.txt, MR-2405-MR-1220-yarn.v11.MR-279-hadoop-yarn.patch.txt, MR-2405-MR-1220-yarn.v8.MR-279-hadoop-yarn.patch.txt Port MAPREDUCE-1220 to MRv2. This is an optimization for small jobs wherein all tasks run on the same node in the same JVM/container. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-824) Support a hierarchy of queues in the capacity scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-824: Release Note: Support hierarchical queues in the CapacityScheduler to allow for more predictable sharing of cluster resources. Editorial pass over 0.21/0.22/0.23 content for hadoop-0.23. Support a hierarchy of queues in the capacity scheduler --- Key: MAPREDUCE-824 URL: https://issues.apache.org/jira/browse/MAPREDUCE-824 Project: Hadoop Map/Reduce Issue Type: New Feature Components: contrib/capacity-sched Reporter: Hemanth Yamijala Assignee: rahul k singh Fix For: 0.21.0 Attachments: HADOOP-824-1.patch, HADOOP-824-2.patch, HADOOP-824-3.patch, HADOOP-824-4.patch, HADOOP-824-5.patch, MAPREDUCE-824-6.patch, MAPREDUCE-824-7.patch Currently in Capacity Scheduler, cluster capacity is divided among the queues based on the queue capacity. These queues typically represent an organization and the capacity of the queue represents the capacity the organization is entitled to. Most organizations are large and need to divide their capacity among sub-organizations they have. Or they may want to divide the capacity based on a category or type of jobs they run. This JIRA covers the requirements and other details to provide the above feature. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-1220) Implement an in-cluster LocalJobRunner
[ https://issues.apache.org/jira/browse/MAPREDUCE-1220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated MAPREDUCE-1220: --- Release Note: An efficient implementation of small jobs by running all tasks in the same JVM, there-by effecting lower latency. (was: An efficient implementation of small jobs by running all tasks in the same JVM, there-by affecting lower latency.) Implement an in-cluster LocalJobRunner -- Key: MAPREDUCE-1220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1220 Project: Hadoop Map/Reduce Issue Type: New Feature Components: client, jobtracker Reporter: Arun C Murthy Assignee: Greg Roelofs Fix For: 0.23.0 Attachments: MAPREDUCE-1220_yhadoop20.patch, MR-1220.v1.trunk-hadoop-common.Progress-dumper.patch.txt, MR-1220.v10e-v11c-v12b.ytrunk-hadoop-mapreduce.delta.patch.txt, MR-1220.v13.ytrunk-hadoop-mapreduce.delta.patch.txt, MR-1220.v14b.ytrunk-hadoop-mapreduce.delta.patch.txt, MR-1220.v15.ytrunk-hadoop-mapreduce.delta.patch.txt, MR-1220.v1b.sshot-02-jobdetails.jsp.png, MR-1220.v1b.sshot-03-jobdetailshistory.jsp.png, MR-1220.v2.trunk-hadoop-mapreduce.patch.txt, MR-1220.v2.trunk-hadoop-mapreduce.patch.txt, MR-1220.v2b.sshot-01-jobtracker.jsp.png, MR-1220.v6.ytrunk-hadoop-mapreduce.patch.txt, MR-1220.v7.ytrunk-hadoop-mapreduce.delta.patch.txt, MR-1220.v8b.ytrunk-hadoop-mapreduce.delta.patch.txt, MR-1220.v9c.ytrunk-hadoop-mapreduce.delta.patch.txt Currently very small map-reduce jobs suffer from latency issues due to overheads in Hadoop Map-Reduce such as scheduling, jvm startup etc. We've periodically tried to optimize all parts of framework to achieve lower latencies. I'd like to turn the problem around a little bit. I propose we allow very small jobs to run as a single task job with multiple maps and reduces i.e. similar to our current implementation of the LocalJobRunner. Thus, under certain conditions (maybe user-set configuration, or if input data is small i.e. less a DFS blocksize) we could launch a special task which will run all maps in a serial manner, followed by the reduces. This would really help small jobs achieve significantly smaller latencies, thanks to lesser scheduling overhead, jvm startup, lack of shuffle over the network etc. This would be a huge benefit, especially on large clusters, to small Hive/Pig queries. Thoughts? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-279) Map-Reduce 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-279: Release Note: MapReduce has undergone a complete re-haul in hadoop-0.23 and we now have, what we call, MapReduce 2.0 (MRv2). The fundamental idea of MRv2 is to split up the two major functionalities of the JobTracker, resource management and job scheduling/monitoring, into separate daemons. The idea is to have a global ResourceManager (RM) and per-application ApplicationMaster (AM). An application is either a single job in the classical sense of Map-Reduce jobs or a DAG of jobs. The ResourceManager and per-node slave, the NodeManager (NM), form the data-computation framework. The ResourceManager is the ultimate authority that arbitrates resources among all the applications in the system. The per-application ApplicationMaster is, in effect, a framework specific library and is tasked with negotiating resources from the ResourceManager and working with the NodeManager(s) to execute and monitor the tasks. The ResourceManager has two main components: * Scheduler (S) * ApplicationsManager (ASM) The Scheduler is responsible for allocating resources to the various running applications subject to familiar constraints of capacities, queues etc. The Scheduler is pure scheduler in the sense that it performs no monitoring or tracking of status for the application. Also, it offers no guarantees on restarting failed tasks either due to application failure or hardware failures. The Scheduler performs its scheduling function based the resource requirements of the applications; it does so based on the abstract notion of a Resource Container which incorporates elements such as memory, cpu, disk, network etc. The Scheduler has a pluggable policy plug-in, which is responsible for partitioning the cluster resources among the various queues, applications etc. The current Map-Reduce schedulers such as the CapacityScheduler and the FairScheduler would be some examples of the plug-in. The CapacityScheduler supports hierarchical queues to allow for more predictable sharing of cluster resources. The ApplicationsManager is responsible for accepting job-submissions, negotiating the first container for executing the application specific ApplicationMaster and provides the service for restarting the ApplicationMaster container on failure. The NodeManager is the per-machine framework agent who is responsible for launching the applications' containers, monitoring their resource usage (cpu, memory, disk, network) and reporting the same to the Scheduler. The per-application ApplicationMaster has the responsibility of negotiating appropriate resource containers from the Scheduler, tracking their status and monitoring for progress. was: MapReduce has undergone a complete re-haul in hadoop-0.23 and we now have, what we call, MapReduce 2.0 (MRv2). The fundamental idea of MRv2 is to split up the two major functionalities of the JobTracker, resource management and job scheduling/monitoring, into separate daemons. The idea is to have a global ResourceManager (RM) and per-application ApplicationMaster (AM). An application is either a single job in the classical sense of Map-Reduce jobs or a DAG of jobs. The ResourceManager and per-node slave, the NodeManager (NM), form the data-computation framework. The ResourceManager is the ultimate authority that arbitrates resources among all the applications in the system. The per-application ApplicationMaster is, in effect, a framework specific library and is tasked with negotiating resources from the ResourceManager and working with the NodeManager(s) to execute and monitor the tasks. The ResourceManager has two main components: * Scheduler (S) * ApplicationsManager (ASM) The Scheduler is responsible for allocating resources to the various running applications subject to familiar constraints of capacities, queues etc. The Scheduler is pure scheduler in the sense that it performs no monitoring or tracking of status for the application. Also, it offers no guarantees on restarting failed tasks either due to application failure or hardware failures. The Scheduler performs its scheduling function based the resource requirements of the applications; it does so based on the abstract notion of a Resource Container which incorporates elements such as memory, cpu, disk, network etc. The Scheduler has a pluggable policy plug-in, which is responsible for partitioning the cluster resources among the various queues, applications etc. The current Map-Reduce schedulers such as the CapacityScheduler and the FairScheduler would be some examples of the plug-in. The ApplicationsManager is responsible for accepting job-submissions, negotiating the first container for executing the application specific ApplicationMaster and provides the service for
[jira] [Updated] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir after job completion
[
https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahadev konar updated MAPREDUCE-2762:
-
Attachment: MAPREDUCE-2762.patch
Updated patch. Tested it on a cluster.
[MR-279] - Cleanup staging dir after job completion
---
Key: MAPREDUCE-2762
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Ramya Sunil
Assignee: Mahadev konar
Priority: Blocker
Fix For: 0.23.0
Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch,
MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch,
MAPREDUCE-2762.patch
The files created under the staging dir have to be deleted after job
completion. Currently, all job.* files remain forever in the
${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir after job completion
[
https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahadev konar updated MAPREDUCE-2762:
-
Status: Patch Available (was: Open)
[MR-279] - Cleanup staging dir after job completion
---
Key: MAPREDUCE-2762
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Ramya Sunil
Assignee: Mahadev konar
Priority: Blocker
Fix For: 0.23.0
Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch,
MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch,
MAPREDUCE-2762.patch
The files created under the staging dir have to be deleted after job
completion. Currently, all job.* files remain forever in the
${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-777) A method for finding and tracking jobs from the new API
[ https://issues.apache.org/jira/browse/MAPREDUCE-777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-777: Release Note: Enhance the Context Objects api to add features to find and track jobs. A method for finding and tracking jobs from the new API --- Key: MAPREDUCE-777 URL: https://issues.apache.org/jira/browse/MAPREDUCE-777 Project: Hadoop Map/Reduce Issue Type: New Feature Components: client Reporter: Owen O'Malley Assignee: Amareshwari Sriramadasu Fix For: 0.21.0 Attachments: m-777.patch, patch-777-1.txt, patch-777-10.txt, patch-777-11.txt, patch-777-12.txt, patch-777-13.txt, patch-777-14.txt, patch-777-15.txt, patch-777-16.txt, patch-777-17.txt, patch-777-2.txt, patch-777-3.txt, patch-777-4.txt, patch-777-5.txt, patch-777-6.txt, patch-777-7.txt, patch-777-8.txt, patch-777-9.txt, patch-777.txt We need to create a replacement interface for the JobClient API in the new interface. In particular, the user needs to be able to query and track jobs that were launched by other processes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir after job completion
[
https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129569#comment-13129569
]
Hadoop QA commented on MAPREDUCE-2762:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12499513/MAPREDUCE-2762.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 1713 javac compiler warnings (more
than the trunk's current 1711 warnings).
-1 findbugs. The patch appears to introduce 160 new Findbugs (version
1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.mapreduce.TestJobMonitorAndPrint
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1053//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1053//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1053//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1053//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1053//console
This message is automatically generated.
[MR-279] - Cleanup staging dir after job completion
---
Key: MAPREDUCE-2762
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Ramya Sunil
Assignee: Mahadev konar
Priority: Blocker
Fix For: 0.23.0
Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch,
MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch,
MAPREDUCE-2762.patch
The files created under the staging dir have to be deleted after job
completion. Currently, all job.* files remain forever in the
${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3118) Backport Gridmix and Rumen features from trunk to Hadoop 0.20 security branch
[ https://issues.apache.org/jira/browse/MAPREDUCE-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129587#comment-13129587 ] Ravi Gummadi commented on MAPREDUCE-3118: - Please come back if there are any issues. We are planning to commit this soon. Backport Gridmix and Rumen features from trunk to Hadoop 0.20 security branch - Key: MAPREDUCE-3118 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3118 Project: Hadoop Map/Reduce Issue Type: New Feature Components: contrib/gridmix, tools/rumen Affects Versions: 0.20.206.0 Reporter: Ravi Gummadi Assignee: Ravi Gummadi Fix For: 0.20.206.0 Attachments: gridmix_rumen_backports.v2.4.patch, gridmix_rumen_backports.v2.5.patch, gridmix_rumen_backports.v2.6.patch Backporting all the features and bugfixes that went into gridmix and rumen of trunk to hadoop 0.20 security branch. This will enable using all these gridmix features and run gridmix/rumen on the history logs of 0.20 security branch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3193) NextGen Mapreduce framework is not able to read the job input recursively.Input is read only for one folder level deep
[
https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129630#comment-13129630
]
Devaraj K commented on MAPREDUCE-3193:
--
Here is the problem. In FileInputFormat.listStatus, It considers the
files/directories in one nested level and takes every thing as file. Finally it
creates splits with directories and fails the task.
{code:title=FileInputFormat.java|borderStyle=solid}
for (int i=0; i dirs.length; ++i) {
Path p = dirs[i];
FileSystem fs = p.getFileSystem(job.getConfiguration());
FileStatus[] matches = fs.globStatus(p, inputFilter);
if (matches == null) {
errors.add(new IOException(Input path does not exist: + p));
} else if (matches.length == 0) {
errors.add(new IOException(Input Pattern + p + matches 0 files));
} else {
for (FileStatus globStat: matches) {
if (globStat.isDirectory()) {
for(FileStatus stat: fs.listStatus(globStat.getPath(),
inputFilter)) {
result.add(stat);
}
} else {
result.add(globStat);
}
}
}
}
{code}
NextGen Mapreduce framework is not able to read the job input
recursively.Input is read only for one folder level deep
--
Key: MAPREDUCE-3193
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Ramgopal N
java.io.FileNotFoundException is thrown,if input file is more than one folder
level deep and the job is getting failed.
Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3199) TestJobMonitorAndPrint is broken on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-3199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli reassigned MAPREDUCE-3199: -- Assignee: Vinod Kumar Vavilapalli TestJobMonitorAndPrint is broken on trunk - Key: MAPREDUCE-3199 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3199 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.0 I bisected this down to MAPREDUCE-3003 changes. The parent project for client-core changed to hadoop-project which doesn't have the log4j configuration unlike the previous parent hadoop-mapreduce-client. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3199) TestJobMonitorAndPrint is broken on trunk
TestJobMonitorAndPrint is broken on trunk - Key: MAPREDUCE-3199 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3199 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Fix For: 0.23.0 I bisected this down to MAPREDUCE-3003 changes. The parent project for client-core changed to hadoop-project which doesn't have the log4j configuration unlike the previous parent hadoop-mapreduce-client. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3199) TestJobMonitorAndPrint is broken on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-3199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3199: --- Status: Patch Available (was: Open) Set the affecting version as 0.23 'coz we'll need this once MAPREDUCE-3003 goes into 0.23 as well. TestJobMonitorAndPrint is broken on trunk - Key: MAPREDUCE-3199 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3199 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.0 Attachments: MAPREDUCE-3199-20111018.txt I bisected this down to MAPREDUCE-3003 changes. The parent project for client-core changed to hadoop-project which doesn't have the log4j configuration unlike the previous parent hadoop-mapreduce-client. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3199) TestJobMonitorAndPrint is broken on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-3199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3199: --- Attachment: MAPREDUCE-3199-20111018.txt Attaching patch that should fix this. Also removed stale basedir properties and duplicate plugin configurations from various pom files. TestJobMonitorAndPrint is broken on trunk - Key: MAPREDUCE-3199 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3199 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.0 Attachments: MAPREDUCE-3199-20111018.txt I bisected this down to MAPREDUCE-3003 changes. The parent project for client-core changed to hadoop-project which doesn't have the log4j configuration unlike the previous parent hadoop-mapreduce-client. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3200) Job got failed with FileNotFoundException during ResourceLocalization
Job got failed with FileNotFoundException during ResourceLocalization
-
Key: MAPREDUCE-3200
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3200
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Ramgopal N
The exception trace is as follows
{code:xml}
2011-10-18 15:37:11,315 WARN
org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=root
OPERATION=Application Finished - Failed TARGET=RMAppManager RESULT=FAILURE
DESCRIPTION=App failed with state: FAILED PERMISSIONS=Application
application_1318930754911_0002 failed 1 times due to AM Container for
appattempt_1318930754911_0002_01 exited with exitCode: -1000 due to:
java.io.FileNotFoundException:
file:/tmp/nm-local-dir/usercache/root/appcache/application_1318930754911_0003/container_1318930754911_0002_01_01.tokens
at
org.apache.hadoop.fs.RawLocalFileSystem.open(RawLocalFileSystem.java:189)
at
org.apache.hadoop.fs.DelegateToFileSystem.open(DelegateToFileSystem.java:150)
at
org.apache.hadoop.fs.AbstractFileSystem.open(AbstractFileSystem.java:595)
at org.apache.hadoop.fs.FilterFs.open(FilterFs.java:197)
at org.apache.hadoop.fs.FileContext$6.next(FileContext.java:759)
at org.apache.hadoop.fs.FileContext$6.next(FileContext.java:756)
at
org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2321)
at org.apache.hadoop.fs.FileContext.open(FileContext.java:756)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:133)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:86)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:855)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3199) TestJobMonitorAndPrint is broken on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-3199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129662#comment-13129662 ] Hadoop QA commented on MAPREDUCE-3199: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12499529/MAPREDUCE-3199-20111018.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1054//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1054//console This message is automatically generated. TestJobMonitorAndPrint is broken on trunk - Key: MAPREDUCE-3199 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3199 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.0 Attachments: MAPREDUCE-3199-20111018.txt I bisected this down to MAPREDUCE-3003 changes. The parent project for client-core changed to hadoop-project which doesn't have the log4j configuration unlike the previous parent hadoop-mapreduce-client. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3201) Even though jobs are getting failed on particular NM, it is not getting blacklisted
Even though jobs are getting failed on particular NM, it is not getting
blacklisted
---
Key: MAPREDUCE-3201
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3201
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Ramgopal N
Priority: Minor
{code:xml}
The yarnchild process on a particular NM are getting killed continuosly.
Still the NM is not getting blacklisted
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3118) Backport Gridmix and Rumen features from trunk to Hadoop 0.20 security branch
[ https://issues.apache.org/jira/browse/MAPREDUCE-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129672#comment-13129672 ] Amar Kamat commented on MAPREDUCE-3118: --- Folks, We ran the wordcount and Gridmix3 from Hadoop 0.20.205 on a Hadoop cluster running a MR-3118 patched version of 0.20.205. All the jobs ran fine. We didn't see any compatibility issues. I will commit this patch. Backport Gridmix and Rumen features from trunk to Hadoop 0.20 security branch - Key: MAPREDUCE-3118 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3118 Project: Hadoop Map/Reduce Issue Type: New Feature Components: contrib/gridmix, tools/rumen Affects Versions: 0.20.206.0 Reporter: Ravi Gummadi Assignee: Ravi Gummadi Fix For: 0.20.206.0 Attachments: gridmix_rumen_backports.v2.4.patch, gridmix_rumen_backports.v2.5.patch, gridmix_rumen_backports.v2.6.patch Backporting all the features and bugfixes that went into gridmix and rumen of trunk to hadoop 0.20 security branch. This will enable using all these gridmix features and run gridmix/rumen on the history logs of 0.20 security branch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3201) Even though jobs are getting failed on particular NM, it is not getting blacklisted
[
https://issues.apache.org/jira/browse/MAPREDUCE-3201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129673#comment-13129673
]
Vinod Kumar Vavilapalli commented on MAPREDUCE-3201:
Ramgopal, we do have job-level blacklist today. Can you look at your AM logs
and grep for the following log messsages? Thanks.
{quote}
nodeBlacklistingEnabled:
maxTaskFailuresPerNode is
failures on node
Blacklisted host
{quote}
There is a known bug related to NM blacklisting by MR jobs - MAPREDUCE-2693,
but doesn't look like you are running into that.
Even though jobs are getting failed on particular NM, it is not getting
blacklisted
---
Key: MAPREDUCE-3201
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3201
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Ramgopal N
Priority: Minor
{code:xml}
The yarnchild process on a particular NM are getting killed continuosly.
Still the NM is not getting blacklisted
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3176) ant mapreduce tests are timing out
[ https://issues.apache.org/jira/browse/MAPREDUCE-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129684#comment-13129684 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-3176: *sigh* My local ant test run finished. Some more tests that are timing out: - org.apache.hadoop.examples.TestWordStats - org.apache.hadoop.mapred.lib.aggregate.TestAggregates - org.apache.hadoop.mapreduce.lib.aggregate.TestMapReduceAggregates - org.apache.hadoop.mapreduce.lib.db.TestDataDrivenDBInputFormat - org.apache.hadoop.mapreduce.lib.db.TestDBJob - org.apache.hadoop.mapreduce.lib.fieldsel.TestMRFieldSelection - org.apache.hadoop.mapreduce.lib.join.TestJoinDatamerge - org.apache.hadoop.mapreduce.TestLocalRunner - org.apache.hadoop.mapreduce.TestMapReduce - org.apache.hadoop.mapreduce.TestValueIterReset - org.apache.hadoop.record.TestRecordMR Test failing because of the need for the framework name: - org.apache.hadoop.mapreduce.TestNoJobSetupCleanup Tests which are failing for unrelated issues but need fixing: - org.apache.hadoop.cli.TestMRCLI - org.apache.hadoop.conf.TestNoDefaultsJobConf - org.apache.hadoop.examples.terasort.TestTeraSort - org.apache.hadoop.ipc.TestSocketFactory - org.apache.hadoop.mapred.TestAuditLogger - org.apache.hadoop.mapreduce.security.TestTokenCache - org.apache.hadoop.mapreduce.security.token.delegation.TestDelegationToken - org.apache.hadoop.mapred.TestCombineOutputCollector - org.apache.hadoop.mapred.TestSubmitJob - org.apache.hadoop.mapreduce.TestTaskContext ant mapreduce tests are timing out -- Key: MAPREDUCE-3176 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3176 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.0 Reporter: Ravi Prakash Assignee: Hitesh Shah Priority: Blocker Labels: test Fix For: 0.23.0 Attachments: MR-3176.1.patch, MR-3176.wip.patch Secondary YARN builds started taking inordinately long and lots of tests started failing. Usually the secondary build would take ~ 2 hours. But recently even after 7 hours it wasn't done. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3200) Job got failed with FileNotFoundException during ResourceLocalization
[
https://issues.apache.org/jira/browse/MAPREDUCE-3200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129687#comment-13129687
]
Vinod Kumar Vavilapalli commented on MAPREDUCE-3200:
This may be same as MAPREDUCE-3159. How often are you hitting this?
Job got failed with FileNotFoundException during ResourceLocalization
-
Key: MAPREDUCE-3200
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3200
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Ramgopal N
The exception trace is as follows
{code:xml}
2011-10-18 15:37:11,315 WARN
org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=root
OPERATION=Application Finished - Failed TARGET=RMAppManager
RESULT=FAILURE DESCRIPTION=App failed with state: FAILED
PERMISSIONS=Application application_1318930754911_0002 failed 1 times due to
AM Container for appattempt_1318930754911_0002_01 exited with exitCode:
-1000 due to: java.io.FileNotFoundException:
file:/tmp/nm-local-dir/usercache/root/appcache/application_1318930754911_0003/container_1318930754911_0002_01_01.tokens
at
org.apache.hadoop.fs.RawLocalFileSystem.open(RawLocalFileSystem.java:189)
at
org.apache.hadoop.fs.DelegateToFileSystem.open(DelegateToFileSystem.java:150)
at
org.apache.hadoop.fs.AbstractFileSystem.open(AbstractFileSystem.java:595)
at org.apache.hadoop.fs.FilterFs.open(FilterFs.java:197)
at org.apache.hadoop.fs.FileContext$6.next(FileContext.java:759)
at org.apache.hadoop.fs.FileContext$6.next(FileContext.java:756)
at
org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2321)
at org.apache.hadoop.fs.FileContext.open(FileContext.java:756)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:133)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:86)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:855)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3196) TestLinuxContainerExecutorWithMocks fails on Mac OSX
[ https://issues.apache.org/jira/browse/MAPREDUCE-3196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129697#comment-13129697 ] Hudson commented on MAPREDUCE-3196: --- Integrated in Hadoop-Hdfs-0.23-Build #43 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/43/]) MAPREDUCE-3196. TestLinuxContainerExecutorWithMocks fails on Mac OSX. (Arun Murthy via mahadev) - Merging r1185417 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185419 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java TestLinuxContainerExecutorWithMocks fails on Mac OSX Key: MAPREDUCE-3196 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3196 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.23.0 Attachments: MAPREDUCE-3196.patch TestLinuxContainerExecutorWithMocks uses /bin/true which isn't present. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3197) TestMRClientService failing on building clean checkout of branch 0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129696#comment-13129696 ] Hudson commented on MAPREDUCE-3197: --- Integrated in Hadoop-Hdfs-0.23-Build #43 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/43/]) MAPREDUCE-3197. TestMRClientService failing on building clean checkout of branch 0.23 (mahadev) - Merging r1185486 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185487 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRClientService.java TestMRClientService failing on building clean checkout of branch 0.23 - Key: MAPREDUCE-3197 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3197 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Anupam Seth Assignee: Mahadev konar Fix For: 0.23.0 Attachments: MAPREDUCE-3197.patch A clean checkout of the branch 0.23 source tree does not pass TestMRClientService#test(), which fails with the error message Num diagnostics is not correct expected 2 but was:1 upon running mvn clean install assembly:assembly inside MR directory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-279) Map-Reduce 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129693#comment-13129693 ] Hudson commented on MAPREDUCE-279: -- Integrated in Hadoop-Hdfs-0.23-Build #43 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/43/]) MAPREDUCE-279. Adding a changelog to branch-0.23. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185489 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt Map-Reduce 2.0 -- Key: MAPREDUCE-279 URL: https://issues.apache.org/jira/browse/MAPREDUCE-279 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: Arun C Murthy Fix For: 0.23.0 Attachments: MR-279-script-20110817.sh, MR-279-script-final.sh, MR-279-script.sh, MR-279-script.sh, MR-279.patch, MR-279.patch, MR-279.sh, MR-279_MR_files_to_move-20110817.txt, MR-279_MR_files_to_move.txt, MR-279_MR_files_to_move.txt, MapReduce_NextGen_Architecture.pdf, NodeManager.gv, NodeManager.png, ResourceManager.gv, ResourceManager.png, capacity-scheduler-dark-theme.png, hadoop_contributors_meet_07_01_2011.pdf, multi-column-stable-sort-default-theme.png, post-move-patch-20110817.2.txt, post-move-patch-final.txt, post-move.patch, post-move.patch, post-move.patch, yarn-state-machine.job.dot, yarn-state-machine.job.png, yarn-state-machine.task-attempt.dot, yarn-state-machine.task-attempt.png, yarn-state-machine.task.dot, yarn-state-machine.task.png Re-factor MapReduce into a generic resource scheduler and a per-job, user-defined component that manages the application execution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129695#comment-13129695 ] Hudson commented on MAPREDUCE-3068: --- Integrated in Hadoop-Hdfs-0.23-Build #43 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/43/]) Merge -c 1185447 from trunk to branch-0.23 to complete fix for MAPREDUCE-3068. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185448 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-config.sh * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Apps.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.blank.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3200) Job got failed with FileNotFoundException during ResourceLocalization
[
https://issues.apache.org/jira/browse/MAPREDUCE-3200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129702#comment-13129702
]
Ramgopal N commented on MAPREDUCE-3200:
---
I am using Fairscheduler and submitted 3 jobs in sequence.The second job got
failed with this exception.The other 2 jobs passed.
Job got failed with FileNotFoundException during ResourceLocalization
-
Key: MAPREDUCE-3200
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3200
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Ramgopal N
The exception trace is as follows
{code:xml}
2011-10-18 15:37:11,315 WARN
org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=root
OPERATION=Application Finished - Failed TARGET=RMAppManager
RESULT=FAILURE DESCRIPTION=App failed with state: FAILED
PERMISSIONS=Application application_1318930754911_0002 failed 1 times due to
AM Container for appattempt_1318930754911_0002_01 exited with exitCode:
-1000 due to: java.io.FileNotFoundException:
file:/tmp/nm-local-dir/usercache/root/appcache/application_1318930754911_0003/container_1318930754911_0002_01_01.tokens
at
org.apache.hadoop.fs.RawLocalFileSystem.open(RawLocalFileSystem.java:189)
at
org.apache.hadoop.fs.DelegateToFileSystem.open(DelegateToFileSystem.java:150)
at
org.apache.hadoop.fs.AbstractFileSystem.open(AbstractFileSystem.java:595)
at org.apache.hadoop.fs.FilterFs.open(FilterFs.java:197)
at org.apache.hadoop.fs.FileContext$6.next(FileContext.java:759)
at org.apache.hadoop.fs.FileContext$6.next(FileContext.java:756)
at
org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2321)
at org.apache.hadoop.fs.FileContext.open(FileContext.java:756)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:133)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:86)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:855)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-279) Map-Reduce 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129710#comment-13129710 ] Hudson commented on MAPREDUCE-279: -- Integrated in Hadoop-Mapreduce-trunk #864 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/864/]) MAPREDUCE-279. Adding a changelog to branch-0.23. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185488 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt Map-Reduce 2.0 -- Key: MAPREDUCE-279 URL: https://issues.apache.org/jira/browse/MAPREDUCE-279 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: Arun C Murthy Fix For: 0.23.0 Attachments: MR-279-script-20110817.sh, MR-279-script-final.sh, MR-279-script.sh, MR-279-script.sh, MR-279.patch, MR-279.patch, MR-279.sh, MR-279_MR_files_to_move-20110817.txt, MR-279_MR_files_to_move.txt, MR-279_MR_files_to_move.txt, MapReduce_NextGen_Architecture.pdf, NodeManager.gv, NodeManager.png, ResourceManager.gv, ResourceManager.png, capacity-scheduler-dark-theme.png, hadoop_contributors_meet_07_01_2011.pdf, multi-column-stable-sort-default-theme.png, post-move-patch-20110817.2.txt, post-move-patch-final.txt, post-move.patch, post-move.patch, post-move.patch, yarn-state-machine.job.dot, yarn-state-machine.job.png, yarn-state-machine.task-attempt.dot, yarn-state-machine.task-attempt.png, yarn-state-machine.task.dot, yarn-state-machine.task.png Re-factor MapReduce into a generic resource scheduler and a per-job, user-defined component that manages the application execution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129713#comment-13129713 ] Hudson commented on MAPREDUCE-3068: --- Integrated in Hadoop-Mapreduce-trunk #864 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/864/]) MAPREDUCE-3068. Added a whitelist of environment variables for containers from the NodeManager and set MALLOC_ARENA_MAX for all daemons and containers. Contributed by Chris Riccomini. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185447 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-config.sh * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Apps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.blank.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3032) JobHistory doesn't have error information from failed tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129712#comment-13129712 ] Hudson commented on MAPREDUCE-3032: --- Integrated in Hadoop-Mapreduce-trunk #864 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/864/]) MAPREDUCE-3032. Fixed TaskAttemptImpl so that JobHistory can have error information about failed tasks. Contributed by Devaraj K. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185247 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/TaskAttemptListenerImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/event/TaskAttemptStatusUpdateEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/recover/RecoveryService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRClientService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java JobHistory doesn't have error information from failed tasks --- Key: MAPREDUCE-3032 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3032 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Devaraj K Priority: Blocker Fix For: 0.23.0 Attachments: Failed map attempts.jpg, JobFail.PNG, MAPREDUCE-3032-1.patch, MAPREDUCE-3032.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3003) Publish MR JARs to Maven snapshot repository
[ https://issues.apache.org/jira/browse/MAPREDUCE-3003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129711#comment-13129711 ] Hudson commented on MAPREDUCE-3003: --- Integrated in Hadoop-Mapreduce-trunk #864 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/864/]) MAPREDUCE-3003. Publish MR JARs to Maven snapshot repository. (tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185286 Files : * /hadoop/common/trunk/BUILDING.txt * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/ivy.xml * /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml * /hadoop/common/trunk/hadoop-project-dist/pom.xml * /hadoop/common/trunk/hadoop-project/pom.xml * /hadoop/common/trunk/pom.xml Publish MR JARs to Maven snapshot repository Key: MAPREDUCE-3003 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3003 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.23.0, 0.24.0 Reporter: Tom White Assignee: Alejandro Abdelnur Fix For: 0.23.0, 0.24.0 Attachments: MAPREDUCE-3003-0.23.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch Currently this is failing since no distribution management section is defined in the POM. https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Common-trunk-Commit/883/consoleFull -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3197) TestMRClientService failing on building clean checkout of branch 0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129714#comment-13129714 ] Hudson commented on MAPREDUCE-3197: --- Integrated in Hadoop-Mapreduce-trunk #864 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/864/]) MAPREDUCE-3197. TestMRClientService failing on building clean checkout of branch 0.23 (mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185486 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRClientService.java TestMRClientService failing on building clean checkout of branch 0.23 - Key: MAPREDUCE-3197 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3197 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Anupam Seth Assignee: Mahadev konar Fix For: 0.23.0 Attachments: MAPREDUCE-3197.patch A clean checkout of the branch 0.23 source tree does not pass TestMRClientService#test(), which fails with the error message Num diagnostics is not correct expected 2 but was:1 upon running mvn clean install assembly:assembly inside MR directory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3196) TestLinuxContainerExecutorWithMocks fails on Mac OSX
[ https://issues.apache.org/jira/browse/MAPREDUCE-3196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129716#comment-13129716 ] Hudson commented on MAPREDUCE-3196: --- Integrated in Hadoop-Mapreduce-trunk #864 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/864/]) MAPREDUCE-3196. TestLinuxContainerExecutorWithMocks fails on Mac OSX. (Arun Murthy via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185417 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java TestLinuxContainerExecutorWithMocks fails on Mac OSX Key: MAPREDUCE-3196 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3196 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.23.0 Attachments: MAPREDUCE-3196.patch TestLinuxContainerExecutorWithMocks uses /bin/true which isn't present. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3032) JobHistory doesn't have error information from failed tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129721#comment-13129721 ] Hudson commented on MAPREDUCE-3032: --- Integrated in Hadoop-Mapreduce-0.23-Build #55 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/55/]) MAPREDUCE-3032. Fixed TaskAttemptImpl so that JobHistory can have error information about failed tasks. Contributed by Devaraj K. svn merge -c r1185247 --ignore-ancestry ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185250 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/TaskAttemptListenerImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/event/TaskAttemptStatusUpdateEvent.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/recover/RecoveryService.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRClientService.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java JobHistory doesn't have error information from failed tasks --- Key: MAPREDUCE-3032 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3032 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Devaraj K Priority: Blocker Fix For: 0.23.0 Attachments: Failed map attempts.jpg, JobFail.PNG, MAPREDUCE-3032-1.patch, MAPREDUCE-3032.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3197) TestMRClientService failing on building clean checkout of branch 0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129723#comment-13129723 ] Hudson commented on MAPREDUCE-3197: --- Integrated in Hadoop-Mapreduce-0.23-Build #55 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/55/]) MAPREDUCE-3197. TestMRClientService failing on building clean checkout of branch 0.23 (mahadev) - Merging r1185486 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185487 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRClientService.java TestMRClientService failing on building clean checkout of branch 0.23 - Key: MAPREDUCE-3197 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3197 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Anupam Seth Assignee: Mahadev konar Fix For: 0.23.0 Attachments: MAPREDUCE-3197.patch A clean checkout of the branch 0.23 source tree does not pass TestMRClientService#test(), which fails with the error message Num diagnostics is not correct expected 2 but was:1 upon running mvn clean install assembly:assembly inside MR directory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129722#comment-13129722 ] Hudson commented on MAPREDUCE-3068: --- Integrated in Hadoop-Mapreduce-0.23-Build #55 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/55/]) Merge -c 1185447 from trunk to branch-0.23 to complete fix for MAPREDUCE-3068. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185448 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-config.sh * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Apps.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.blank.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3196) TestLinuxContainerExecutorWithMocks fails on Mac OSX
[ https://issues.apache.org/jira/browse/MAPREDUCE-3196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129724#comment-13129724 ] Hudson commented on MAPREDUCE-3196: --- Integrated in Hadoop-Mapreduce-0.23-Build #55 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/55/]) MAPREDUCE-3196. TestLinuxContainerExecutorWithMocks fails on Mac OSX. (Arun Murthy via mahadev) - Merging r1185417 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185419 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java TestLinuxContainerExecutorWithMocks fails on Mac OSX Key: MAPREDUCE-3196 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3196 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.23.0 Attachments: MAPREDUCE-3196.patch TestLinuxContainerExecutorWithMocks uses /bin/true which isn't present. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3193) NextGen Mapreduce framework is not able to read the job input recursively.Input is read only for one folder level deep
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K reassigned MAPREDUCE-3193: Assignee: Devaraj K NextGen Mapreduce framework is not able to read the job input recursively.Input is read only for one folder level deep -- Key: MAPREDUCE-3193 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramgopal N Assignee: Devaraj K java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed. Example:Input file is /r1/r2/input.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3003) Publish MR JARs to Maven snapshot repository
[ https://issues.apache.org/jira/browse/MAPREDUCE-3003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129729#comment-13129729 ] Hudson commented on MAPREDUCE-3003: --- Integrated in Hadoop-Hdfs-trunk #834 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/834/]) MAPREDUCE-3003. Publish MR JARs to Maven snapshot repository. (tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185286 Files : * /hadoop/common/trunk/BUILDING.txt * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/ivy.xml * /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml * /hadoop/common/trunk/hadoop-project-dist/pom.xml * /hadoop/common/trunk/hadoop-project/pom.xml * /hadoop/common/trunk/pom.xml Publish MR JARs to Maven snapshot repository Key: MAPREDUCE-3003 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3003 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.23.0, 0.24.0 Reporter: Tom White Assignee: Alejandro Abdelnur Fix For: 0.23.0, 0.24.0 Attachments: MAPREDUCE-3003-0.23.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch Currently this is failing since no distribution management section is defined in the POM. https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Common-trunk-Commit/883/consoleFull -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3197) TestMRClientService failing on building clean checkout of branch 0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129732#comment-13129732 ] Hudson commented on MAPREDUCE-3197: --- Integrated in Hadoop-Hdfs-trunk #834 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/834/]) MAPREDUCE-3197. TestMRClientService failing on building clean checkout of branch 0.23 (mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185486 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRClientService.java TestMRClientService failing on building clean checkout of branch 0.23 - Key: MAPREDUCE-3197 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3197 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Anupam Seth Assignee: Mahadev konar Fix For: 0.23.0 Attachments: MAPREDUCE-3197.patch A clean checkout of the branch 0.23 source tree does not pass TestMRClientService#test(), which fails with the error message Num diagnostics is not correct expected 2 but was:1 upon running mvn clean install assembly:assembly inside MR directory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-279) Map-Reduce 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129728#comment-13129728 ] Hudson commented on MAPREDUCE-279: -- Integrated in Hadoop-Hdfs-trunk #834 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/834/]) MAPREDUCE-279. Adding a changelog to branch-0.23. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185488 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt Map-Reduce 2.0 -- Key: MAPREDUCE-279 URL: https://issues.apache.org/jira/browse/MAPREDUCE-279 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: Arun C Murthy Fix For: 0.23.0 Attachments: MR-279-script-20110817.sh, MR-279-script-final.sh, MR-279-script.sh, MR-279-script.sh, MR-279.patch, MR-279.patch, MR-279.sh, MR-279_MR_files_to_move-20110817.txt, MR-279_MR_files_to_move.txt, MR-279_MR_files_to_move.txt, MapReduce_NextGen_Architecture.pdf, NodeManager.gv, NodeManager.png, ResourceManager.gv, ResourceManager.png, capacity-scheduler-dark-theme.png, hadoop_contributors_meet_07_01_2011.pdf, multi-column-stable-sort-default-theme.png, post-move-patch-20110817.2.txt, post-move-patch-final.txt, post-move.patch, post-move.patch, post-move.patch, yarn-state-machine.job.dot, yarn-state-machine.job.png, yarn-state-machine.task-attempt.dot, yarn-state-machine.task-attempt.png, yarn-state-machine.task.dot, yarn-state-machine.task.png Re-factor MapReduce into a generic resource scheduler and a per-job, user-defined component that manages the application execution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3032) JobHistory doesn't have error information from failed tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129730#comment-13129730 ] Hudson commented on MAPREDUCE-3032: --- Integrated in Hadoop-Hdfs-trunk #834 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/834/]) MAPREDUCE-3032. Fixed TaskAttemptImpl so that JobHistory can have error information about failed tasks. Contributed by Devaraj K. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185247 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/TaskAttemptListenerImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/event/TaskAttemptStatusUpdateEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/recover/RecoveryService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRClientService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java JobHistory doesn't have error information from failed tasks --- Key: MAPREDUCE-3032 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3032 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Devaraj K Priority: Blocker Fix For: 0.23.0 Attachments: Failed map attempts.jpg, JobFail.PNG, MAPREDUCE-3032-1.patch, MAPREDUCE-3032.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129731#comment-13129731 ] Hudson commented on MAPREDUCE-3068: --- Integrated in Hadoop-Hdfs-trunk #834 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/834/]) MAPREDUCE-3068. Added a whitelist of environment variables for containers from the NodeManager and set MALLOC_ARENA_MAX for all daemons and containers. Contributed by Chris Riccomini. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185447 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-config.sh * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Apps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Chris Riccomini Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3068.blank.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, MAPREDUCE-3068.diff This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3196) TestLinuxContainerExecutorWithMocks fails on Mac OSX
[ https://issues.apache.org/jira/browse/MAPREDUCE-3196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129734#comment-13129734 ] Hudson commented on MAPREDUCE-3196: --- Integrated in Hadoop-Hdfs-trunk #834 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/834/]) MAPREDUCE-3196. TestLinuxContainerExecutorWithMocks fails on Mac OSX. (Arun Murthy via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185417 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java TestLinuxContainerExecutorWithMocks fails on Mac OSX Key: MAPREDUCE-3196 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3196 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.23.0 Attachments: MAPREDUCE-3196.patch TestLinuxContainerExecutorWithMocks uses /bin/true which isn't present. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3181) Terasort fails with Kerberos exception on secure cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-3181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129741#comment-13129741 ] Thomas Graves commented on MAPREDUCE-3181: -- I haven't looked in detail at this but taking a quick look the bug doesn't show up without kerberos enabled because it doesn't ever try to load the yarn-site.xml without security on. Looking at the stack trace: at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:83) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:205) The obtainTokensForNamenodes eventually calls Master.getMasterPrincipal(conf) but only if security is enabled. There is if isSecurityEnabled in obtainTokensForNamenodes. getMasterPrinicipal then expects yarn conf to be loaded because if the framework is yarn it does: masterAddress = conf.get(YarnConfiguration.RM_ADDRESS, YarnConfiguration.DEFAULT_RM_ADDRESS); Its not all jobs that fail. teragen, wordcount, etc. work fine but they don't take the same path. Terasort fails with Kerberos exception on secure cluster Key: MAPREDUCE-3181 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3181 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Anupam Seth Assignee: Anupam Seth Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3181-branch0_23.patch, MAPREDUCE-3181-branch0_23.patch We are seeing the following Kerberos exception upon trying to run terasort on secure single and multi-node clusters using the latest build from branch 0.23. java.io.IOException: Can't get JobTracker Kerberos principal for use as renewer at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:106) at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:90) at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:83) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:205) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:269) at org.apache.hadoop.examples.terasort.TeraInputFormat.getSplits(TeraInputFormat.java:318) at org.apache.hadoop.examples.terasort.TeraInputFormat.writePartitionFile(TeraInputFormat.java:169) at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:306) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:325) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:68) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:189) Adding debug output shows that the job configuration is not loading up yarn-site.xml causing the above failure to happen. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3181) Terasort fails with Kerberos exception on secure cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-3181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129745#comment-13129745 ] Anupam Seth commented on MAPREDUCE-3181: Thanks Tom. I hope your explanation helps clear up Aaron's and Arun's questions. As Tom pointed out, this problem is seem only on Terasort in a secure cluster because of the conditional to not even try this if security is disabled and because this gets hit before the ResourceManagerDelegate is ever called or created (causing yarn-site.xml to be 'normally' loaded). This does not happen with Teragen, for example, because TeraSort attempts to write the input splits to HDFS before running the actual MR. Regarding the ant tests asked for by Mahadev, I see numerous failures due to timeout issues, most likely an artifact of MAPREDUCE-3176, as my changes are very simple. Also, I have patch ready with an added unit test to ensure this does not regress - am running tests on it, will put it up shortly. Terasort fails with Kerberos exception on secure cluster Key: MAPREDUCE-3181 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3181 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Anupam Seth Assignee: Anupam Seth Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3181-branch0_23.patch, MAPREDUCE-3181-branch0_23.patch We are seeing the following Kerberos exception upon trying to run terasort on secure single and multi-node clusters using the latest build from branch 0.23. java.io.IOException: Can't get JobTracker Kerberos principal for use as renewer at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:106) at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:90) at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:83) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:205) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:269) at org.apache.hadoop.examples.terasort.TeraInputFormat.getSplits(TeraInputFormat.java:318) at org.apache.hadoop.examples.terasort.TeraInputFormat.writePartitionFile(TeraInputFormat.java:169) at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:306) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:325) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:68) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:189) Adding debug output shows that the job configuration is not loading up yarn-site.xml causing the above failure to happen. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3176) ant mapreduce tests are timing out
[ https://issues.apache.org/jira/browse/MAPREDUCE-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129761#comment-13129761 ] Ravi Prakash commented on MAPREDUCE-3176: - The secondary build passed yesterday but it took 5 hours. These were the tests which timed out. [junit] Test org.apache.hadoop.cli.TestMRCLI FAILED (timeout) [junit] Test org.apache.hadoop.fs.TestDFSIO FAILED (timeout) [junit] Test org.apache.hadoop.mapred.lib.aggregate.TestAggregates FAILED (timeout) [junit] Test org.apache.hadoop.mapreduce.TestLocalRunner FAILED (timeout) [junit] Test org.apache.hadoop.mapreduce.TestMapReduce FAILED (timeout) [junit] Test org.apache.hadoop.mapreduce.TestValueIterReset FAILED (timeout) [junit] Test org.apache.hadoop.mapreduce.lib.aggregate.TestMapReduceAggregates FAILED (timeout) [junit] Test org.apache.hadoop.mapreduce.lib.db.TestDataDrivenDBInputFormat FAILED (timeout) [junit] Test org.apache.hadoop.mapreduce.lib.fieldsel.TestMRFieldSelection FAILED (timeout) [junit] Test org.apache.hadoop.mapreduce.lib.join.TestJoinDatamerge FAILED (timeout) [junit] Test org.apache.hadoop.record.TestRecordMR FAILED (timeout) They all fail with a large number of these lines above them bq. [junit] 2011-10-17 17:31:26,613 INFO [communication thread] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(489)) - reduce copy Another test timed out but didn't have this (reduce copy) line. [junit] Test org.apache.hadoop.mapreduce.lib.db.TestDBJob FAILED (timeout) @Vinod: TestWordStats did not run in my case. Also TestMRCLI timed out. ant mapreduce tests are timing out -- Key: MAPREDUCE-3176 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3176 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.0 Reporter: Ravi Prakash Assignee: Hitesh Shah Priority: Blocker Labels: test Fix For: 0.23.0 Attachments: MR-3176.1.patch, MR-3176.wip.patch Secondary YARN builds started taking inordinately long and lots of tests started failing. Usually the secondary build would take ~ 2 hours. But recently even after 7 hours it wasn't done. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3176) ant mapreduce tests are timing out
[ https://issues.apache.org/jira/browse/MAPREDUCE-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129766#comment-13129766 ] Ravi Prakash commented on MAPREDUCE-3176: - Oh! Vinod probably ran with MR-3176.1.patch , while I ran with MR-3176.wip.patch ant mapreduce tests are timing out -- Key: MAPREDUCE-3176 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3176 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.0 Reporter: Ravi Prakash Assignee: Hitesh Shah Priority: Blocker Labels: test Fix For: 0.23.0 Attachments: MR-3176.1.patch, MR-3176.wip.patch Secondary YARN builds started taking inordinately long and lots of tests started failing. Usually the secondary build would take ~ 2 hours. But recently even after 7 hours it wasn't done. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3028) Support job end notification in .next /0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129768#comment-13129768 ] Ravi Prakash commented on MAPREDUCE-3028: - Hi Arun! Yes I've been working on this. From your and Amol's discussions: Spec is as such (details to be borrowed from current JT). Please use same config variable names to avoid compatibility. If the name has JT (or jobtracker)then lets discuss. The spec is as such - AM should trigger a http call upon job completion - The http call should be fire and forget by default. - If user specifies number of tries then the following should be true 1. Allow hadoop config to be set to ignore this in case becomes a problem (too many retries) 2. Set max retries for all AM; i.e. users cannot specify a number above this. If Hadoop config does not specify it the value should be 1 (i.e fire and forget) 3. Lets say that the Hadoop config on max retries is set to 3. Then AM should try 3 times with timeout (again a Hadoop variable). If timeout is not specified, the value should be 30 seconds. The aim is to let AM get off quickly. Since unlike JT there is one AM per job there would not bottleneck and no hanging of queue. So 30 seconds is ok. Support job end notification in .next /0.23 --- Key: MAPREDUCE-3028 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3028 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Mohammad Kamrul Islam Assignee: Ravi Prakash Priority: Blocker Fix For: 0.23.0 Oozie primarily depends on the job end notification to determine when the job finishes. In the current version, job end notification is implemented in job tracker. Since job tracker will be removed in the upcoming hadoop release (.next), we wander where this support will move. I think this best effort notification could be implemented in the new Application Manager as one of the last step of job completion. Whatever implementation will it be, Oozie badly needs this feature to be continued in next releases as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3118) Backport Gridmix and Rumen features from trunk to Hadoop 0.20 security branch
[ https://issues.apache.org/jira/browse/MAPREDUCE-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat resolved MAPREDUCE-3118. --- Tags: gridmix rumen backport Resolution: Fixed Release Note: Backports latest features from trunk to 0.20.206 branch. Hadoop Flags: Reviewed I just committed this to the 0.20.206 branch. Great job Ravi! Thanks Vinay for helping with the testing. Backport Gridmix and Rumen features from trunk to Hadoop 0.20 security branch - Key: MAPREDUCE-3118 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3118 Project: Hadoop Map/Reduce Issue Type: New Feature Components: contrib/gridmix, tools/rumen Affects Versions: 0.20.206.0 Reporter: Ravi Gummadi Assignee: Ravi Gummadi Fix For: 0.20.206.0 Attachments: gridmix_rumen_backports.v2.4.patch, gridmix_rumen_backports.v2.5.patch, gridmix_rumen_backports.v2.6.patch Backporting all the features and bugfixes that went into gridmix and rumen of trunk to hadoop 0.20 security branch. This will enable using all these gridmix features and run gridmix/rumen on the history logs of 0.20 security branch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3028) Support job end notification in .next /0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129769#comment-13129769 ] Ravi Prakash commented on MAPREDUCE-3028: - I have been trying to see if I cannot include job-end notification as a part of YARN. So that the Node Manager, as part of maybe the AM Container's cleanup would issue the callback. Would you be opposed to that? Support job end notification in .next /0.23 --- Key: MAPREDUCE-3028 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3028 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Mohammad Kamrul Islam Assignee: Ravi Prakash Priority: Blocker Fix For: 0.23.0 Oozie primarily depends on the job end notification to determine when the job finishes. In the current version, job end notification is implemented in job tracker. Since job tracker will be removed in the upcoming hadoop release (.next), we wander where this support will move. I think this best effort notification could be implemented in the new Application Manager as one of the last step of job completion. Whatever implementation will it be, Oozie badly needs this feature to be continued in next releases as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3087) CLASSPATH not the same after MAPREDUCE-2880
[ https://issues.apache.org/jira/browse/MAPREDUCE-3087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated MAPREDUCE-3087: Priority: Critical (was: Major) CLASSPATH not the same after MAPREDUCE-2880 --- Key: MAPREDUCE-3087 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3087 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ravi Prakash Assignee: Ravi Prakash Priority: Critical Fix For: 0.23.0 Attachments: MAPREDUCE-3087.patch, MAPREDUCE-3087.patch, MAPREDUCE-3087.patch, MAPREDUCE-3087.patch, notworking.classpath.uniq, test-mr-app.jar, working.classpath.uniq After MAPREDUCE-2880, my classpath was missing key jar files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3199) TestJobMonitorAndPrint is broken on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-3199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3199: --- Status: Open (was: Patch Available) Turns out the mr.basedir and yarn.basedir are used for locating findBugs exclude file. Cancelling patch. TestJobMonitorAndPrint is broken on trunk - Key: MAPREDUCE-3199 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3199 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.0 Attachments: MAPREDUCE-3199-20111018.1.txt, MAPREDUCE-3199-20111018.txt I bisected this down to MAPREDUCE-3003 changes. The parent project for client-core changed to hadoop-project which doesn't have the log4j configuration unlike the previous parent hadoop-mapreduce-client. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3104) Implement Application ACLs, Queue ACLs and their interaction
[ https://issues.apache.org/jira/browse/MAPREDUCE-3104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3104: --- Status: Open (was: Patch Available) Cancelling patch for addressing the review comments. The default for Job-acls is ' ' (a space) so that all job information is as secure as possible when acls are enabled. When acls are disabled, the default won't matter anyways. This is the same behaviour from 0.20.* line of releases. Will work on all the other comments. Thanks for the review! Implement Application ACLs, Queue ACLs and their interaction Key: MAPREDUCE-3104 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3104 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, resourcemanager, security Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3104-20111017.3.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3176) ant mapreduce tests are timing out
[ https://issues.apache.org/jira/browse/MAPREDUCE-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129783#comment-13129783 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-3176: bq. Oh! Vinod probably ran with MR-3176.1.patch , while I ran with MR-3176.wip.patch Yes, I ran the tests with the patch. ant mapreduce tests are timing out -- Key: MAPREDUCE-3176 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3176 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.0 Reporter: Ravi Prakash Assignee: Hitesh Shah Priority: Blocker Labels: test Fix For: 0.23.0 Attachments: MR-3176.1.patch, MR-3176.wip.patch Secondary YARN builds started taking inordinately long and lots of tests started failing. Usually the secondary build would take ~ 2 hours. But recently even after 7 hours it wasn't done. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3176) ant mapreduce tests are timing out
[ https://issues.apache.org/jira/browse/MAPREDUCE-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3176: --- Status: Open (was: Patch Available) Hitesh, please let me know if you need me/Ravi help with a share of the remaining tests for fixing. Thanks. ant mapreduce tests are timing out -- Key: MAPREDUCE-3176 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3176 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.0 Reporter: Ravi Prakash Assignee: Hitesh Shah Priority: Blocker Labels: test Fix For: 0.23.0 Attachments: MR-3176.1.patch, MR-3176.wip.patch Secondary YARN builds started taking inordinately long and lots of tests started failing. Usually the secondary build would take ~ 2 hours. But recently even after 7 hours it wasn't done. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3028) Support job end notification in .next /0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129786#comment-13129786 ] Mahadev konar commented on MAPREDUCE-3028: -- Ravi, Why do you want to include the job end notification as part of container cleanup? Job end notification should probably be just a Map Reduce feature. As soon as the AM tell the RM that its done running the job (look at the RMCommunicator:unregister()), you should be able to make the http callback to say if the job is successful or not. Thoughts? Support job end notification in .next /0.23 --- Key: MAPREDUCE-3028 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3028 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Mohammad Kamrul Islam Assignee: Ravi Prakash Priority: Blocker Fix For: 0.23.0 Oozie primarily depends on the job end notification to determine when the job finishes. In the current version, job end notification is implemented in job tracker. Since job tracker will be removed in the upcoming hadoop release (.next), we wander where this support will move. I think this best effort notification could be implemented in the new Application Manager as one of the last step of job completion. Whatever implementation will it be, Oozie badly needs this feature to be continued in next releases as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3028) Support job end notification in .next /0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129789#comment-13129789 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-3028: No, I'd not put this in YARN (i.e Nodemanager). The notification is currently a MapReduce only feature (atleast it seems to be for now). I can imagine a generic application finish notification from YARN itself (RM or NM), but I'd postpone any discussion about that if we can pull it off in MapReduce AM for now. Also 30secs seems way too much, even if it for that one job. Make it something in the order of 5s of seconds? Support job end notification in .next /0.23 --- Key: MAPREDUCE-3028 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3028 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Mohammad Kamrul Islam Assignee: Ravi Prakash Priority: Blocker Fix For: 0.23.0 Oozie primarily depends on the job end notification to determine when the job finishes. In the current version, job end notification is implemented in job tracker. Since job tracker will be removed in the upcoming hadoop release (.next), we wander where this support will move. I think this best effort notification could be implemented in the new Application Manager as one of the last step of job completion. Whatever implementation will it be, Oozie badly needs this feature to be continued in next releases as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3199) TestJobMonitorAndPrint is broken on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-3199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129797#comment-13129797 ] Hadoop QA commented on MAPREDUCE-3199: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12499546/MAPREDUCE-3199-20111018.1.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 161 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1055//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1055//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1055//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1055//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1055//console This message is automatically generated. TestJobMonitorAndPrint is broken on trunk - Key: MAPREDUCE-3199 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3199 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.0 Attachments: MAPREDUCE-3199-20111018.1.txt, MAPREDUCE-3199-20111018.txt I bisected this down to MAPREDUCE-3003 changes. The parent project for client-core changed to hadoop-project which doesn't have the log4j configuration unlike the previous parent hadoop-mapreduce-client. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3144) Augment JobHistory to include information needed for serving aggregated logs.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3144: --- Status: Open (was: Patch Available) This looks almost there. One final look: - Can you modify the documentation to describe the new application envs? (./hadoop-yarn/hadoop-yarn-site/src/site/apt/WritingYarnApplications.apt.vm) - Can we somehow make the AMInfos logged in and retrieved from the JobHistory be ordered? So that we can know the original order. Granted the startTime should define the sort order, may be we should return the sorted list from the parser APIs? Augment JobHistory to include information needed for serving aggregated logs. - Key: MAPREDUCE-3144 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3144 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Siddharth Seth Priority: Critical Fix For: 0.23.0 Attachments: MR3144.patch, MR3144_v2.txt, MR3144_v3.txt, MR3144_v4.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3028) Support job end notification in .next /0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129816#comment-13129816 ] Alejandro Abdelnur commented on MAPREDUCE-3028: --- Wouldn't make more sense to have the jobhistory service to do the notification? Then regardless of the AM implementation it would be done. And, as I've suggested before I think we should change from push to pull (a la RSS feed). Support job end notification in .next /0.23 --- Key: MAPREDUCE-3028 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3028 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Mohammad Kamrul Islam Assignee: Ravi Prakash Priority: Blocker Fix For: 0.23.0 Oozie primarily depends on the job end notification to determine when the job finishes. In the current version, job end notification is implemented in job tracker. Since job tracker will be removed in the upcoming hadoop release (.next), we wander where this support will move. I think this best effort notification could be implemented in the new Application Manager as one of the last step of job completion. Whatever implementation will it be, Oozie badly needs this feature to be continued in next releases as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3087) CLASSPATH not the same after MAPREDUCE-2880
[ https://issues.apache.org/jira/browse/MAPREDUCE-3087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3087: --- Status: Open (was: Patch Available) CLASSPATH not the same after MAPREDUCE-2880 --- Key: MAPREDUCE-3087 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3087 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ravi Prakash Assignee: Ravi Prakash Priority: Critical Fix For: 0.23.0 Attachments: MAPREDUCE-3087.patch, MAPREDUCE-3087.patch, MAPREDUCE-3087.patch, MAPREDUCE-3087.patch, notworking.classpath.uniq, test-mr-app.jar, working.classpath.uniq After MAPREDUCE-2880, my classpath was missing key jar files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3195) MapReduce PreCommit Trunk build is silently failing when running tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129844#comment-13129844 ] Jonathan Eagles commented on MAPREDUCE-3195: Seems like the build has started working again. 216 tests run https://builds.apache.org/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/1042/testReport/ 326 tests run https://builds.apache.org/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/1054/testReport/ MapReduce PreCommit Trunk build is silently failing when running tests -- Key: MAPREDUCE-3195 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3195 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.23.0, 0.24.0 Reporter: Jonathan Eagles Priority: Blocker MapReduce PreCommit build is silently failing only running a very small portion of tests. The build then errors out, yet +1 it given to the patch. Last known Success build - 307 tests run and passed https://builds.apache.org/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/990/testReport/ First known Error build - 69 tests run and passed https://builds.apache.org/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/994/testReport/ Snippet from failed build log - Errors out and then +1 the patch https://builds.apache.org/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/994/console [INFO] [INFO] Reactor Summary: [INFO] [INFO] hadoop-yarn-api ... SUCCESS [19.512s] [INFO] hadoop-yarn-common FAILURE [13.835s] [INFO] hadoop-yarn-server-common . SKIPPED [INFO] hadoop-yarn-server-nodemanager SKIPPED [INFO] hadoop-yarn-server-resourcemanager SKIPPED [INFO] hadoop-yarn-server-tests .. SKIPPED [INFO] hadoop-yarn-server SKIPPED [INFO] hadoop-yarn-applications-distributedshell . SKIPPED [INFO] hadoop-yarn-applications .. SKIPPED [INFO] hadoop-yarn-site .. SKIPPED [INFO] hadoop-yarn ... SKIPPED [INFO] hadoop-mapreduce-client-core .. SKIPPED [INFO] hadoop-mapreduce-client-common SKIPPED [INFO] hadoop-mapreduce-client-shuffle ... SKIPPED [INFO] hadoop-mapreduce-client-app ... SKIPPED [INFO] hadoop-mapreduce-client-hs SKIPPED [INFO] hadoop-mapreduce-client-jobclient . SKIPPED [INFO] hadoop-mapreduce-client ... SKIPPED [INFO] hadoop-mapreduce .. SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 33.784s [INFO] Finished at: Wed Oct 12 12:03:22 UTC 2011 [INFO] Final Memory: 40M/630M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:2.2-beta-5:single (tar) on project hadoop-yarn-common: Failed to create assembly: Error adding file 'org.apache.hadoop:hadoop-yarn-api:jar:0.24.0-SNAPSHOT' to archive: /home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/target/classes isn't a file. - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn goals -rf :hadoop-yarn-common == == Running contrib tests. == == /bin/kill -9 27543 kill: No such process NOP == == Checking the integrity of system test framework code. ==
[jira] [Commented] (MAPREDUCE-3195) MapReduce PreCommit Trunk build is silently failing when running tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129845#comment-13129845 ] Jonathan Eagles commented on MAPREDUCE-3195: I was never able to fully reproduce this error exactly as the reactor build still always had Auth problems. Not sure how to remove those from the build. [INFO] Apache Hadoop Main SUCCESS [1.028s] [INFO] Apache Hadoop Project POM . SUCCESS [0.784s] [INFO] Apache Hadoop Annotations . SUCCESS [3.069s] [INFO] Apache Hadoop Project Dist POM SUCCESS [0.666s] [INFO] Apache Hadoop Assemblies .. SUCCESS [0.286s] [INFO] Apache Hadoop Auth SUCCESS [7.141s] [INFO] Apache Hadoop Auth Examples ... FAILURE [1.472s] [INFO] Apache Hadoop Common .. SKIPPED [INFO] Apache Hadoop Common Project .. SKIPPED [INFO] Apache Hadoop HDFS SKIPPED [INFO] Apache Hadoop HDFS Project SKIPPED [INFO] hadoop-yarn ... SKIPPED [INFO] hadoop-yarn-api ... SKIPPED [INFO] hadoop-yarn-common SKIPPED [INFO] hadoop-yarn-server SKIPPED [INFO] hadoop-yarn-server-common . SKIPPED [INFO] hadoop-yarn-server-nodemanager SKIPPED [INFO] hadoop-yarn-server-resourcemanager SKIPPED [INFO] hadoop-yarn-server-tests .. SKIPPED [INFO] hadoop-yarn-applications .. SKIPPED [INFO] hadoop-yarn-applications-distributedshell . SKIPPED [INFO] hadoop-yarn-site .. SKIPPED [INFO] hadoop-mapreduce-client ... SKIPPED [INFO] hadoop-mapreduce-client-core .. SKIPPED [INFO] hadoop-mapreduce-client-common SKIPPED [INFO] hadoop-mapreduce-client-shuffle ... SKIPPED [INFO] hadoop-mapreduce-client-app ... SKIPPED [INFO] hadoop-mapreduce-client-hs SKIPPED [INFO] hadoop-mapreduce-client-jobclient . SKIPPED [INFO] hadoop-mapreduce .. SKIPPED [INFO] Apache Hadoop Tools ... SKIPPED [INFO] Apache Hadoop Distribution SKIPPED MapReduce PreCommit Trunk build is silently failing when running tests -- Key: MAPREDUCE-3195 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3195 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.23.0, 0.24.0 Reporter: Jonathan Eagles Priority: Blocker MapReduce PreCommit build is silently failing only running a very small portion of tests. The build then errors out, yet +1 it given to the patch. Last known Success build - 307 tests run and passed https://builds.apache.org/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/990/testReport/ First known Error build - 69 tests run and passed https://builds.apache.org/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/994/testReport/ Snippet from failed build log - Errors out and then +1 the patch https://builds.apache.org/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/994/console [INFO] [INFO] Reactor Summary: [INFO] [INFO] hadoop-yarn-api ... SUCCESS [19.512s] [INFO] hadoop-yarn-common FAILURE [13.835s] [INFO] hadoop-yarn-server-common . SKIPPED [INFO] hadoop-yarn-server-nodemanager SKIPPED [INFO] hadoop-yarn-server-resourcemanager SKIPPED [INFO] hadoop-yarn-server-tests .. SKIPPED [INFO] hadoop-yarn-server SKIPPED [INFO] hadoop-yarn-applications-distributedshell . SKIPPED [INFO] hadoop-yarn-applications .. SKIPPED [INFO] hadoop-yarn-site .. SKIPPED [INFO] hadoop-yarn ... SKIPPED [INFO] hadoop-mapreduce-client-core .. SKIPPED [INFO] hadoop-mapreduce-client-common SKIPPED [INFO] hadoop-mapreduce-client-shuffle ... SKIPPED [INFO] hadoop-mapreduce-client-app ... SKIPPED [INFO] hadoop-mapreduce-client-hs SKIPPED [INFO] hadoop-mapreduce-client-jobclient . SKIPPED
[jira] [Commented] (MAPREDUCE-3087) CLASSPATH not the same after MAPREDUCE-2880
[ https://issues.apache.org/jira/browse/MAPREDUCE-3087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129846#comment-13129846 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-3087: I've been conveniently ignoring this all this while. mr-generated-classpath needs to be automatically generated. And it(its content) is tied to the app module (hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml). We cannot check-in some version of it; if something changes in app module, this file(and/or the test-jar) needs to change and thus is unmaintainable. So we should only add the real-app jar as the dependency. We only need it to be present for jobclient module which uses the setFrameWorkClasspath() util, which already is there. So in sum, your patch without any pom changes should work for you, I think. Shouldn't it? CLASSPATH not the same after MAPREDUCE-2880 --- Key: MAPREDUCE-3087 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3087 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ravi Prakash Assignee: Ravi Prakash Priority: Critical Fix For: 0.23.0 Attachments: MAPREDUCE-3087.patch, MAPREDUCE-3087.patch, MAPREDUCE-3087.patch, MAPREDUCE-3087.patch, notworking.classpath.uniq, test-mr-app.jar, working.classpath.uniq After MAPREDUCE-2880, my classpath was missing key jar files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3186) User jobs are getting hanged if the Resource manager process goes down and comes up while job is getting executed.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne reassigned MAPREDUCE-3186: - Assignee: Eric Payne User jobs are getting hanged if the Resource manager process goes down and comes up while job is getting executed. -- Key: MAPREDUCE-3186 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3186 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Environment: linux Reporter: Ramgopal N Assignee: Eric Payne Labels: test If the resource manager is restarted while the job execution is in progress, the job is getting hanged. UI shows the job as running. In the RM log, it is throwing an error ERROR org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: AppAttemptId doesnt exist in cache appattempt_1318579738195_0004_01 In the console MRAppMaster and Runjar processes are not getting killed -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3028) Support job end notification in .next /0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129864#comment-13129864 ] Robert Joseph Evans commented on MAPREDUCE-3028: I don't like the job history server doing it because launching the job history server is some what optional (The default start scripts do not launch it currently). Also it will add in a new delay, and a lot more churn on the Job history server. Currently the JobHistory server polls HDFS for new jobs that have finished. If the job has finished it will move it to its own directory, and then cache some info about the app based only on the name of the jhist file. If we want this to do notification also then we need to either store the URL in the jhist file name, or force the server to read in the config file every time it pulls in a new Application. I agree that having applications poll for status with something like a very low latency RSS is the preferable solution and as such I would like to see the minimal done to support backwards compatibility, which would be to put the notification in the ApplicationMaster. But it is Ravi that is doing the fix and if he wants to do more then that and it is not something that is going to cause more problems then it fixes then I would say let him do it. What I mean by cause more problems then it fixes is that we have had lots of issues in production with having the notification be in a single thread blocking other notifications. What ever solution it is it cannot allow other notifications to become blocked or severely delayed by a single bad URL. (Or our case of a remote server crashing in the middle of the request and some how causing the socket be in a psudo state where the read blocks indefinitely and there is no one on the other side that is ever going to write) Support job end notification in .next /0.23 --- Key: MAPREDUCE-3028 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3028 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Mohammad Kamrul Islam Assignee: Ravi Prakash Priority: Blocker Fix For: 0.23.0 Oozie primarily depends on the job end notification to determine when the job finishes. In the current version, job end notification is implemented in job tracker. Since job tracker will be removed in the upcoming hadoop release (.next), we wander where this support will move. I think this best effort notification could be implemented in the new Application Manager as one of the last step of job completion. Whatever implementation will it be, Oozie badly needs this feature to be continued in next releases as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2905) CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated MAPREDUCE-2905: --- Target Version/s: 0.20.206.0 CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job) --- Key: MAPREDUCE-2905 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2905 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/fair-share Affects Versions: 0.20.2 Reporter: Jeff Bean Attachments: MR-2905.10-13-2011, MR-2905.patch, MR-2905.patch.2, mr-2905.txt, screenshot-1.jpg We encountered a situation where in the same cluster, large jobs benefit from mapred.fairscheduler.assignmultiple, but small jobs with small numbers of mappers do not: the mappers all clump to fully occupy just a few nodes, which causes those nodes to saturate and bottleneck. The desired behavior is to spread the job across more nodes so that a relatively small job doesn't saturate any node in the cluster. Testing has shown that setting mapred.fairscheduler.assignmultiple to false gives the desired behavior for small jobs, but is unnecessary for large jobs. However, since this is a cluster-wide setting, we can't properly tune. It'd be nice if jobs can set a param similar to mapred.fairscheduler.assignmultiple on submission to better control the task distribution of a particular job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3028) Support job end notification in .next /0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129881#comment-13129881 ] Mahadev konar commented on MAPREDUCE-3028: -- Agree with Robert here. Lets just do the notification in AM for now. We can look for other enhancements/generalization/improvements later. Support job end notification in .next /0.23 --- Key: MAPREDUCE-3028 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3028 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Mohammad Kamrul Islam Assignee: Ravi Prakash Priority: Blocker Fix For: 0.23.0 Oozie primarily depends on the job end notification to determine when the job finishes. In the current version, job end notification is implemented in job tracker. Since job tracker will be removed in the upcoming hadoop release (.next), we wander where this support will move. I think this best effort notification could be implemented in the new Application Manager as one of the last step of job completion. Whatever implementation will it be, Oozie badly needs this feature to be continued in next releases as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2905) CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins reassigned MAPREDUCE-2905: -- Assignee: Jeff Bean CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job) --- Key: MAPREDUCE-2905 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2905 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/fair-share Affects Versions: 0.20.2 Reporter: Jeff Bean Assignee: Jeff Bean Attachments: MR-2905.10-13-2011, MR-2905.patch, MR-2905.patch.2, mr-2905.txt, screenshot-1.jpg We encountered a situation where in the same cluster, large jobs benefit from mapred.fairscheduler.assignmultiple, but small jobs with small numbers of mappers do not: the mappers all clump to fully occupy just a few nodes, which causes those nodes to saturate and bottleneck. The desired behavior is to spread the job across more nodes so that a relatively small job doesn't saturate any node in the cluster. Testing has shown that setting mapred.fairscheduler.assignmultiple to false gives the desired behavior for small jobs, but is unnecessary for large jobs. However, since this is a cluster-wide setting, we can't properly tune. It'd be nice if jobs can set a param similar to mapred.fairscheduler.assignmultiple on submission to better control the task distribution of a particular job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3104) Implement Application ACLs, Queue ACLs and their interaction
[ https://issues.apache.org/jira/browse/MAPREDUCE-3104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129891#comment-13129891 ] Siddharth Seth commented on MAPREDUCE-3104: --- bq. MRJobConfig.JOB_ACL_* can have default values in MRJobConfig instead of specifying in YARNRunner I meant defining constants MRJobConfig.JOB_ACL_VIEW_JOB_DEFAULT and JOB_ACL_MODIFY_JOB_DEFAULT= , instead of specifying the default in YARNRunner. Implement Application ACLs, Queue ACLs and their interaction Key: MAPREDUCE-3104 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3104 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, resourcemanager, security Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3104-20111017.3.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2788) Normalize requests in FifoScheduler.allocate to prevent NPEs later
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129902#comment-13129902 ] Ahmed Radwan commented on MAPREDUCE-2788: - Will do, thanks Mahadev. Normalize requests in FifoScheduler.allocate to prevent NPEs later -- Key: MAPREDUCE-2788 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Ahmed Radwan Assignee: Ahmed Radwan Attachments: MAPREDUCE-2788.patch, MAPREDUCE-2788_rev2.patch, MAPREDUCE-2788_rev3.patch, MAPREDUCE-2788_rev4.patch The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3195) Detect MapReduce PreCommit Trunk builds silently failing when running test-patch.sh
[ https://issues.apache.org/jira/browse/MAPREDUCE-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3195: --- Target Version/s: 0.23.0, 0.24.0 (was: 0.24.0, 0.23.0) Summary: Detect MapReduce PreCommit Trunk builds silently failing when running test-patch.sh (was: MapReduce PreCommit Trunk build is silently failing when running tests) Detect MapReduce PreCommit Trunk builds silently failing when running test-patch.sh --- Key: MAPREDUCE-3195 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3195 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.23.0, 0.24.0 Reporter: Jonathan Eagles Priority: Blocker MapReduce PreCommit build is silently failing only running a very small portion of tests. The build then errors out, yet +1 it given to the patch. Last known Success build - 307 tests run and passed https://builds.apache.org/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/990/testReport/ First known Error build - 69 tests run and passed https://builds.apache.org/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/994/testReport/ Snippet from failed build log - Errors out and then +1 the patch https://builds.apache.org/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/994/console [INFO] [INFO] Reactor Summary: [INFO] [INFO] hadoop-yarn-api ... SUCCESS [19.512s] [INFO] hadoop-yarn-common FAILURE [13.835s] [INFO] hadoop-yarn-server-common . SKIPPED [INFO] hadoop-yarn-server-nodemanager SKIPPED [INFO] hadoop-yarn-server-resourcemanager SKIPPED [INFO] hadoop-yarn-server-tests .. SKIPPED [INFO] hadoop-yarn-server SKIPPED [INFO] hadoop-yarn-applications-distributedshell . SKIPPED [INFO] hadoop-yarn-applications .. SKIPPED [INFO] hadoop-yarn-site .. SKIPPED [INFO] hadoop-yarn ... SKIPPED [INFO] hadoop-mapreduce-client-core .. SKIPPED [INFO] hadoop-mapreduce-client-common SKIPPED [INFO] hadoop-mapreduce-client-shuffle ... SKIPPED [INFO] hadoop-mapreduce-client-app ... SKIPPED [INFO] hadoop-mapreduce-client-hs SKIPPED [INFO] hadoop-mapreduce-client-jobclient . SKIPPED [INFO] hadoop-mapreduce-client ... SKIPPED [INFO] hadoop-mapreduce .. SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 33.784s [INFO] Finished at: Wed Oct 12 12:03:22 UTC 2011 [INFO] Final Memory: 40M/630M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:2.2-beta-5:single (tar) on project hadoop-yarn-common: Failed to create assembly: Error adding file 'org.apache.hadoop:hadoop-yarn-api:jar:0.24.0-SNAPSHOT' to archive: /home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/target/classes isn't a file. - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn goals -rf :hadoop-yarn-common == == Running contrib tests. == == /bin/kill -9 27543 kill: No such process NOP == == Checking the integrity of system test framework code. ==
[jira] [Commented] (MAPREDUCE-3176) ant mapreduce tests are timing out
[ https://issues.apache.org/jira/browse/MAPREDUCE-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129906#comment-13129906 ] Hitesh Shah commented on MAPREDUCE-3176: @Vinod, @Ravi - thanks for running the tests and helping in verifying the patch. However, I believe the current attached patch is wrong and the actual fix needs to be applied in ReduceTask as to how it detects whether the job is local or not. Trivial change to the code to now do the following: if framework not set and master address is local, assume local runner else if framework set and framework == local, assume local runner else not a local runner Running the tests locally at the moment - will send out an update once it completes. ant mapreduce tests are timing out -- Key: MAPREDUCE-3176 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3176 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.0 Reporter: Ravi Prakash Assignee: Hitesh Shah Priority: Blocker Labels: test Fix For: 0.23.0 Attachments: MR-3176.1.patch, MR-3176.wip.patch Secondary YARN builds started taking inordinately long and lots of tests started failing. Usually the secondary build would take ~ 2 hours. But recently even after 7 hours it wasn't done. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3202) Integrating Hadoop Vaidya with JobHistory Server
Integrating Hadoop Vaidya with JobHistory Server Key: MAPREDUCE-3202 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3202 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobhistoryserver Affects Versions: 0.20.205.1 Reporter: vitthal (Suhas) Gogate At present jobdetailshistory page served by JobHistory Server provides elementary job analysis through link Analyze This job. Hadoop Vaidya provides a detailed analysis of the M/R job in terms of various execution inefficiencies and the associated remedies that user can easily understand and fix. Integrating Hadoop Vaidya with JobHistory server would really improve the usability of this tool and also benefit many naive users understanding various performance problems and/or best practices violations associated with their job. Integration would also aim at providing users a convenient interface where they can manage the existing rules as well as write their own new rules. During my tenure at Yahoo, Vaidya tool was successfully deployed in production analyzing tens of thousands of jobs every day with lot more useful rules than the sample ones present in the contrib project. Many of these rules are open sourced already (big thanks to Yahoo! MAPREDUCE-1530) but yet to integrate with the tool. I will add more design details for this feature in near future as work towards getting prototype running.. Any thoughts/comments are welcome. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3195) Detect MapReduce PreCommit Trunk builds silently failing when running test-patch.sh
[ https://issues.apache.org/jira/browse/MAPREDUCE-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129904#comment-13129904 ] Jonathan Eagles commented on MAPREDUCE-3195: Since the issue is fixed for now. Plan is to use this ticket to fix test-patch.sh to permanently prevent this type of error in the future. Detect MapReduce PreCommit Trunk builds silently failing when running test-patch.sh --- Key: MAPREDUCE-3195 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3195 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.23.0, 0.24.0 Reporter: Jonathan Eagles Priority: Blocker MapReduce PreCommit build is silently failing only running a very small portion of tests. The build then errors out, yet +1 it given to the patch. Last known Success build - 307 tests run and passed https://builds.apache.org/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/990/testReport/ First known Error build - 69 tests run and passed https://builds.apache.org/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/994/testReport/ Snippet from failed build log - Errors out and then +1 the patch https://builds.apache.org/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/994/console [INFO] [INFO] Reactor Summary: [INFO] [INFO] hadoop-yarn-api ... SUCCESS [19.512s] [INFO] hadoop-yarn-common FAILURE [13.835s] [INFO] hadoop-yarn-server-common . SKIPPED [INFO] hadoop-yarn-server-nodemanager SKIPPED [INFO] hadoop-yarn-server-resourcemanager SKIPPED [INFO] hadoop-yarn-server-tests .. SKIPPED [INFO] hadoop-yarn-server SKIPPED [INFO] hadoop-yarn-applications-distributedshell . SKIPPED [INFO] hadoop-yarn-applications .. SKIPPED [INFO] hadoop-yarn-site .. SKIPPED [INFO] hadoop-yarn ... SKIPPED [INFO] hadoop-mapreduce-client-core .. SKIPPED [INFO] hadoop-mapreduce-client-common SKIPPED [INFO] hadoop-mapreduce-client-shuffle ... SKIPPED [INFO] hadoop-mapreduce-client-app ... SKIPPED [INFO] hadoop-mapreduce-client-hs SKIPPED [INFO] hadoop-mapreduce-client-jobclient . SKIPPED [INFO] hadoop-mapreduce-client ... SKIPPED [INFO] hadoop-mapreduce .. SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 33.784s [INFO] Finished at: Wed Oct 12 12:03:22 UTC 2011 [INFO] Final Memory: 40M/630M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:2.2-beta-5:single (tar) on project hadoop-yarn-common: Failed to create assembly: Error adding file 'org.apache.hadoop:hadoop-yarn-api:jar:0.24.0-SNAPSHOT' to archive: /home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/target/classes isn't a file. - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn goals -rf :hadoop-yarn-common == == Running contrib tests. == == /bin/kill -9 27543 kill: No such process NOP == == Checking the integrity of system test framework code. == == /bin/kill -9 27548 kill:
[jira] [Updated] (MAPREDUCE-3144) Augment JobHistory to include information needed for serving aggregated logs.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3144: -- Attachment: MR3144_v5.txt Thanks for the review. Updated the documentation. bq. Can we somehow make the AMInfos logged in and retrieved from the JobHistory be ordered. They are already implicitly ordered (oldest to newest). After recovery, the older ones are logged first and the HistoryParser will see the older events before the newer ones. Or am I missing something ? Augment JobHistory to include information needed for serving aggregated logs. - Key: MAPREDUCE-3144 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3144 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Siddharth Seth Priority: Critical Fix For: 0.23.0 Attachments: MR3144.patch, MR3144_v2.txt, MR3144_v3.txt, MR3144_v4.txt, MR3144_v5.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3035) MR V2 jobhistory does not contain rack information
[ https://issues.apache.org/jira/browse/MAPREDUCE-3035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129987#comment-13129987 ] Arun C Murthy commented on MAPREDUCE-3035: -- Hi Chackaravarthy, any update on this? Thanks. MR V2 jobhistory does not contain rack information -- Key: MAPREDUCE-3035 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3035 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Karam Singh Assignee: chackaravarthy Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-3035.patch When topology.node.switch.mapping.impl is set to enable rack-locality resolution via the topology script, from the RM web-UI, we can see the rack information for each node. Running a job also reveals the information about rack-local map tasks launched at end of job completion on the client side. But the hostname field for attempts in the JobHistory does not contain this rack information. In case of hadoop-0.20 securiy or MRV1, hostname field of job history does contain rackid/hostname whereas in MRV2, hostname field only contains the hostIP. Thus this is a regression. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2821) [MR-279] Missing fields in job summary logs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129989#comment-13129989 ] Mahadev konar commented on MAPREDUCE-2821: -- Harsh, any update on this? [MR-279] Missing fields in job summary logs Key: MAPREDUCE-2821 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2821 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Assignee: Harsh J Priority: Blocker Fix For: 0.23.0 The following fields are missing in the job summary logs in mrv2: - numSlotsPerMap - numSlotsPerReduce - clusterCapacity (Earlier known as clusterMapCapacity and clusterReduceCapacity in 0.20.x) The first two fields are important to know if the job was a High RAM job or not and the last field is important to know the total available resource in the cluster during job execution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3028) Support job end notification in .next /0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3028: -- Component/s: mrv2 Support job end notification in .next /0.23 --- Key: MAPREDUCE-3028 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3028 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Mohammad Kamrul Islam Assignee: Ravi Prakash Priority: Blocker Fix For: 0.23.0 Oozie primarily depends on the job end notification to determine when the job finishes. In the current version, job end notification is implemented in job tracker. Since job tracker will be removed in the upcoming hadoop release (.next), we wander where this support will move. I think this best effort notification could be implemented in the new Application Manager as one of the last step of job completion. Whatever implementation will it be, Oozie badly needs this feature to be continued in next releases as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2696) Container logs aren't getting cleaned up
[ https://issues.apache.org/jira/browse/MAPREDUCE-2696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-2696: -- Assignee: (was: Siddharth Seth) Container logs aren't getting cleaned up Key: MAPREDUCE-2696 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2696 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Priority: Blocker Fix For: 0.23.0 Container logs aren't getting cleaned up causing NMs to fill up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2766) [MR-279] Set correct permissions for files in dist cache
[ https://issues.apache.org/jira/browse/MAPREDUCE-2766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-2766: -- Assignee: (was: Siddharth Seth) [MR-279] Set correct permissions for files in dist cache Key: MAPREDUCE-2766 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2766 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Priority: Blocker Fix For: 0.23.0 Currently, the files in both public and private dist cache are having 777 permission. Also, the group ownership of files on private cache have to be set to $TT_SPECIAL_GROUP -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2696) Container logs aren't getting cleaned up
[ https://issues.apache.org/jira/browse/MAPREDUCE-2696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth reassigned MAPREDUCE-2696: - Assignee: Siddharth Seth Container logs aren't getting cleaned up Key: MAPREDUCE-2696 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2696 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Siddharth Seth Priority: Blocker Fix For: 0.23.0 Container logs aren't getting cleaned up causing NMs to fill up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3103) Implement Job ACLs for MRAppMaster
[ https://issues.apache.org/jira/browse/MAPREDUCE-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3103: - Priority: Blocker (was: Major) Implement Job ACLs for MRAppMaster -- Key: MAPREDUCE-3103 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3103 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: security Reporter: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir after job completion
[
https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13130004#comment-13130004
]
Arun C Murthy commented on MAPREDUCE-2762:
--
+1
[MR-279] - Cleanup staging dir after job completion
---
Key: MAPREDUCE-2762
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Ramya Sunil
Assignee: Mahadev konar
Priority: Blocker
Fix For: 0.23.0
Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch,
MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch,
MAPREDUCE-2762.patch
The files created under the staging dir have to be deleted after job
completion. Currently, all job.* files remain forever in the
${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3144) Augment JobHistory to include information needed for serving aggregated logs.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3144: -- Status: Patch Available (was: Open) Augment JobHistory to include information needed for serving aggregated logs. - Key: MAPREDUCE-3144 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3144 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Siddharth Seth Priority: Critical Fix For: 0.23.0 Attachments: MR3144.patch, MR3144_v2.txt, MR3144_v3.txt, MR3144_v4.txt, MR3144_v5.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3144) Augment JobHistory to include information needed for serving aggregated logs.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13130024#comment-13130024 ] Hadoop QA commented on MAPREDUCE-3144: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12499576/MR3144_v5.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 42 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 1713 javac compiler warnings (more than the trunk's current 1711 warnings). -1 findbugs. The patch appears to introduce 161 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.mapreduce.TestJobMonitorAndPrint +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1056//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1056//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1056//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1056//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1056//console This message is automatically generated. Augment JobHistory to include information needed for serving aggregated logs. - Key: MAPREDUCE-3144 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3144 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Siddharth Seth Priority: Critical Fix For: 0.23.0 Attachments: MR3144.patch, MR3144_v2.txt, MR3144_v3.txt, MR3144_v4.txt, MR3144_v5.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir after job completion
[
https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-2762:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks Mahadev!
[MR-279] - Cleanup staging dir after job completion
---
Key: MAPREDUCE-2762
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Ramya Sunil
Assignee: Mahadev konar
Priority: Blocker
Fix For: 0.23.0
Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch,
MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch,
MAPREDUCE-2762.patch
The files created under the staging dir have to be deleted after job
completion. Currently, all job.* files remain forever in the
${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir after job completion
[
https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13130041#comment-13130041
]
Hudson commented on MAPREDUCE-2762:
---
Integrated in Hadoop-Hdfs-0.23-Commit #17 (See
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/17/])
Merge -c 1185880 from trunk to branch-0.23 to complete fix for
MAPREDUCE-2762.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185881
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java
[MR-279] - Cleanup staging dir after job completion
---
Key: MAPREDUCE-2762
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Ramya Sunil
Assignee: Mahadev konar
Priority: Blocker
Fix For: 0.23.0
Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch,
MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch,
MAPREDUCE-2762.patch
The files created under the staging dir have to be deleted after job
completion. Currently, all job.* files remain forever in the
${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3165) Ensure logging option is set on child command line
[ https://issues.apache.org/jira/browse/MAPREDUCE-3165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3165: - Resolution: Fixed Status: Resolved (was: Patch Available) I just committed this. Thanks Todd! Ensure logging option is set on child command line -- Key: MAPREDUCE-3165 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3165 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Reporter: Arun C Murthy Assignee: Todd Lipcon Priority: Blocker Fix For: 0.23.0 Attachments: mr-3165.txt, mr-3165.txt Currently the logging config is set in env in MapReduceChildJVM - we need to set it on command line. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir after job completion
[
https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13130048#comment-13130048
]
Hudson commented on MAPREDUCE-2762:
---
Integrated in Hadoop-Common-0.23-Commit #16 (See
[https://builds.apache.org/job/Hadoop-Common-0.23-Commit/16/])
Merge -c 1185880 from trunk to branch-0.23 to complete fix for
MAPREDUCE-2762.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185881
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java
[MR-279] - Cleanup staging dir after job completion
---
Key: MAPREDUCE-2762
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Ramya Sunil
Assignee: Mahadev konar
Priority: Blocker
Fix For: 0.23.0
Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch,
MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch,
MAPREDUCE-2762.patch
The files created under the staging dir have to be deleted after job
completion. Currently, all job.* files remain forever in the
${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir after job completion
[
https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13130046#comment-13130046
]
Hudson commented on MAPREDUCE-2762:
---
Integrated in Hadoop-Hdfs-trunk-Commit #1181 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1181/])
MAPREDUCE-2762. Cleanup MR staging directory on completion. Contributed by
Mahadev Konar.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185880
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java
[MR-279] - Cleanup staging dir after job completion
---
Key: MAPREDUCE-2762
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Ramya Sunil
Assignee: Mahadev konar
Priority: Blocker
Fix For: 0.23.0
Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch,
MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch,
MAPREDUCE-2762.patch
The files created under the staging dir have to be deleted after job
completion. Currently, all job.* files remain forever in the
${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3165) Ensure logging option is set on child command line
[ https://issues.apache.org/jira/browse/MAPREDUCE-3165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13130054#comment-13130054 ] Hudson commented on MAPREDUCE-3165: --- Integrated in Hadoop-Hdfs-0.23-Commit #18 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/18/]) Merge -c 1185887 from trunk to branch-0.23 to complete fix for MAPREDUCE-3165. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185888 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java Ensure logging option is set on child command line -- Key: MAPREDUCE-3165 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3165 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Reporter: Arun C Murthy Assignee: Todd Lipcon Priority: Blocker Fix For: 0.23.0 Attachments: mr-3165.txt, mr-3165.txt Currently the logging config is set in env in MapReduceChildJVM - we need to set it on command line. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
