[jira] [Updated] (MAPREDUCE-4463) JobTracker recovery fails with HDFS permission issue
[ https://issues.apache.org/jira/browse/MAPREDUCE-4463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-4463: - Priority: Blocker (was: Major) JobTracker recovery fails with HDFS permission issue Key: MAPREDUCE-4463 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4463 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 1.2.0 Reporter: Tom White Assignee: Tom White Priority: Blocker Attachments: MAPREDUCE-4463.patch, MAPREDUCE-4463.patch Recovery fails when the job user is different to the JT owner (i.e. on anything bigger than a pseudo-distributed cluster). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4463) JobTracker recovery fails with HDFS permission issue
[ https://issues.apache.org/jira/browse/MAPREDUCE-4463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621965#comment-13621965 ] Arun C Murthy commented on MAPREDUCE-4463: -- Marking this as a blocker for 1.2. I'll run tests again and commit. JobTracker recovery fails with HDFS permission issue Key: MAPREDUCE-4463 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4463 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 1.2.0 Reporter: Tom White Assignee: Tom White Priority: Blocker Attachments: MAPREDUCE-4463.patch, MAPREDUCE-4463.patch Recovery fails when the job user is different to the JT owner (i.e. on anything bigger than a pseudo-distributed cluster). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4463) JobTracker recovery fails with HDFS permission issue
[ https://issues.apache.org/jira/browse/MAPREDUCE-4463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621994#comment-13621994 ] Arun C Murthy commented on MAPREDUCE-4463: -- [~tomwhite] This patch seems wildly off for branch-1. Will you have time to rebase this? What revision did you patch this against? Tx. JobTracker recovery fails with HDFS permission issue Key: MAPREDUCE-4463 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4463 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 1.2.0 Reporter: Tom White Assignee: Tom White Priority: Blocker Attachments: MAPREDUCE-4463.patch, MAPREDUCE-4463.patch Recovery fails when the job user is different to the JT owner (i.e. on anything bigger than a pseudo-distributed cluster). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5126) Add possibility to set a custom system classloader for mapred child processes, separate from mapred.child.java.opts
Piotr Kołaczkowski created MAPREDUCE-5126: - Summary: Add possibility to set a custom system classloader for mapred child processes, separate from mapred.child.java.opts Key: MAPREDUCE-5126 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5126 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Piotr Kołaczkowski Priority: Minor Some third party frameworks / systems based on Hadoop might want to set a custom classloader for loading classes of their jobs to better resolve conflicts with their libraries. While it is possible to set a custom classloader using the mapred.child.java.opts, this field is often overriden by users in their job configuration. So in order to change e.g. heap sizes the user would need to remember also to include the custom classloader property from the framework-defaults or otherwise he would break the framework. This small patch introduces another parameter: mapred.child.java.class.loader that allows to set the classloader separately. This gives custom frameworks built on top of Hadoop more flexibility to supply their own classloader, without need to force users to adjust any settings. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5126) Add possibility to set a custom system classloader for mapred child processes, separate from mapred.child.java.opts
[ https://issues.apache.org/jira/browse/MAPREDUCE-5126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piotr Kołaczkowski updated MAPREDUCE-5126: -- Attachment: custom-classloader.patch Add possibility to set a custom system classloader for mapred child processes, separate from mapred.child.java.opts --- Key: MAPREDUCE-5126 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5126 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Piotr Kołaczkowski Priority: Minor Attachments: custom-classloader.patch Some third party frameworks / systems based on Hadoop might want to set a custom classloader for loading classes of their jobs to better resolve conflicts with their libraries. While it is possible to set a custom classloader using the mapred.child.java.opts, this field is often overriden by users in their job configuration. So in order to change e.g. heap sizes the user would need to remember also to include the custom classloader property from the framework-defaults or otherwise he would break the framework. This small patch introduces another parameter: mapred.child.java.class.loader that allows to set the classloader separately. This gives custom frameworks built on top of Hadoop more flexibility to supply their own classloader, without need to force users to adjust any settings. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5126) Add possibility to set a custom system classloader for mapred child processes, separate from mapred.child.java.opts
[ https://issues.apache.org/jira/browse/MAPREDUCE-5126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piotr Kołaczkowski updated MAPREDUCE-5126: -- Affects Version/s: 1.0.4 Add possibility to set a custom system classloader for mapred child processes, separate from mapred.child.java.opts --- Key: MAPREDUCE-5126 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5126 Project: Hadoop Map/Reduce Issue Type: New Feature Affects Versions: 1.0.4 Reporter: Piotr Kołaczkowski Priority: Minor Attachments: custom-classloader.patch Some third party frameworks / systems based on Hadoop might want to set a custom classloader for loading classes of their jobs to better resolve conflicts with their libraries. While it is possible to set a custom classloader using the mapred.child.java.opts, this field is often overriden by users in their job configuration. So in order to change e.g. heap sizes the user would need to remember also to include the custom classloader property from the framework-defaults or otherwise he would break the framework. This small patch introduces another parameter: mapred.child.java.class.loader that allows to set the classloader separately. This gives custom frameworks built on top of Hadoop more flexibility to supply their own classloader, without need to force users to adjust any settings. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4991) coverage for gridmix
[ https://issues.apache.org/jira/browse/MAPREDUCE-4991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622011#comment-13622011 ] Hudson commented on MAPREDUCE-4991: --- Integrated in Hadoop-Yarn-trunk #174 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/174/]) MAPREDUCE-4991. coverage for gridmix (Aleksey Gorshkov via tgraves) (Revision 1463806) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1463806 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/pom.xml * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/Gridmix.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/SerialJobFactory.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/CommonJobTest.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/DebugJobFactory.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/DebugJobProducer.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/GridmixTestUtils.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestDistCacheEmulation.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestGridMixClasses.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestGridmixSubmission.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestGridmixSummary.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestLoadJob.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestSleepJob.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/resources/data/wordcount.json * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/resources/data/wordcount2.json coverage for gridmix Key: MAPREDUCE-4991 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4991 Project: Hadoop Map/Reduce Issue Type: Test Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.7 Reporter: Aleksey Gorshkov Assignee: Aleksey Gorshkov Fix For: 3.0.0, 0.23.7, 2.0.5-beta Attachments: MAPREDUCE-4991-branch-0.23-a.patch, MAPREDUCE-4991-branch-0.23-b.patch, MAPREDUCE-4991-branch-0.23.patch, MAPREDUCE-4991-branch-2-b.patch, MAPREDUCE-4991-branch-2.patch, MAPREDUCE-4991-trunk-a.patch, MAPREDUCE-4991-trunk-b.patch, MAPREDUCE-4991-trunk.patch fix coverage for GridMix MAPREDUCE-4991-trunk.patch patch for thunk MAPREDUCE-4991-branch-2.patch for branch-2 and MAPREDUCE-4991-branch-0.23.patch for branch-0.23 known fail -org.apache.hadoop.mapred.gridmix.TestGridmixSummary.testExecutionSummarizer. It is for next issue -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5117) With security enabled HS delegation token renewer fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-5117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622017#comment-13622017
]
Hudson commented on MAPREDUCE-5117:
---
Integrated in Hadoop-Yarn-trunk #174 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/174/])
MAPREDUCE-5117. Changed MRClientProtocolPBClientImpl to be closeable and
thus fix failures in renewal of HistoryServer's delegations tokens. Contributed
by Siddharth Seth. (Revision 1463828)
Result = SUCCESS
vinodkv :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1463828
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/impl/pb/client/MRClientProtocolPBClientImpl.java
With security enabled HS delegation token renewer fails
---
Key: MAPREDUCE-5117
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5117
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: security
Affects Versions: 2.0.4-alpha
Reporter: Roman Shaposhnik
Assignee: Siddharth Seth
Priority: Blocker
Fix For: 2.0.4-alpha
Attachments: MAPREDUCE-5117.txt, yarn.log
It seems that the HSClientProtocolPBClientImpl should implement Closeable as
per the attached stack trace. The problem can be observed on a cluster
running the latest branch-2.0.4-alpha with MAPREDUCE-5088 applied on top. The
easiest way to reproduce it is to run an oozie pig job:
{noformat}
$ oozie job -oozie http://`hostname -f`:11000/oozie -run
-DjobTracker=`hostname -f`:8032 -DnameNode=hdfs://`hostname -f`:17020
-DexamplesRoot=examples -config /tmp/examples/apps/pig/job.properties
{noformat}
Please also note that I can successfully submit simple jobs (Pi/Sleep) from a
command line using hadoop jar command. Thus it *seems* related to
MAPREDUCE-5088 change.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5098) Fix findbugs warnings in gridmix
[ https://issues.apache.org/jira/browse/MAPREDUCE-5098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622027#comment-13622027 ] Hudson commented on MAPREDUCE-5098: --- Integrated in Hadoop-Yarn-trunk #174 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/174/]) MAPREDUCE-5098. Fix findbugs warnings in gridmix. (kkambatl via tucu) (Revision 1464209) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1464209 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/dev-support * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/dev-support/findbugs-exclude.xml * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/pom.xml * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/JobMonitor.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/SleepJob.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/emulators/resourceusage/TotalHeapUsageEmulatorPlugin.java Fix findbugs warnings in gridmix Key: MAPREDUCE-5098 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5098 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/gridmix Affects Versions: 2.0.3-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Labels: findbugs Attachments: mr-5098.patch, mr-5098.patch, mr-5098.patch Work on MAPREDUCE-5077 has exposed a bunch of findbugs warnings in gridmix code. https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3459//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-gridmix.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4991) coverage for gridmix
[ https://issues.apache.org/jira/browse/MAPREDUCE-4991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622101#comment-13622101 ] Hudson commented on MAPREDUCE-4991: --- Integrated in Hadoop-Hdfs-0.23-Build #572 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/572/]) MAPREDUCE-4991. coverage for gridmix (Aleksey Gorshkov via tgraves) (Revision 1463807) Result = UNSTABLE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1463807 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-gridmix/pom.xml * /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/Gridmix.java * /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/SerialJobFactory.java * /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/CommonJobTest.java * /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/DebugJobFactory.java * /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/DebugJobProducer.java * /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/GridmixTestUtils.java * /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestDistCacheEmulation.java * /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestGridMixClasses.java * /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestGridmixSubmission.java * /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestGridmixSummary.java * /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestLoadJob.java * /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestSleepJob.java * /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-gridmix/src/test/resources/data/wordcount.json * /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-gridmix/src/test/resources/data/wordcount2.json coverage for gridmix Key: MAPREDUCE-4991 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4991 Project: Hadoop Map/Reduce Issue Type: Test Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.7 Reporter: Aleksey Gorshkov Assignee: Aleksey Gorshkov Fix For: 3.0.0, 0.23.7, 2.0.5-beta Attachments: MAPREDUCE-4991-branch-0.23-a.patch, MAPREDUCE-4991-branch-0.23-b.patch, MAPREDUCE-4991-branch-0.23.patch, MAPREDUCE-4991-branch-2-b.patch, MAPREDUCE-4991-branch-2.patch, MAPREDUCE-4991-trunk-a.patch, MAPREDUCE-4991-trunk-b.patch, MAPREDUCE-4991-trunk.patch fix coverage for GridMix MAPREDUCE-4991-trunk.patch patch for thunk MAPREDUCE-4991-branch-2.patch for branch-2 and MAPREDUCE-4991-branch-0.23.patch for branch-0.23 known fail -org.apache.hadoop.mapred.gridmix.TestGridmixSummary.testExecutionSummarizer. It is for next issue -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4463) JobTracker recovery fails with HDFS permission issue
[ https://issues.apache.org/jira/browse/MAPREDUCE-4463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-4463: - Fix Version/s: 1.2.0 Status: Patch Available (was: Open) JobTracker recovery fails with HDFS permission issue Key: MAPREDUCE-4463 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4463 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 1.2.0 Reporter: Tom White Assignee: Tom White Priority: Blocker Fix For: 1.2.0 Attachments: MAPREDUCE-4463.patch, MAPREDUCE-4463.patch, MAPREDUCE-4463.patch Recovery fails when the job user is different to the JT owner (i.e. on anything bigger than a pseudo-distributed cluster). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4463) JobTracker recovery fails with HDFS permission issue
[ https://issues.apache.org/jira/browse/MAPREDUCE-4463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622119#comment-13622119 ] Arun C Murthy commented on MAPREDUCE-4463: -- Since I haven't changed the meat of the patch which are straight-fwd and have been reviewed, I'll commit this shortly. Thanks. JobTracker recovery fails with HDFS permission issue Key: MAPREDUCE-4463 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4463 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 1.2.0 Reporter: Tom White Assignee: Tom White Priority: Blocker Fix For: 1.2.0 Attachments: MAPREDUCE-4463.patch, MAPREDUCE-4463.patch, MAPREDUCE-4463.patch Recovery fails when the job user is different to the JT owner (i.e. on anything bigger than a pseudo-distributed cluster). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4991) coverage for gridmix
[ https://issues.apache.org/jira/browse/MAPREDUCE-4991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622137#comment-13622137 ] Hudson commented on MAPREDUCE-4991: --- Integrated in Hadoop-Hdfs-trunk #1363 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1363/]) MAPREDUCE-4991. coverage for gridmix (Aleksey Gorshkov via tgraves) (Revision 1463806) Result = FAILURE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1463806 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/pom.xml * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/Gridmix.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/SerialJobFactory.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/CommonJobTest.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/DebugJobFactory.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/DebugJobProducer.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/GridmixTestUtils.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestDistCacheEmulation.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestGridMixClasses.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestGridmixSubmission.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestGridmixSummary.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestLoadJob.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestSleepJob.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/resources/data/wordcount.json * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/resources/data/wordcount2.json coverage for gridmix Key: MAPREDUCE-4991 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4991 Project: Hadoop Map/Reduce Issue Type: Test Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.7 Reporter: Aleksey Gorshkov Assignee: Aleksey Gorshkov Fix For: 3.0.0, 0.23.7, 2.0.5-beta Attachments: MAPREDUCE-4991-branch-0.23-a.patch, MAPREDUCE-4991-branch-0.23-b.patch, MAPREDUCE-4991-branch-0.23.patch, MAPREDUCE-4991-branch-2-b.patch, MAPREDUCE-4991-branch-2.patch, MAPREDUCE-4991-trunk-a.patch, MAPREDUCE-4991-trunk-b.patch, MAPREDUCE-4991-trunk.patch fix coverage for GridMix MAPREDUCE-4991-trunk.patch patch for thunk MAPREDUCE-4991-branch-2.patch for branch-2 and MAPREDUCE-4991-branch-0.23.patch for branch-0.23 known fail -org.apache.hadoop.mapred.gridmix.TestGridmixSummary.testExecutionSummarizer. It is for next issue -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5088) MR Client gets an renewer token exception while Oozie is submitting a job
[
https://issues.apache.org/jira/browse/MAPREDUCE-5088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622150#comment-13622150
]
Hudson commented on MAPREDUCE-5088:
---
Integrated in Hadoop-Hdfs-trunk #1363 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1363/])
MAPREDUCE-5088. MR Client gets an renewer token exception while Oozie is
submitting a job (daryn)
merge -c1463804 from branch-2.0.4-alpha (Revision 1464153)
Result = FAILURE
cos : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1464153
Files :
* /hadoop/common/trunk
* /hadoop/common/trunk/hadoop-mapreduce-project
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestYARNRunner.java
MR Client gets an renewer token exception while Oozie is submitting a job
-
Key: MAPREDUCE-5088
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5088
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.0.3-alpha
Reporter: Roman Shaposhnik
Assignee: Daryn Sharp
Priority: Blocker
Fix For: 2.0.4-alpha
Attachments: HADOOP-9409.patch, HADOOP-9409.patch,
MAPREDUCE-5088.patch, MAPREDUCE-5088.patch, MAPREDUCE-5088.txt
After the fix for HADOOP-9299 I'm now getting the following bizzare exception
in Oozie while trying to submit a job. This also seems to be KRB related:
{noformat}
2013-03-15 13:34:16,555 WARN ActionStartXCommand:542 - USER[hue] GROUP[-]
TOKEN[] APP[MapReduce] JOB[001-130315123130987-oozie-oozi-W]
ACTION[001-130315123130987-oozie-oozi-W@Sleep] Error starting action
[Sleep]. ErrorType [ERROR], ErrorCode [UninitializedMessageException],
Message [UninitializedMessageException: Message missing required fields:
renewer]
org.apache.oozie.action.ActionExecutorException:
UninitializedMessageException: Message missing required fields: renewer
at
org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:401)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:738)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:889)
at
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:211)
at
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:59)
at org.apache.oozie.command.XCommand.call(XCommand.java:277)
at
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:326)
at
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:255)
at
org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: com.google.protobuf.UninitializedMessageException: Message missing
required fields: renewer
at
com.google.protobuf.AbstractMessage$Builder.newUninitializedMessageException(AbstractMessage.java:605)
at
org.apache.hadoop.security.proto.SecurityProtos$GetDelegationTokenRequestProto$Builder.build(SecurityProtos.java:973)
at
org.apache.hadoop.mapreduce.v2.api.protocolrecords.impl.pb.GetDelegationTokenRequestPBImpl.mergeLocalToProto(GetDelegationTokenRequestPBImpl.java:84)
at
org.apache.hadoop.mapreduce.v2.api.protocolrecords.impl.pb.GetDelegationTokenRequestPBImpl.getProto(GetDelegationTokenRequestPBImpl.java:67)
at
org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getDelegationToken(MRClientProtocolPBClientImpl.java:200)
at
org.apache.hadoop.mapred.YARNRunner.getDelegationTokenFromHS(YARNRunner.java:194)
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:273)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:392)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
[jira] [Updated] (MAPREDUCE-4463) JobTracker recovery fails with HDFS permission issue
[ https://issues.apache.org/jira/browse/MAPREDUCE-4463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-4463: - Resolution: Fixed Status: Resolved (was: Patch Available) I just committed this. Thanks Tom! JobTracker recovery fails with HDFS permission issue Key: MAPREDUCE-4463 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4463 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 1.2.0 Reporter: Tom White Assignee: Tom White Priority: Blocker Fix For: 1.2.0 Attachments: MAPREDUCE-4463.patch, MAPREDUCE-4463.patch, MAPREDUCE-4463.patch Recovery fails when the job user is different to the JT owner (i.e. on anything bigger than a pseudo-distributed cluster). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5079) Recovery should restore task state from job history info directly
[ https://issues.apache.org/jira/browse/MAPREDUCE-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-5079: - Priority: Critical (was: Blocker) Recovery should restore task state from job history info directly - Key: MAPREDUCE-5079 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5079 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mr-am Affects Versions: 0.23.7 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Critical Attachments: MAPREDUCE-5079.patch We've encountered a lot of hanging issues during MR-AM recovery because the state machines don't always end up in the same states after recovery. This is especially true when speculative execution is enabled. It should be straightforward to restore task and task attempt states directly from the TaskInfo and TaskAttemptInfo records in the job history file to avoid relying on the task state machines ending up in the proper states with the proper number of attempts. This should be a more robust solution that would also give us the option of recovering start time and log locations for tasks that were in-progress when the AM crashed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5079) Recovery should restore task state from job history info directly
[ https://issues.apache.org/jira/browse/MAPREDUCE-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-5079: - Target Version/s: 3.0.0, 0.23.7, 2.0.5-beta (was: 3.0.0, 2.0.5-beta, 0.23.8) Recovery should restore task state from job history info directly - Key: MAPREDUCE-5079 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5079 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mr-am Affects Versions: 0.23.7 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Critical Attachments: MAPREDUCE-5079.patch We've encountered a lot of hanging issues during MR-AM recovery because the state machines don't always end up in the same states after recovery. This is especially true when speculative execution is enabled. It should be straightforward to restore task and task attempt states directly from the TaskInfo and TaskAttemptInfo records in the job history file to avoid relying on the task state machines ending up in the proper states with the proper number of attempts. This should be a more robust solution that would also give us the option of recovering start time and log locations for tasks that were in-progress when the AM crashed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5079) Recovery should restore task state from job history info directly
[ https://issues.apache.org/jira/browse/MAPREDUCE-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-5079: - Priority: Blocker (was: Major) Recovery should restore task state from job history info directly - Key: MAPREDUCE-5079 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5079 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mr-am Affects Versions: 0.23.7 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Blocker Attachments: MAPREDUCE-5079.patch We've encountered a lot of hanging issues during MR-AM recovery because the state machines don't always end up in the same states after recovery. This is especially true when speculative execution is enabled. It should be straightforward to restore task and task attempt states directly from the TaskInfo and TaskAttemptInfo records in the job history file to avoid relying on the task state machines ending up in the proper states with the proper number of attempts. This should be a more robust solution that would also give us the option of recovering start time and log locations for tasks that were in-progress when the AM crashed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5007) fix coverage org.apache.hadoop.mapreduce.v2.hs
[ https://issues.apache.org/jira/browse/MAPREDUCE-5007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Gorshkov updated MAPREDUCE-5007: Attachment: MAPREDUCE-5007-trunk-b.patch MAPREDUCE-5007-branch-2-b.patch MAPREDUCE-5007-branch-0.23-b.patch fix coverage org.apache.hadoop.mapreduce.v2.hs -- Key: MAPREDUCE-5007 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5007 Project: Hadoop Map/Reduce Issue Type: Test Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.7 Reporter: Aleksey Gorshkov Assignee: Aleksey Gorshkov Attachments: MAPREDUCE-5007-branch-0.23-a.patch, MAPREDUCE-5007-branch-0.23-b.patch, MAPREDUCE-5007-branch-0.23.patch, MAPREDUCE-5007-branch-2-a.patch, MAPREDUCE-5007-branch-2-b.patch, MAPREDUCE-5007-branch-2.patch, MAPREDUCE-5007-trunk-a.patch, MAPREDUCE-5007-trunk-b.patch, MAPREDUCE-5007-trunk.patch fix coverage org.apache.hadoop.mapreduce.v2.hs MAPREDUCE-5007-trunk.patch patch for trunk MAPREDUCE-5007-branch-2.patch patch for branch-2 MAPREDUCE-5007-branch-0.23.patch patch for branch-0.23 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5007) fix coverage org.apache.hadoop.mapreduce.v2.hs
[ https://issues.apache.org/jira/browse/MAPREDUCE-5007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622214#comment-13622214 ] Aleksey Gorshkov commented on MAPREDUCE-5007: - Thanks Thomas for your help. Could you test this issue. I cannot reproduce this in my computers. I've updated patch. MAPREDUCE-5007-trunk-b.patch patch for trunk MAPREDUCE-5007-branch-2-b.patch patch for branch-2 MAPREDUCE-5007-branch-0.23-b.patch patch for branch-0.23 fix coverage org.apache.hadoop.mapreduce.v2.hs -- Key: MAPREDUCE-5007 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5007 Project: Hadoop Map/Reduce Issue Type: Test Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.7 Reporter: Aleksey Gorshkov Assignee: Aleksey Gorshkov Attachments: MAPREDUCE-5007-branch-0.23-a.patch, MAPREDUCE-5007-branch-0.23-b.patch, MAPREDUCE-5007-branch-0.23.patch, MAPREDUCE-5007-branch-2-a.patch, MAPREDUCE-5007-branch-2-b.patch, MAPREDUCE-5007-branch-2.patch, MAPREDUCE-5007-trunk-a.patch, MAPREDUCE-5007-trunk-b.patch, MAPREDUCE-5007-trunk.patch fix coverage org.apache.hadoop.mapreduce.v2.hs MAPREDUCE-5007-trunk.patch patch for trunk MAPREDUCE-5007-branch-2.patch patch for branch-2 MAPREDUCE-5007-branch-0.23.patch patch for branch-0.23 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4824) Provide a mechanism for jobs to indicate they should not be recovered on restart
[ https://issues.apache.org/jira/browse/MAPREDUCE-4824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-4824: - Attachment: MAPREDUCE-4824.patch Rebased patch. Tom, some changes for you to review: # I've renamed the config to be mapreduce.job.recover.on.restart to be more explicit/clear. We should use 'mapreduce' for new configs to avoid deprecations in future. # I've also introduced a static final MAPREDUCE_RECOVER_JOB variable in JobConf to avoid using the actual config string by hand. Provide a mechanism for jobs to indicate they should not be recovered on restart Key: MAPREDUCE-4824 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4824 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mrv1 Affects Versions: 1.1.0 Reporter: Tom White Assignee: Tom White Attachments: MAPREDUCE-4824.patch, MAPREDUCE-4824.patch, MAPREDUCE-4824.patch, MAPREDUCE-4824.patch, MAPREDUCE-4824.patch, MAPREDUCE-4824.patch, MAPREDUCE-4824.patch Some jobs (like Sqoop or HBase jobs) are not idempotent, so should not be recovered on jobtracker restart. MAPREDUCE-2702 solves this problem for MR2, however the approach there is not applicable for MR1, since even if we only use the job-level part of the patch and add a isRecoverySupported method to OutputCommitter, there is no way to use that information from the JT (which initiates recovery), since the JT does not instantiate OutputCommitters - and it shouldn't since they are user-level code. (In MR2 it's OK since the MR AM calls the method.) Instead, we can add a MR configuration property to say that a job is not recoverable, and the JT could safely read this from the job conf. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5117) With security enabled HS delegation token renewer fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-5117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622143#comment-13622143
]
Hudson commented on MAPREDUCE-5117:
---
Integrated in Hadoop-Hdfs-trunk #1363 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1363/])
MAPREDUCE-5117. Changed MRClientProtocolPBClientImpl to be closeable and
thus fix failures in renewal of HistoryServer's delegations tokens. Contributed
by Siddharth Seth. (Revision 1463828)
Result = FAILURE
vinodkv :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1463828
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/impl/pb/client/MRClientProtocolPBClientImpl.java
With security enabled HS delegation token renewer fails
---
Key: MAPREDUCE-5117
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5117
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: security
Affects Versions: 2.0.4-alpha
Reporter: Roman Shaposhnik
Assignee: Siddharth Seth
Priority: Blocker
Fix For: 2.0.4-alpha
Attachments: MAPREDUCE-5117.txt, yarn.log
It seems that the HSClientProtocolPBClientImpl should implement Closeable as
per the attached stack trace. The problem can be observed on a cluster
running the latest branch-2.0.4-alpha with MAPREDUCE-5088 applied on top. The
easiest way to reproduce it is to run an oozie pig job:
{noformat}
$ oozie job -oozie http://`hostname -f`:11000/oozie -run
-DjobTracker=`hostname -f`:8032 -DnameNode=hdfs://`hostname -f`:17020
-DexamplesRoot=examples -config /tmp/examples/apps/pig/job.properties
{noformat}
Please also note that I can successfully submit simple jobs (Pi/Sleep) from a
command line using hadoop jar command. Thus it *seems* related to
MAPREDUCE-5088 change.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5086) MR app master deletes staging dir when sent a reboot command from the RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-5086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622152#comment-13622152 ] Hudson commented on MAPREDUCE-5086: --- Integrated in Hadoop-Hdfs-trunk #1363 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1363/]) MAPREDUCE-5086. MR app master deletes staging dir when sent a reboot command from the RM. Contributed by Jian He (Revision 1464255) Result = FAILURE jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1464255 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/JobStateInternal.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/event/JobEventType.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/local/LocalContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java MR app master deletes staging dir when sent a reboot command from the RM Key: MAPREDUCE-5086 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5086 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jian He Assignee: Jian He Attachments: YARN-472.1.patch, YARN-472.2.patch, YARN-472.3.patch, YARN-472.4.patch, YARN-472.5.patch, YARN-472.6.patch, YARN-472.7.patch If the RM is restarted when the MR job is running, then it sends a reboot command to the job. The job ends up deleting the staging dir and that causes the next attempt to fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5098) Fix findbugs warnings in gridmix
[ https://issues.apache.org/jira/browse/MAPREDUCE-5098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622153#comment-13622153 ] Hudson commented on MAPREDUCE-5098: --- Integrated in Hadoop-Hdfs-trunk #1363 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1363/]) MAPREDUCE-5098. Fix findbugs warnings in gridmix. (kkambatl via tucu) (Revision 1464209) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1464209 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/dev-support * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/dev-support/findbugs-exclude.xml * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/pom.xml * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/JobMonitor.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/SleepJob.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/emulators/resourceusage/TotalHeapUsageEmulatorPlugin.java Fix findbugs warnings in gridmix Key: MAPREDUCE-5098 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5098 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/gridmix Affects Versions: 2.0.3-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Labels: findbugs Attachments: mr-5098.patch, mr-5098.patch, mr-5098.patch Work on MAPREDUCE-5077 has exposed a bunch of findbugs warnings in gridmix code. https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3459//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-gridmix.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4463) JobTracker recovery fails with HDFS permission issue
[ https://issues.apache.org/jira/browse/MAPREDUCE-4463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-4463: - Attachment: MAPREDUCE-4463.patch Rebased - looks like MAPREDUCE-4859 already introduced the TaskTracker changes. Also, I made some simplifying changes to the new test in TestRecoveryManager - this way we don't start multiple MiniMRClusters via setup and clusterSetup methods in other tests in the same file. JobTracker recovery fails with HDFS permission issue Key: MAPREDUCE-4463 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4463 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 1.2.0 Reporter: Tom White Assignee: Tom White Priority: Blocker Attachments: MAPREDUCE-4463.patch, MAPREDUCE-4463.patch, MAPREDUCE-4463.patch Recovery fails when the job user is different to the JT owner (i.e. on anything bigger than a pseudo-distributed cluster). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4991) coverage for gridmix
[ https://issues.apache.org/jira/browse/MAPREDUCE-4991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622331#comment-13622331 ] Hudson commented on MAPREDUCE-4991: --- Integrated in Hadoop-Mapreduce-trunk #1390 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1390/]) MAPREDUCE-4991. coverage for gridmix (Aleksey Gorshkov via tgraves) (Revision 1463806) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1463806 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/pom.xml * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/Gridmix.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/SerialJobFactory.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/CommonJobTest.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/DebugJobFactory.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/DebugJobProducer.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/GridmixTestUtils.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestDistCacheEmulation.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestGridMixClasses.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestGridmixSubmission.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestGridmixSummary.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestLoadJob.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestSleepJob.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/resources/data/wordcount.json * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/resources/data/wordcount2.json coverage for gridmix Key: MAPREDUCE-4991 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4991 Project: Hadoop Map/Reduce Issue Type: Test Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.7 Reporter: Aleksey Gorshkov Assignee: Aleksey Gorshkov Fix For: 3.0.0, 0.23.7, 2.0.5-beta Attachments: MAPREDUCE-4991-branch-0.23-a.patch, MAPREDUCE-4991-branch-0.23-b.patch, MAPREDUCE-4991-branch-0.23.patch, MAPREDUCE-4991-branch-2-b.patch, MAPREDUCE-4991-branch-2.patch, MAPREDUCE-4991-trunk-a.patch, MAPREDUCE-4991-trunk-b.patch, MAPREDUCE-4991-trunk.patch fix coverage for GridMix MAPREDUCE-4991-trunk.patch patch for thunk MAPREDUCE-4991-branch-2.patch for branch-2 and MAPREDUCE-4991-branch-0.23.patch for branch-0.23 known fail -org.apache.hadoop.mapred.gridmix.TestGridmixSummary.testExecutionSummarizer. It is for next issue -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5117) With security enabled HS delegation token renewer fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-5117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622337#comment-13622337
]
Hudson commented on MAPREDUCE-5117:
---
Integrated in Hadoop-Mapreduce-trunk #1390 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1390/])
MAPREDUCE-5117. Changed MRClientProtocolPBClientImpl to be closeable and
thus fix failures in renewal of HistoryServer's delegations tokens. Contributed
by Siddharth Seth. (Revision 1463828)
Result = SUCCESS
vinodkv :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1463828
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/impl/pb/client/MRClientProtocolPBClientImpl.java
With security enabled HS delegation token renewer fails
---
Key: MAPREDUCE-5117
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5117
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: security
Affects Versions: 2.0.4-alpha
Reporter: Roman Shaposhnik
Assignee: Siddharth Seth
Priority: Blocker
Fix For: 2.0.4-alpha
Attachments: MAPREDUCE-5117.txt, yarn.log
It seems that the HSClientProtocolPBClientImpl should implement Closeable as
per the attached stack trace. The problem can be observed on a cluster
running the latest branch-2.0.4-alpha with MAPREDUCE-5088 applied on top. The
easiest way to reproduce it is to run an oozie pig job:
{noformat}
$ oozie job -oozie http://`hostname -f`:11000/oozie -run
-DjobTracker=`hostname -f`:8032 -DnameNode=hdfs://`hostname -f`:17020
-DexamplesRoot=examples -config /tmp/examples/apps/pig/job.properties
{noformat}
Please also note that I can successfully submit simple jobs (Pi/Sleep) from a
command line using hadoop jar command. Thus it *seems* related to
MAPREDUCE-5088 change.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5088) MR Client gets an renewer token exception while Oozie is submitting a job
[
https://issues.apache.org/jira/browse/MAPREDUCE-5088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622345#comment-13622345
]
Hudson commented on MAPREDUCE-5088:
---
Integrated in Hadoop-Mapreduce-trunk #1390 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1390/])
MAPREDUCE-5088. MR Client gets an renewer token exception while Oozie is
submitting a job (daryn)
merge -c1463804 from branch-2.0.4-alpha (Revision 1464153)
Result = SUCCESS
cos : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1464153
Files :
* /hadoop/common/trunk
* /hadoop/common/trunk/hadoop-mapreduce-project
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestYARNRunner.java
MR Client gets an renewer token exception while Oozie is submitting a job
-
Key: MAPREDUCE-5088
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5088
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.0.3-alpha
Reporter: Roman Shaposhnik
Assignee: Daryn Sharp
Priority: Blocker
Fix For: 2.0.4-alpha
Attachments: HADOOP-9409.patch, HADOOP-9409.patch,
MAPREDUCE-5088.patch, MAPREDUCE-5088.patch, MAPREDUCE-5088.txt
After the fix for HADOOP-9299 I'm now getting the following bizzare exception
in Oozie while trying to submit a job. This also seems to be KRB related:
{noformat}
2013-03-15 13:34:16,555 WARN ActionStartXCommand:542 - USER[hue] GROUP[-]
TOKEN[] APP[MapReduce] JOB[001-130315123130987-oozie-oozi-W]
ACTION[001-130315123130987-oozie-oozi-W@Sleep] Error starting action
[Sleep]. ErrorType [ERROR], ErrorCode [UninitializedMessageException],
Message [UninitializedMessageException: Message missing required fields:
renewer]
org.apache.oozie.action.ActionExecutorException:
UninitializedMessageException: Message missing required fields: renewer
at
org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:401)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:738)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:889)
at
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:211)
at
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:59)
at org.apache.oozie.command.XCommand.call(XCommand.java:277)
at
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:326)
at
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:255)
at
org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: com.google.protobuf.UninitializedMessageException: Message missing
required fields: renewer
at
com.google.protobuf.AbstractMessage$Builder.newUninitializedMessageException(AbstractMessage.java:605)
at
org.apache.hadoop.security.proto.SecurityProtos$GetDelegationTokenRequestProto$Builder.build(SecurityProtos.java:973)
at
org.apache.hadoop.mapreduce.v2.api.protocolrecords.impl.pb.GetDelegationTokenRequestPBImpl.mergeLocalToProto(GetDelegationTokenRequestPBImpl.java:84)
at
org.apache.hadoop.mapreduce.v2.api.protocolrecords.impl.pb.GetDelegationTokenRequestPBImpl.getProto(GetDelegationTokenRequestPBImpl.java:67)
at
org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getDelegationToken(MRClientProtocolPBClientImpl.java:200)
at
org.apache.hadoop.mapred.YARNRunner.getDelegationTokenFromHS(YARNRunner.java:194)
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:273)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:392)
at
[jira] [Commented] (MAPREDUCE-5086) MR app master deletes staging dir when sent a reboot command from the RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-5086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622347#comment-13622347 ] Hudson commented on MAPREDUCE-5086: --- Integrated in Hadoop-Mapreduce-trunk #1390 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1390/]) MAPREDUCE-5086. MR app master deletes staging dir when sent a reboot command from the RM. Contributed by Jian He (Revision 1464255) Result = SUCCESS jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1464255 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/JobStateInternal.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/event/JobEventType.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/local/LocalContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java MR app master deletes staging dir when sent a reboot command from the RM Key: MAPREDUCE-5086 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5086 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jian He Assignee: Jian He Attachments: YARN-472.1.patch, YARN-472.2.patch, YARN-472.3.patch, YARN-472.4.patch, YARN-472.5.patch, YARN-472.6.patch, YARN-472.7.patch If the RM is restarted when the MR job is running, then it sends a reboot command to the job. The job ends up deleting the staging dir and that causes the next attempt to fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5098) Fix findbugs warnings in gridmix
[ https://issues.apache.org/jira/browse/MAPREDUCE-5098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622348#comment-13622348 ] Hudson commented on MAPREDUCE-5098: --- Integrated in Hadoop-Mapreduce-trunk #1390 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1390/]) MAPREDUCE-5098. Fix findbugs warnings in gridmix. (kkambatl via tucu) (Revision 1464209) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1464209 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/dev-support * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/dev-support/findbugs-exclude.xml * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/pom.xml * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/JobMonitor.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/SleepJob.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/emulators/resourceusage/TotalHeapUsageEmulatorPlugin.java Fix findbugs warnings in gridmix Key: MAPREDUCE-5098 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5098 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/gridmix Affects Versions: 2.0.3-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Labels: findbugs Attachments: mr-5098.patch, mr-5098.patch, mr-5098.patch Work on MAPREDUCE-5077 has exposed a bunch of findbugs warnings in gridmix code. https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3459//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-gridmix.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4974) Optimising the LineRecordReader initialize() method
[ https://issues.apache.org/jira/browse/MAPREDUCE-4974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622452#comment-13622452 ] Gelesh commented on MAPREDUCE-4974: --- Since this is just an optimization and existing test case would suffice, hope this is +1 Could some body kindly review. Optimising the LineRecordReader initialize() method --- Key: MAPREDUCE-4974 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4974 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1, mrv2, performance Affects Versions: 2.0.2-alpha, 0.23.5 Environment: Hadoop Linux Reporter: Arun A K Assignee: Gelesh Labels: patch, performance Fix For: 0.23.7, 2.0.5-beta Attachments: MAPREDUCE-4974.2.patch, MAPREDUCE-4974.3.patch, MAPREDUCE-4974.4.patch, MAPREDUCE-4974.5.patch Original Estimate: 1h Remaining Estimate: 1h I found there is a a scope of optimizing the code, over initialize() if we have compressionCodecs codec instantiated only if its a compressed input. Mean while Gelesh George Omathil, added if we could avoid the null check of key value. This would time save, since for every next key value generation, null check is done. The intention being to instantiate only once and avoid NPE as well. Hope both could be met if initialize key value over initialize() method. We both have worked on it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5083) MiniMRCluster should use a random component when creating an actual cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622520#comment-13622520 ] Hitesh Shah commented on MAPREDUCE-5083: @Stack, I will be committing shortly to branch-2.0.4. MiniMRCluster should use a random component when creating an actual cluster --- Key: MAPREDUCE-5083 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5083 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.0.3-alpha Reporter: Siddharth Seth Assignee: Siddharth Seth Fix For: 2.0.5-beta Attachments: MAPREDUCE-5083-branch2.txt, MAPREDUCE-5083-trunk_2.txt, MAPREDUCE-5083-trunk.txt Currently all unit tests end up using the same work dir - which can affect anyone trying to run parallel instances. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-5083) MiniMRCluster should use a random component when creating an actual cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah reopened MAPREDUCE-5083: MiniMRCluster should use a random component when creating an actual cluster --- Key: MAPREDUCE-5083 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5083 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.0.3-alpha Reporter: Siddharth Seth Assignee: Siddharth Seth Fix For: 2.0.5-beta Attachments: MAPREDUCE-5083-branch2.txt, MAPREDUCE-5083-trunk_2.txt, MAPREDUCE-5083-trunk.txt Currently all unit tests end up using the same work dir - which can affect anyone trying to run parallel instances. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5083) MiniMRCluster should use a random component when creating an actual cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah resolved MAPREDUCE-5083. Resolution: Fixed Fix Version/s: (was: 2.0.5-beta) 2.0.4-alpha Target Version/s: (was: 2.0.5-beta) Release Note: Committed to branch-2.0.4. Modified changes.txt in trunk, branch-2 and branch-2.0.4 accordingly. MiniMRCluster should use a random component when creating an actual cluster --- Key: MAPREDUCE-5083 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5083 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.0.3-alpha Reporter: Siddharth Seth Assignee: Siddharth Seth Fix For: 2.0.4-alpha Attachments: MAPREDUCE-5083-branch2.txt, MAPREDUCE-5083-trunk_2.txt, MAPREDUCE-5083-trunk.txt Currently all unit tests end up using the same work dir - which can affect anyone trying to run parallel instances. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5083) MiniMRCluster should use a random component when creating an actual cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622537#comment-13622537 ] Hitesh Shah commented on MAPREDUCE-5083: Minor clarification - changes.txt was modified in branch-2 and branch-2.0.4 - trunk has some additional mayhem to clear out first. MiniMRCluster should use a random component when creating an actual cluster --- Key: MAPREDUCE-5083 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5083 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.0.3-alpha Reporter: Siddharth Seth Assignee: Siddharth Seth Fix For: 2.0.4-alpha Attachments: MAPREDUCE-5083-branch2.txt, MAPREDUCE-5083-trunk_2.txt, MAPREDUCE-5083-trunk.txt Currently all unit tests end up using the same work dir - which can affect anyone trying to run parallel instances. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4885) streaming tests have multiple failures on Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622581#comment-13622581 ] Bikas Saha commented on MAPREDUCE-4885: --- Have we removed existing unix utilities dependencies? If not, then why not add cat/xargs Windows versions as a build/test requirement instead of adding code that needs to be maintained? This patch probably needs rebasing after MAPREDUCE-5006. A new patch will also help get a clean test run on Jenkins. streaming tests have multiple failures on Windows - Key: MAPREDUCE-4885 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4885 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/streaming, test Affects Versions: 3.0.0 Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: MAPREDUCE-4885.1.patch, MAPREDUCE-4885.2.patch There are multiple test failures due to Queue configuration missing child queue names for root. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Moved] (MAPREDUCE-5127) MR job succeeds and exits even when unregister with RM fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-5127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha moved YARN-529 to MAPREDUCE-5127: Component/s: (was: resourcemanager) resourcemanager applicationmaster Key: MAPREDUCE-5127 (was: YARN-529) Project: Hadoop Map/Reduce (was: Hadoop YARN) MR job succeeds and exits even when unregister with RM fails Key: MAPREDUCE-5127 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5127 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster, resourcemanager Reporter: Jian He Assignee: Jian He MR app master will clean staging dir, if the job is already succeeded and asked to reboot. If the finishApplicationMaster call fails, RM will consider this job unfinished and launch further attempts, further attempts will fail because staging dir is cleaned -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2389) Spurious EOFExceptions reading SpillRecord index files
[ https://issues.apache.org/jira/browse/MAPREDUCE-2389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622671#comment-13622671 ] Todd Lipcon commented on MAPREDUCE-2389: Still unclear why this happens on Jetty 6.1.26, but in CDH we've been shipping a custom build of Jetty for a while now - see https://issues.apache.org/jira/browse/MAPREDUCE-2980 -- and it seems to fix the issue. Spurious EOFExceptions reading SpillRecord index files -- Key: MAPREDUCE-2389 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2389 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.22.0 Environment: Seen on RHEL 5.5, RHEL 6.0, local dirs on ext3, Java 6u20 and 6u24 Reporter: Todd Lipcon Attachments: stap-output.txt In large jobs, I see around 1 shuffle fetch out of every million fetches fail with an EOFException reading the SpillRecord index file. After lots of investigation, including systemtap, it looks like the read() syscall is actually returning a premature 0 result for no reason, so this is likely a kernel or filesystem bug which is exacerbated by some workload the TT does. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5094) Disable mem monitoring by default in MiniMRYarnCluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-5094: -- Attachment: MAPREDUCE-5094.txt Patch disables VMEM monitoring by default. PMEM monitoring is enabled. Setting the parameters explicitly in conf is not enough - since the source of the configuration is not known (user or default). Controlling these values requires another parameter to be set. Disable mem monitoring by default in MiniMRYarnCluster -- Key: MAPREDUCE-5094 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5094 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Siddharth Seth Assignee: Siddharth Seth Attachments: MAPREDUCE-5094.txt YARN-449. Some hbase tests were failing since containers were getting killed. I believe these checks are disabled by default on the branch-1 MiniMRCluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5094) Disable mem monitoring by default in MiniMRYarnCluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-5094: -- Target Version/s: 2.0.4-alpha Affects Version/s: 2.0.3-alpha Status: Patch Available (was: Open) Disable mem monitoring by default in MiniMRYarnCluster -- Key: MAPREDUCE-5094 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5094 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Siddharth Seth Assignee: Siddharth Seth Attachments: MAPREDUCE-5094.txt YARN-449. Some hbase tests were failing since containers were getting killed. I believe these checks are disabled by default on the branch-1 MiniMRCluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4885) streaming tests have multiple failures on Windows
[
https://issues.apache.org/jira/browse/MAPREDUCE-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated MAPREDUCE-4885:
-
Attachment: MAPREDUCE-4885.3.patch
Attaching a rebased patch (though there were no conflicts).
{quote}
Have we removed existing unix utilities dependencies?
{quote}
The current status is that there are no Unix utility dependencies for runtime
or test. There are a few stragglers remaining as build-time dependencies only
for creating the distribution tarballs (sh, cp, rm, tar, gzip). HADOOP-9082
tracks permanent removal of those remaining build dependencies.
Considering that, it seemed unfavorable to re-introduce Unix utility
dependencies, and adding a couple of cmd one-liners doesn't seem like much
burden.
streaming tests have multiple failures on Windows
-
Key: MAPREDUCE-4885
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4885
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming, test
Affects Versions: 3.0.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: MAPREDUCE-4885.1.patch, MAPREDUCE-4885.2.patch,
MAPREDUCE-4885.3.patch
There are multiple test failures due to Queue configuration missing child
queue names for root.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3949) If AM fails due to overrunning resource limits, error not visible through UI sometimes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622823#comment-13622823 ] Ravi Prakash commented on MAPREDUCE-3949: - I discussed this with Tom and Daryn. * If the NM was able to tell the AM why its being shot, the AM could technically include that diagnostic message in the FinishApplicationMasterRequest however such a mechnanism (to tell the AM the reason its being shot) doesn't exist yet and would be very brittle. * If the NM told the RM before shooting the AM, there would still be a race. What if the AM had completed everything it needed to do, and then got shot by the NM? Then the job would have been successful but be marked as FAILED by the RM. * If we changed the State of a FINISHED / KILLED application to FAILED / FAILED on receiving RMAppAttemptEventType.CONTAINER_FINISHED, the client might still only get the FINISHED / KILLED message and the user would have to go to the RM page to see what really happened. Our current opinion is that this is probably the best way to go for now. Opinions anybody? If AM fails due to overrunning resource limits, error not visible through UI sometimes -- Key: MAPREDUCE-3949 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3949 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.24.0, 0.23.2 Reporter: Todd Lipcon Assignee: Ravi Prakash Priority: Minor I had a case where an MR AM eclipsed the configured memory limit. This caused the AM's container to get killed, but nowhere accessible through the web UI showed these diagnostics. I had to go view the NM's logs via ssh before I could figure out what had happened to my application. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5128) mapred-default.xml is missing a bunch of history server configs
[ https://issues.apache.org/jira/browse/MAPREDUCE-5128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-5128: -- Attachment: MAPREDUCE-5128.patch mapred-default.xml is missing a bunch of history server configs --- Key: MAPREDUCE-5128 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5128 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation, jobhistoryserver Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-5128.patch mapred-default.xml is missing many configs that work for the job history server. mapreduce.jobhistory.cleaner.enable, mapreduce.jobhistory.done-dir, and mapreduce.jobhistory.datestring.cache.size are a few examples. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5128) mapred-default.xml is missing a bunch of history server configs
[ https://issues.apache.org/jira/browse/MAPREDUCE-5128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-5128: -- Status: Patch Available (was: Open) mapred-default.xml is missing a bunch of history server configs --- Key: MAPREDUCE-5128 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5128 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation, jobhistoryserver Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-5128.patch mapred-default.xml is missing many configs that work for the job history server. mapreduce.jobhistory.cleaner.enable, mapreduce.jobhistory.done-dir, and mapreduce.jobhistory.datestring.cache.size are a few examples. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4885) streaming tests have multiple failures on Windows
[
https://issues.apache.org/jira/browse/MAPREDUCE-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622844#comment-13622844
]
Hadoop QA commented on MAPREDUCE-4885:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12577066/MAPREDUCE-4885.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 11 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-tools/hadoop-streaming.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3496//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3496//console
This message is automatically generated.
streaming tests have multiple failures on Windows
-
Key: MAPREDUCE-4885
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4885
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming, test
Affects Versions: 3.0.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: MAPREDUCE-4885.1.patch, MAPREDUCE-4885.2.patch,
MAPREDUCE-4885.3.patch
There are multiple test failures due to Queue configuration missing child
queue names for root.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5066) JobTracker should set a timeout when calling into job.end.notification.url
[ https://issues.apache.org/jira/browse/MAPREDUCE-5066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622851#comment-13622851 ] Ivan Mitic commented on MAPREDUCE-5066: --- Arun, did you get a chance to take a look at my latest branch-1/branch-2 patches? Please check my comment from above from some context. Thx! JobTracker should set a timeout when calling into job.end.notification.url -- Key: MAPREDUCE-5066 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5066 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1-win, 2.0.3-alpha, 1.3.0 Reporter: Ivan Mitic Assignee: Ivan Mitic Attachments: MAPREDUCE-5066.2.patch, MAPREDUCE-5066.branch-1-win.2.patch, MAPREDUCE-5066.branch-1-win.3.patch, MAPREDUCE-5066.branch-1-win.4.patch, MAPREDUCE-5066.branch-1-win.patch, MAPREDUCE-5066.patch In current code, timeout is not specified when JobTracker (JobEndNotifier) calls into the notification URL. When the given URL points to a server that will not respond for a long time, job notifications are completely stuck (given that we have only a single thread processing all notifications). We've seen this cause noticeable delays in job execution in components that rely on job end notifications (like Oozie workflows). I propose we introduce a configurable timeout option and set a default to a reasonably small value. If we want, we can also introduce a configurable number of workers processing the notification queue (not sure if this is needed though at this point). I will prepare a patch soon. Please comment back. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5056) TestProcfsBasedProcessTree fails on Windows with Process-tree dump doesn't start with a proper header
[ https://issues.apache.org/jira/browse/MAPREDUCE-5056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622852#comment-13622852 ] Ivan Mitic commented on MAPREDUCE-5056: --- Bikas, can you please take a look at the latest patch? Thx! TestProcfsBasedProcessTree fails on Windows with Process-tree dump doesn't start with a proper header - Key: MAPREDUCE-5056 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5056 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0 Reporter: Ivan Mitic Assignee: Ivan Mitic Attachments: MAPREDUCE-5056.trunk.2.patch, MAPREDUCE-5056.trunk.3.patch, MAPREDUCE-5056.trunk.patch Test fails on the below assertion: Running org.apache.hadoop.mapreduce.util.TestProcfsBasedProcessTree Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.266 sec FAILURE! testProcessTreeDump(org.apache.hadoop.mapreduce.util.TestProcfsBasedProcessTree) Time elapsed: 0 sec FAILURE! junit.framework.AssertionFailedError: Process-tree dump doesn't start with a proper header at junit.framework.Assert.fail(Assert.java:47) at junit.framework.Assert.assertTrue(Assert.java:20) at org.apache.hadoop.mapreduce.util.TestProcfsBasedProcessTree.testProcessTreeDump(TestProcfsBasedProcessTree.java:564) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:243) at junit.framework.TestSuite.run(TestSuite.java:238) at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5128) mapred-default.xml is missing a bunch of history server configs
[
https://issues.apache.org/jira/browse/MAPREDUCE-5128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622855#comment-13622855
]
Hadoop QA commented on MAPREDUCE-5128:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12577069/MAPREDUCE-5128.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3497//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3497//console
This message is automatically generated.
mapred-default.xml is missing a bunch of history server configs
---
Key: MAPREDUCE-5128
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5128
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: documentation, jobhistoryserver
Affects Versions: 2.0.3-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: MAPREDUCE-5128.patch
mapred-default.xml is missing many configs that work for the job history
server. mapreduce.jobhistory.cleaner.enable, mapreduce.jobhistory.done-dir,
and mapreduce.jobhistory.datestring.cache.size are a few examples.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5079) Recovery should restore task state from job history info directly
[ https://issues.apache.org/jira/browse/MAPREDUCE-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-5079: -- Attachment: MAPREDUCE-5079.patch Updated patch with a fix for the issue Rob found along with a number of new unit tests for the recover transition. Kudos to Rob who developed the new tests. Recovery should restore task state from job history info directly - Key: MAPREDUCE-5079 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5079 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mr-am Affects Versions: 0.23.7 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Critical Attachments: MAPREDUCE-5079.patch, MAPREDUCE-5079.patch We've encountered a lot of hanging issues during MR-AM recovery because the state machines don't always end up in the same states after recovery. This is especially true when speculative execution is enabled. It should be straightforward to restore task and task attempt states directly from the TaskInfo and TaskAttemptInfo records in the job history file to avoid relying on the task state machines ending up in the proper states with the proper number of attempts. This should be a more robust solution that would also give us the option of recovering start time and log locations for tasks that were in-progress when the AM crashed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5079) Recovery should restore task state from job history info directly
[ https://issues.apache.org/jira/browse/MAPREDUCE-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-5079: -- Status: Patch Available (was: Open) Recovery should restore task state from job history info directly - Key: MAPREDUCE-5079 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5079 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mr-am Affects Versions: 0.23.7 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Critical Attachments: MAPREDUCE-5079.patch, MAPREDUCE-5079.patch We've encountered a lot of hanging issues during MR-AM recovery because the state machines don't always end up in the same states after recovery. This is especially true when speculative execution is enabled. It should be straightforward to restore task and task attempt states directly from the TaskInfo and TaskAttemptInfo records in the job history file to avoid relying on the task state machines ending up in the proper states with the proper number of attempts. This should be a more robust solution that would also give us the option of recovering start time and log locations for tasks that were in-progress when the AM crashed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5129) Add tag info to JH files
Billie Rinaldi created MAPREDUCE-5129: - Summary: Add tag info to JH files Key: MAPREDUCE-5129 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5129 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Billie Rinaldi Priority: Minor It will be useful to add tags to the existing workflow info logged by JH. This will allow jobs to be filtered/grouped for analysis more easily. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5129) Add tag info to JH files
[ https://issues.apache.org/jira/browse/MAPREDUCE-5129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billie Rinaldi updated MAPREDUCE-5129: -- Attachment: MAPREDUCE-5129.patch Add tag info to JH files Key: MAPREDUCE-5129 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5129 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Billie Rinaldi Priority: Minor Attachments: MAPREDUCE-5129.patch It will be useful to add tags to the existing workflow info logged by JH. This will allow jobs to be filtered/grouped for analysis more easily. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5094) Disable mem monitoring by default in MiniMRYarnCluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622885#comment-13622885
]
Hadoop QA commented on MAPREDUCE-5094:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12577060/MAPREDUCE-5094.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
org.apache.hadoop.mapred.TestMiniMRClientCluster
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3495//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3495//console
This message is automatically generated.
Disable mem monitoring by default in MiniMRYarnCluster
--
Key: MAPREDUCE-5094
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5094
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.0.3-alpha
Reporter: Siddharth Seth
Assignee: Siddharth Seth
Attachments: MAPREDUCE-5094.txt
YARN-449. Some hbase tests were failing since containers were getting killed.
I believe these checks are disabled by default on the branch-1 MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5079) Recovery should restore task state from job history info directly
[
https://issues.apache.org/jira/browse/MAPREDUCE-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622886#comment-13622886
]
Hadoop QA commented on MAPREDUCE-5079:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12577074/MAPREDUCE-5079.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 eclipse:eclipse{color}. The patch failed to build with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app:
org.apache.hadoop.mapreduce.v2.app.job.impl.TestJobImpl
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3498//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3498//console
This message is automatically generated.
Recovery should restore task state from job history info directly
-
Key: MAPREDUCE-5079
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5079
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mr-am
Affects Versions: 0.23.7
Reporter: Jason Lowe
Assignee: Jason Lowe
Priority: Critical
Attachments: MAPREDUCE-5079.patch, MAPREDUCE-5079.patch
We've encountered a lot of hanging issues during MR-AM recovery because the
state machines don't always end up in the same states after recovery. This
is especially true when speculative execution is enabled. It should be
straightforward to restore task and task attempt states directly from the
TaskInfo and TaskAttemptInfo records in the job history file to avoid relying
on the task state machines ending up in the proper states with the proper
number of attempts.
This should be a more robust solution that would also give us the option of
recovering start time and log locations for tasks that were in-progress when
the AM crashed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5130) Add missing job config options to mapred-default.xml
Sandy Ryza created MAPREDUCE-5130: - Summary: Add missing job config options to mapred-default.xml Key: MAPREDUCE-5130 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5130 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Sandy Ryza Assignee: Sandy Ryza I came across that mapreduce.map.child.java.opts and mapreduce.reduce.child.java.opts were missing in mapred-default.xml. I'll do a fuller sweep to see what else is missing before posting a patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5079) Recovery should restore task state from job history info directly
[ https://issues.apache.org/jira/browse/MAPREDUCE-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-5079: -- Attachment: MAPREDUCE-5079.patch Needed to update to the changes in MAPREDUCE-5086. The eclipse failures are unrelated, as I am unable to reproduce the failure locally and it's failing in areas untouched by this patch. Recovery should restore task state from job history info directly - Key: MAPREDUCE-5079 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5079 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mr-am Affects Versions: 0.23.7 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Critical Attachments: MAPREDUCE-5079.patch, MAPREDUCE-5079.patch, MAPREDUCE-5079.patch We've encountered a lot of hanging issues during MR-AM recovery because the state machines don't always end up in the same states after recovery. This is especially true when speculative execution is enabled. It should be straightforward to restore task and task attempt states directly from the TaskInfo and TaskAttemptInfo records in the job history file to avoid relying on the task state machines ending up in the proper states with the proper number of attempts. This should be a more robust solution that would also give us the option of recovering start time and log locations for tasks that were in-progress when the AM crashed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5130) Add missing config options to mapred-default.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-5130: -- Summary: Add missing config options to mapred-default.xml (was: Add missing job config options to mapred-default.xml) Add missing config options to mapred-default.xml Key: MAPREDUCE-5130 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5130 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Sandy Ryza Assignee: Sandy Ryza I came across that mapreduce.map.child.java.opts and mapreduce.reduce.child.java.opts were missing in mapred-default.xml. I'll do a fuller sweep to see what else is missing before posting a patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5130) Add missing job config options to mapred-default.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-5130: -- Summary: Add missing job config options to mapred-default.xml (was: Add missing config options to mapred-default.xml) Add missing job config options to mapred-default.xml Key: MAPREDUCE-5130 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5130 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Sandy Ryza Assignee: Sandy Ryza I came across that mapreduce.map.child.java.opts and mapreduce.reduce.child.java.opts were missing in mapred-default.xml. I'll do a fuller sweep to see what else is missing before posting a patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5130) Add missing job config options to mapred-default.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-5130: -- Description: I came across that mapreduce.map.child.java.opts and mapreduce.reduce.child.java.opts were missing in mapred-default.xml. I'll do a fuller sweep to see what else is missing before posting a patch. List so far: mapreduce.map/reduce.child.java.opts mapreduce.map/reduce.memory.mb was:I came across that mapreduce.map.child.java.opts and mapreduce.reduce.child.java.opts were missing in mapred-default.xml. I'll do a fuller sweep to see what else is missing before posting a patch. Add missing job config options to mapred-default.xml Key: MAPREDUCE-5130 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5130 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Sandy Ryza Assignee: Sandy Ryza I came across that mapreduce.map.child.java.opts and mapreduce.reduce.child.java.opts were missing in mapred-default.xml. I'll do a fuller sweep to see what else is missing before posting a patch. List so far: mapreduce.map/reduce.child.java.opts mapreduce.map/reduce.memory.mb -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5130) Add missing job config options to mapred-default.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-5130: -- Description: I came across that mapreduce.map.child.java.opts and mapreduce.reduce.child.java.opts were missing in mapred-default.xml. I'll do a fuller sweep to see what else is missing before posting a patch. List so far: mapreduce.map/reduce.child.java.opts mapreduce.map/reduce.memory.mb mapreduce.map/reduce.log.level was: I came across that mapreduce.map.child.java.opts and mapreduce.reduce.child.java.opts were missing in mapred-default.xml. I'll do a fuller sweep to see what else is missing before posting a patch. List so far: mapreduce.map/reduce.child.java.opts mapreduce.map/reduce.memory.mb Add missing job config options to mapred-default.xml Key: MAPREDUCE-5130 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5130 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Sandy Ryza Assignee: Sandy Ryza I came across that mapreduce.map.child.java.opts and mapreduce.reduce.child.java.opts were missing in mapred-default.xml. I'll do a fuller sweep to see what else is missing before posting a patch. List so far: mapreduce.map/reduce.child.java.opts mapreduce.map/reduce.memory.mb mapreduce.map/reduce.log.level -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5130) Add missing job config options to mapred-default.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-5130: -- Description: I came across that mapreduce.map.child.java.opts and mapreduce.reduce.child.java.opts were missing in mapred-default.xml. I'll do a fuller sweep to see what else is missing before posting a patch. List so far: mapreduce.map/reduce.child.java.opts mapreduce.map/reduce.memory.mb mapreduce.map/reduce.log.level mapreduce.job.jvm.numtasks was: I came across that mapreduce.map.child.java.opts and mapreduce.reduce.child.java.opts were missing in mapred-default.xml. I'll do a fuller sweep to see what else is missing before posting a patch. List so far: mapreduce.map/reduce.child.java.opts mapreduce.map/reduce.memory.mb mapreduce.map/reduce.log.level Add missing job config options to mapred-default.xml Key: MAPREDUCE-5130 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5130 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Sandy Ryza Assignee: Sandy Ryza I came across that mapreduce.map.child.java.opts and mapreduce.reduce.child.java.opts were missing in mapred-default.xml. I'll do a fuller sweep to see what else is missing before posting a patch. List so far: mapreduce.map/reduce.child.java.opts mapreduce.map/reduce.memory.mb mapreduce.map/reduce.log.level mapreduce.job.jvm.numtasks -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5056) TestProcfsBasedProcessTree fails on Windows with Process-tree dump doesn't start with a proper header
[ https://issues.apache.org/jira/browse/MAPREDUCE-5056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622940#comment-13622940 ] Bikas Saha commented on MAPREDUCE-5056: --- I did not quite get what the String.format(...%n) is doing and why we need to change it. TestProcfsBasedProcessTree fails on Windows with Process-tree dump doesn't start with a proper header - Key: MAPREDUCE-5056 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5056 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0 Reporter: Ivan Mitic Assignee: Ivan Mitic Attachments: MAPREDUCE-5056.trunk.2.patch, MAPREDUCE-5056.trunk.3.patch, MAPREDUCE-5056.trunk.patch Test fails on the below assertion: Running org.apache.hadoop.mapreduce.util.TestProcfsBasedProcessTree Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.266 sec FAILURE! testProcessTreeDump(org.apache.hadoop.mapreduce.util.TestProcfsBasedProcessTree) Time elapsed: 0 sec FAILURE! junit.framework.AssertionFailedError: Process-tree dump doesn't start with a proper header at junit.framework.Assert.fail(Assert.java:47) at junit.framework.Assert.assertTrue(Assert.java:20) at org.apache.hadoop.mapreduce.util.TestProcfsBasedProcessTree.testProcessTreeDump(TestProcfsBasedProcessTree.java:564) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:243) at junit.framework.TestSuite.run(TestSuite.java:238) at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5079) Recovery should restore task state from job history info directly
[
https://issues.apache.org/jira/browse/MAPREDUCE-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622944#comment-13622944
]
Hadoop QA commented on MAPREDUCE-5079:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12577094/MAPREDUCE-5079.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3500//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3500//console
This message is automatically generated.
Recovery should restore task state from job history info directly
-
Key: MAPREDUCE-5079
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5079
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mr-am
Affects Versions: 0.23.7
Reporter: Jason Lowe
Assignee: Jason Lowe
Priority: Critical
Attachments: MAPREDUCE-5079.patch, MAPREDUCE-5079.patch,
MAPREDUCE-5079.patch
We've encountered a lot of hanging issues during MR-AM recovery because the
state machines don't always end up in the same states after recovery. This
is especially true when speculative execution is enabled. It should be
straightforward to restore task and task attempt states directly from the
TaskInfo and TaskAttemptInfo records in the job history file to avoid relying
on the task state machines ending up in the proper states with the proper
number of attempts.
This should be a more robust solution that would also give us the option of
recovering start time and log locations for tasks that were in-progress when
the AM crashed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4932) mapreduce.job#getTaskCompletionEvents incompatible with MR1
[
https://issues.apache.org/jira/browse/MAPREDUCE-4932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated MAPREDUCE-4932:
-
Attachment: MAPREDUCE-4932.patch
That's a good point; not sure why I didn't see that it was 10 before.
New patch uses {{10}} instead of {{Integer.MAX_VALUE}}
mapreduce.job#getTaskCompletionEvents incompatible with MR1
---
Key: MAPREDUCE-4932
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4932
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.0.2-alpha
Reporter: Robert Kanter
Assignee: Robert Kanter
Attachments: MAPREDUCE-4932.patch, MAPREDUCE-4932.patch,
MAPREDUCE-4932.patch
In MR1, {{org.apache.hadoop.mapreduce.Job#getTaskCompletionEvents}} takes one
argument: {{int startFrom}}. In MR2, it now takes an additional argument:
{{int numEvents}} (which is the max number of events to get). This makes
them incompatible.
I propose we add a second {{getTaskCompletionEvents}} method that simply
calls the other one with {{numEvents}} set to {{Integer.MAX_VALUE}} to
replicate the behavior of the MR1 version.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5056) TestProcfsBasedProcessTree fails on Windows with Process-tree dump doesn't start with a proper header
[ https://issues.apache.org/jira/browse/MAPREDUCE-5056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622973#comment-13622973 ] Chris Nauroth commented on MAPREDUCE-5056: -- That accounts for the line ending difference on Unix vs. Windows. String.format with %n in the format string gives you the system-specific line separator instead of hard-coding to \n. TestProcfsBasedProcessTree fails on Windows with Process-tree dump doesn't start with a proper header - Key: MAPREDUCE-5056 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5056 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0 Reporter: Ivan Mitic Assignee: Ivan Mitic Attachments: MAPREDUCE-5056.trunk.2.patch, MAPREDUCE-5056.trunk.3.patch, MAPREDUCE-5056.trunk.patch Test fails on the below assertion: Running org.apache.hadoop.mapreduce.util.TestProcfsBasedProcessTree Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.266 sec FAILURE! testProcessTreeDump(org.apache.hadoop.mapreduce.util.TestProcfsBasedProcessTree) Time elapsed: 0 sec FAILURE! junit.framework.AssertionFailedError: Process-tree dump doesn't start with a proper header at junit.framework.Assert.fail(Assert.java:47) at junit.framework.Assert.assertTrue(Assert.java:20) at org.apache.hadoop.mapreduce.util.TestProcfsBasedProcessTree.testProcessTreeDump(TestProcfsBasedProcessTree.java:564) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:243) at junit.framework.TestSuite.run(TestSuite.java:238) at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5056) TestProcfsBasedProcessTree fails on Windows with Process-tree dump doesn't start with a proper header
[
https://issues.apache.org/jira/browse/MAPREDUCE-5056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622978#comment-13622978
]
Ivan Mitic commented on MAPREDUCE-5056:
---
Thanks Chris for the comment. Bikas, this also makes the test consistent with
the implementation {{ProcfsBasedProcessTree#getProcessTreeDump}}.
TestProcfsBasedProcessTree fails on Windows with Process-tree dump doesn't
start with a proper header
-
Key: MAPREDUCE-5056
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5056
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Ivan Mitic
Assignee: Ivan Mitic
Attachments: MAPREDUCE-5056.trunk.2.patch,
MAPREDUCE-5056.trunk.3.patch, MAPREDUCE-5056.trunk.patch
Test fails on the below assertion:
Running org.apache.hadoop.mapreduce.util.TestProcfsBasedProcessTree
Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.266 sec
FAILURE!
testProcessTreeDump(org.apache.hadoop.mapreduce.util.TestProcfsBasedProcessTree)
Time elapsed: 0 sec FAILURE!
junit.framework.AssertionFailedError: Process-tree dump doesn't start with a
proper header
at junit.framework.Assert.fail(Assert.java:47)
at junit.framework.Assert.assertTrue(Assert.java:20)
at
org.apache.hadoop.mapreduce.util.TestProcfsBasedProcessTree.testProcessTreeDump(TestProcfsBasedProcessTree.java:564)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at junit.framework.TestCase.runTest(TestCase.java:168)
at junit.framework.TestCase.runBare(TestCase.java:134)
at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
at junit.framework.TestSuite.runTest(TestSuite.java:243)
at junit.framework.TestSuite.run(TestSuite.java:238)
at
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5028) Maps fail when io.sort.mb is set to high value
[
https://issues.apache.org/jira/browse/MAPREDUCE-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated MAPREDUCE-5028:
Attachment: repro-mr-5028.patch
Arun, uploaded a patch (repro-mr-5028.patch) with unit tests that reproduce the
problem. Unfortunately, the methods causing the interger overflow do not have
integer arithmetic stubbed out and so the tests actually end up creating large
buffers.
I don't think we should include them, but I am sure they will come handy to
make sure the posted patch actually fixes the issue. The tests cover 3 of the 4
changes - the 4th is embedded too deep inside to write a unit test for.
If we want to include the tests, we might have to stub out the array index
calculations to other methods which doesn't look very useful either.
Maps fail when io.sort.mb is set to high value
--
Key: MAPREDUCE-5028
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5028
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 1.1.1, 2.0.3-alpha, 0.23.5
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Priority: Critical
Fix For: 1.2.0, 2.0.5-beta
Attachments: mr-5028-branch1.patch, mr-5028-branch1.patch,
mr-5028-branch1.patch, mr-5028-trunk.patch, mr-5028-trunk.patch,
mr-5028-trunk.patch, repro-mr-5028.patch
Verified the problem exists on branch-1 with the following configuration:
Pseudo-dist mode: 2 maps/ 1 reduce, mapred.child.java.opts=-Xmx2048m,
io.sort.mb=1280, dfs.block.size=2147483648
Run teragen to generate 4 GB data
Maps fail when you run wordcount on this configuration with the following
error:
{noformat}
java.io.IOException: Spill failed
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1031)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:692)
at
org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at
org.apache.hadoop.examples.WordCount$TokenizerMapper.map(WordCount.java:45)
at
org.apache.hadoop.examples.WordCount$TokenizerMapper.map(WordCount.java:34)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:766)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at org.apache.hadoop.io.IntWritable.readFields(IntWritable.java:38)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
at
org.apache.hadoop.mapreduce.ReduceContext.nextKeyValue(ReduceContext.java:116)
at
org.apache.hadoop.mapreduce.ReduceContext.nextKey(ReduceContext.java:92)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:175)
at
org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1505)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1438)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:855)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1346)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5028) Maps fail when io.sort.mb is set to high value
[
https://issues.apache.org/jira/browse/MAPREDUCE-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated MAPREDUCE-5028:
Attachment: (was: org.apache.hadoop.mapreduce.v2.TestMRJobs-output.txt)
Maps fail when io.sort.mb is set to high value
--
Key: MAPREDUCE-5028
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5028
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 1.1.1, 2.0.3-alpha, 0.23.5
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Priority: Critical
Fix For: 1.2.0, 2.0.5-beta
Attachments: mr-5028-branch1.patch, mr-5028-branch1.patch,
mr-5028-branch1.patch, mr-5028-trunk.patch, mr-5028-trunk.patch,
mr-5028-trunk.patch, repro-mr-5028.patch
Verified the problem exists on branch-1 with the following configuration:
Pseudo-dist mode: 2 maps/ 1 reduce, mapred.child.java.opts=-Xmx2048m,
io.sort.mb=1280, dfs.block.size=2147483648
Run teragen to generate 4 GB data
Maps fail when you run wordcount on this configuration with the following
error:
{noformat}
java.io.IOException: Spill failed
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1031)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:692)
at
org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at
org.apache.hadoop.examples.WordCount$TokenizerMapper.map(WordCount.java:45)
at
org.apache.hadoop.examples.WordCount$TokenizerMapper.map(WordCount.java:34)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:766)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at org.apache.hadoop.io.IntWritable.readFields(IntWritable.java:38)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
at
org.apache.hadoop.mapreduce.ReduceContext.nextKeyValue(ReduceContext.java:116)
at
org.apache.hadoop.mapreduce.ReduceContext.nextKey(ReduceContext.java:92)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:175)
at
org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1505)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1438)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:855)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1346)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4932) mapreduce.job#getTaskCompletionEvents incompatible with MR1
[
https://issues.apache.org/jira/browse/MAPREDUCE-4932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622989#comment-13622989
]
Hadoop QA commented on MAPREDUCE-4932:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12577108/MAPREDUCE-4932.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 eclipse:eclipse{color}. The patch failed to build with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3501//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3501//console
This message is automatically generated.
mapreduce.job#getTaskCompletionEvents incompatible with MR1
---
Key: MAPREDUCE-4932
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4932
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.0.2-alpha
Reporter: Robert Kanter
Assignee: Robert Kanter
Attachments: MAPREDUCE-4932.patch, MAPREDUCE-4932.patch,
MAPREDUCE-4932.patch
In MR1, {{org.apache.hadoop.mapreduce.Job#getTaskCompletionEvents}} takes one
argument: {{int startFrom}}. In MR2, it now takes an additional argument:
{{int numEvents}} (which is the max number of events to get). This makes
them incompatible.
I propose we add a second {{getTaskCompletionEvents}} method that simply
calls the other one with {{numEvents}} set to {{Integer.MAX_VALUE}} to
replicate the behavior of the MR1 version.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5094) Disable mem monitoring by default in MiniMRYarnCluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622991#comment-13622991
]
Hadoop QA commented on MAPREDUCE-5094:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12577060/MAPREDUCE-5094.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3499//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3499//console
This message is automatically generated.
Disable mem monitoring by default in MiniMRYarnCluster
--
Key: MAPREDUCE-5094
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5094
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.0.3-alpha
Reporter: Siddharth Seth
Assignee: Siddharth Seth
Attachments: MAPREDUCE-5094.txt
YARN-449. Some hbase tests were failing since containers were getting killed.
I believe these checks are disabled by default on the branch-1 MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5094) Disable mem monitoring by default in MiniMRYarnCluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5094: --- Status: Open (was: Patch Available) bq. I believe these checks are disabled by default on the branch-1 MiniMRCluster. That is very right. Let's also disable pmem, no point in doing it for tests. If we want to validate the monitoring, specific tests should do that. Rename mapreduce.minicluster.control-resource-monitoring to mapreduce.minicluster.enable-memory-monitoring? Disable mem monitoring by default in MiniMRYarnCluster -- Key: MAPREDUCE-5094 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5094 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Siddharth Seth Assignee: Siddharth Seth Attachments: MAPREDUCE-5094.txt YARN-449. Some hbase tests were failing since containers were getting killed. I believe these checks are disabled by default on the branch-1 MiniMRCluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5094) Disable mem monitoring by default in MiniMRYarnCluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-5094: -- Attachment: MAPREDUCE-5094.txt Thanks for the quick review. bq. Let's also disable pmem, no point in doing it for tests. If we want to validate the monitoring, specific tests should do that. Done. I don't have a strong preference on enabling / disabling this by default for the miniMRCluster. bq. Rename mapreduce.minicluster.control-resource-monitoring to mapreduce.minicluster.enable-memory-monitoring? I prefer the old name - since that applies to resoueces in general, and lets users control what they want (instead of an enable all). Disable mem monitoring by default in MiniMRYarnCluster -- Key: MAPREDUCE-5094 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5094 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Siddharth Seth Assignee: Siddharth Seth Attachments: MAPREDUCE-5094.txt, MAPREDUCE-5094.txt YARN-449. Some hbase tests were failing since containers were getting killed. I believe these checks are disabled by default on the branch-1 MiniMRCluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5094) Disable mem monitoring by default in MiniMRYarnCluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-5094: -- Status: Patch Available (was: Open) Disable mem monitoring by default in MiniMRYarnCluster -- Key: MAPREDUCE-5094 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5094 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Siddharth Seth Assignee: Siddharth Seth Attachments: MAPREDUCE-5094.txt, MAPREDUCE-5094.txt YARN-449. Some hbase tests were failing since containers were getting killed. I believe these checks are disabled by default on the branch-1 MiniMRCluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-5036) Default shuffle handler port should not be 8080
[ https://issues.apache.org/jira/browse/MAPREDUCE-5036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza reassigned MAPREDUCE-5036: - Assignee: Sandy Ryza Default shuffle handler port should not be 8080 --- Key: MAPREDUCE-5036 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5036 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza The shuffle handler port defaults to 8080. This is a pretty common port for web services, and is likely to cause unnecessary port conflicts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5036) Default shuffle handler port should not be 8080
[ https://issues.apache.org/jira/browse/MAPREDUCE-5036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-5036: -- Description: The shuffle handler port (mapreduce.shuffle.port) defaults to 8080. This is a pretty common port for web services, and is likely to cause unnecessary port conflicts. (was: The shuffle handler port defaults to 8080. This is a pretty common port for web services, and is likely to cause unnecessary port conflicts.) Default shuffle handler port should not be 8080 --- Key: MAPREDUCE-5036 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5036 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza The shuffle handler port (mapreduce.shuffle.port) defaults to 8080. This is a pretty common port for web services, and is likely to cause unnecessary port conflicts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5094) Disable mem monitoring by default in MiniMRYarnCluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13623233#comment-13623233
]
Hadoop QA commented on MAPREDUCE-5094:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12577118/MAPREDUCE-5094.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3502//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3502//console
This message is automatically generated.
Disable mem monitoring by default in MiniMRYarnCluster
--
Key: MAPREDUCE-5094
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5094
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.0.3-alpha
Reporter: Siddharth Seth
Assignee: Siddharth Seth
Attachments: MAPREDUCE-5094.txt, MAPREDUCE-5094.txt
YARN-449. Some hbase tests were failing since containers were getting killed.
I believe these checks are disabled by default on the branch-1 MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5056) TestProcfsBasedProcessTree fails on Windows with Process-tree dump doesn't start with a proper header
[ https://issues.apache.org/jira/browse/MAPREDUCE-5056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13623280#comment-13623280 ] Bikas Saha commented on MAPREDUCE-5056: --- Thanks. Looks good to me. +1. TestProcfsBasedProcessTree fails on Windows with Process-tree dump doesn't start with a proper header - Key: MAPREDUCE-5056 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5056 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0 Reporter: Ivan Mitic Assignee: Ivan Mitic Attachments: MAPREDUCE-5056.trunk.2.patch, MAPREDUCE-5056.trunk.3.patch, MAPREDUCE-5056.trunk.patch Test fails on the below assertion: Running org.apache.hadoop.mapreduce.util.TestProcfsBasedProcessTree Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.266 sec FAILURE! testProcessTreeDump(org.apache.hadoop.mapreduce.util.TestProcfsBasedProcessTree) Time elapsed: 0 sec FAILURE! junit.framework.AssertionFailedError: Process-tree dump doesn't start with a proper header at junit.framework.Assert.fail(Assert.java:47) at junit.framework.Assert.assertTrue(Assert.java:20) at org.apache.hadoop.mapreduce.util.TestProcfsBasedProcessTree.testProcessTreeDump(TestProcfsBasedProcessTree.java:564) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:243) at junit.framework.TestSuite.run(TestSuite.java:238) at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4885) streaming tests have multiple failures on Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13623281#comment-13623281 ] Bikas Saha commented on MAPREDUCE-4885: --- Its great that we dont have other Unix dependencies. +1. streaming tests have multiple failures on Windows - Key: MAPREDUCE-4885 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4885 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/streaming, test Affects Versions: 3.0.0 Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: MAPREDUCE-4885.1.patch, MAPREDUCE-4885.2.patch, MAPREDUCE-4885.3.patch There are multiple test failures due to Queue configuration missing child queue names for root. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4824) Provide a mechanism for jobs to indicate they should not be recovered on restart
[ https://issues.apache.org/jira/browse/MAPREDUCE-4824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13623375#comment-13623375 ] Tom White commented on MAPREDUCE-4824: -- Arun, thanks for updating the patch. I selected the property name to be similar to the existing mapred.jobtracker.restart.recover, but your point about using the mapreduce prefix is a good one (especially if we support this in MR2), so how about mapreduce.jobtracker.restart.recover? The rest looks good. Provide a mechanism for jobs to indicate they should not be recovered on restart Key: MAPREDUCE-4824 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4824 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mrv1 Affects Versions: 1.1.0 Reporter: Tom White Assignee: Tom White Attachments: MAPREDUCE-4824.patch, MAPREDUCE-4824.patch, MAPREDUCE-4824.patch, MAPREDUCE-4824.patch, MAPREDUCE-4824.patch, MAPREDUCE-4824.patch, MAPREDUCE-4824.patch Some jobs (like Sqoop or HBase jobs) are not idempotent, so should not be recovered on jobtracker restart. MAPREDUCE-2702 solves this problem for MR2, however the approach there is not applicable for MR1, since even if we only use the job-level part of the patch and add a isRecoverySupported method to OutputCommitter, there is no way to use that information from the JT (which initiates recovery), since the JT does not instantiate OutputCommitters - and it shouldn't since they are user-level code. (In MR2 it's OK since the MR AM calls the method.) Instead, we can add a MR configuration property to say that a job is not recoverable, and the JT could safely read this from the job conf. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5130) Add missing job config options to mapred-default.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13623376#comment-13623376 ] Harsh J commented on MAPREDUCE-5130: bq. mapreduce.map/reduce.child.java.opts These are on purpose. If we added defaults, they would override the mapred.child.java.opts being set by clients today and break compatibility. Is the list for 1.x or trunk? Cause I see some of these other options on trunk's mapred-default.xml? Add missing job config options to mapred-default.xml Key: MAPREDUCE-5130 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5130 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Sandy Ryza Assignee: Sandy Ryza I came across that mapreduce.map.child.java.opts and mapreduce.reduce.child.java.opts were missing in mapred-default.xml. I'll do a fuller sweep to see what else is missing before posting a patch. List so far: mapreduce.map/reduce.child.java.opts mapreduce.map/reduce.memory.mb mapreduce.map/reduce.log.level mapreduce.job.jvm.numtasks -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5108) Review support for Binary Compatibility for MR applications via YARN
[ https://issues.apache.org/jira/browse/MAPREDUCE-5108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13623379#comment-13623379 ] Tom White commented on MAPREDUCE-5108: -- Ah I'd forgotten about MAPREDUCE-3697. Glad to hear it's working. Review support for Binary Compatibility for MR applications via YARN Key: MAPREDUCE-5108 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5108 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.0.3-alpha Reporter: Arun C Murthy Assignee: Zhijie Shen Priority: Blocker Fix For: 2.0.5-beta Attachments: Binary Backward Compatibility.pdf, mr1_mr2_api_diff.tar.gz As we get ready to ship out a beta/stable version of hadoop-2, it makes sense to spend time reviewing support for existing MR applications (hadoop-1) to migrate seamlessly. We've done various pieces of work over time, let's track progress and document things clearly. [~zjshen] has done a bunch of testing and results look very promising so far. The aim is to support applications using org.apache.hadoop.mapred.* api in a binary compatible manner in hadoop-2 - thus, users can just take existing MR applications jars, point them at YARN clusters and things just work. Clearly, we might have some corner cases (haven't seen many so far), including semantics (not just apis); however the intent is to, at least, document them throughly if not actually fix them as feasible. Also, it's clear that we will *not* be able to support org.apache.hadoop.mapreduce api in a *binary compatible* manner due to the interface changes we made in hadoop-0.21 (sigh), and hence, users using the _new_ apis will have to re-compile (i.e. source compatible only). Net, given that vast majority of users use the org.apache.hadoop.mapred api, it's a very reasonable way to ease migration to hadoop-2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
