[jira] [Commented] (MAPREDUCE-2716) MR279: MRReliabilityTest job fails because of missing job-file.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096590#comment-13096590 ] Hadoop QA commented on MAPREDUCE-2716: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12492874/MAPREDUCE-2716-20110903.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 8 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/595//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/595//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/595//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/595//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/595//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/595//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-jobclient.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/595//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/595//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/595//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/595//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/595//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/595//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-api.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/595//console This message is automatically generated. MR279: MRReliabilityTest job fails because of missing job-file. --- Key: MAPREDUCE-2716 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Jeffrey Naisbitt Assignee: Jeffrey Naisbitt Fix For: 0.23.0 Attachments: MAPREDUCE-2716-20110903.txt, MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716-v5.patch, MAPREDUCE-2716-v6.patch, MAPREDUCE-2716-v7.patch, MAPREDUCE-2716-v8.patch, MAPREDUCE-2716.patch The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-USER/mapred/staging/USER/.staging/job_201107121640_0001/job.xml) Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to in TypeConverter.java): e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88) at org.apache.hadoop.fs.Path.init(Path.java:96) at org.apache.hadoop.mapred.JobConf.init(JobConf.java:445) at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104) at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218) at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757) at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741) at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219) at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133
[jira] [Commented] (MAPREDUCE-279) Map-Reduce 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096591#comment-13096591 ] Binglin Chang commented on MAPREDUCE-279: - bq. Ultimately a version of these should be produced natively in some StateMachine method (toDot()?), and I think Chris Douglas may take that up eventually. However, some of the desirable info (e.g., which states send events to or receive them from other state machines) can't really be discovered automatically, so there will continue to be a place for hand-rolled graphs. What's the current progress of this work? I find visualization of state machine really help when reading learning MRv2 code, both YARN MRv2. I add some code in yarn-common to generate graphviz dot file automatically when I try to learn YARN code yesterday, it works fine for me, maybe it is useful for others too. Map-Reduce 2.0 -- Key: MAPREDUCE-279 URL: https://issues.apache.org/jira/browse/MAPREDUCE-279 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: Arun C Murthy Fix For: 0.23.0 Attachments: MR-279-script-20110817.sh, MR-279-script-final.sh, MR-279-script.sh, MR-279-script.sh, MR-279.patch, MR-279.patch, MR-279.sh, MR-279_MR_files_to_move-20110817.txt, MR-279_MR_files_to_move.txt, MR-279_MR_files_to_move.txt, MapReduce_NextGen_Architecture.pdf, capacity-scheduler-dark-theme.png, hadoop_contributors_meet_07_01_2011.pdf, multi-column-stable-sort-default-theme.png, post-move-patch-20110817.2.txt, post-move-patch-final.txt, post-move.patch, post-move.patch, post-move.patch, yarn-state-machine.job.dot, yarn-state-machine.job.png, yarn-state-machine.task-attempt.dot, yarn-state-machine.task-attempt.png, yarn-state-machine.task.dot, yarn-state-machine.task.png Re-factor MapReduce into a generic resource scheduler and a per-job, user-defined component that manages the application execution. Check it out by following [the instructions|http://goo.gl/rSJJC]. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-279) Map-Reduce 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated MAPREDUCE-279: Attachment: ResourceManager.gv ResourceManager.png State graph for ResourceManager Map-Reduce 2.0 -- Key: MAPREDUCE-279 URL: https://issues.apache.org/jira/browse/MAPREDUCE-279 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: Arun C Murthy Fix For: 0.23.0 Attachments: MR-279-script-20110817.sh, MR-279-script-final.sh, MR-279-script.sh, MR-279-script.sh, MR-279.patch, MR-279.patch, MR-279.sh, MR-279_MR_files_to_move-20110817.txt, MR-279_MR_files_to_move.txt, MR-279_MR_files_to_move.txt, MapReduce_NextGen_Architecture.pdf, ResourceManager.gv, ResourceManager.png, capacity-scheduler-dark-theme.png, hadoop_contributors_meet_07_01_2011.pdf, multi-column-stable-sort-default-theme.png, post-move-patch-20110817.2.txt, post-move-patch-final.txt, post-move.patch, post-move.patch, post-move.patch, yarn-state-machine.job.dot, yarn-state-machine.job.png, yarn-state-machine.task-attempt.dot, yarn-state-machine.task-attempt.png, yarn-state-machine.task.dot, yarn-state-machine.task.png Re-factor MapReduce into a generic resource scheduler and a per-job, user-defined component that manages the application execution. Check it out by following [the instructions|http://goo.gl/rSJJC]. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2716) MR279: MRReliabilityTest job fails because of missing job-file.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096593#comment-13096593 ] Hudson commented on MAPREDUCE-2716: --- Integrated in Hadoop-Common-trunk-Commit #827 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/827/]) MAPREDUCE-2716. MRReliabilityTest job fails because of missing job-file. Contributed by Jeffrey Naisbitt. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1164805 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/records/JobReport.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/records/impl/pb/JobReportPBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/proto/mr_protos.proto * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/NotRunningJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientRedirect.java MR279: MRReliabilityTest job fails because of missing job-file. --- Key: MAPREDUCE-2716 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Jeffrey Naisbitt Assignee: Jeffrey Naisbitt Fix For: 0.23.0 Attachments: MAPREDUCE-2716-20110903.txt, MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716-v5.patch, MAPREDUCE-2716-v6.patch, MAPREDUCE-2716-v7.patch, MAPREDUCE-2716-v8.patch, MAPREDUCE-2716.patch The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-USER/mapred/staging/USER/.staging/job_201107121640_0001/job.xml) Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to in TypeConverter.java): e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88) at org.apache.hadoop.fs.Path.init(Path.java:96) at org.apache.hadoop.mapred.JobConf.init(JobConf.java:445) at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104) at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218) at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757) at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741) at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219) at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133) at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke
[jira] [Commented] (MAPREDUCE-2716) MR279: MRReliabilityTest job fails because of missing job-file.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096594#comment-13096594 ] Hudson commented on MAPREDUCE-2716: --- Integrated in Hadoop-Hdfs-trunk-Commit #904 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/904/]) MAPREDUCE-2716. MRReliabilityTest job fails because of missing job-file. Contributed by Jeffrey Naisbitt. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1164805 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/records/JobReport.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/records/impl/pb/JobReportPBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/proto/mr_protos.proto * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/NotRunningJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientRedirect.java MR279: MRReliabilityTest job fails because of missing job-file. --- Key: MAPREDUCE-2716 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Jeffrey Naisbitt Assignee: Jeffrey Naisbitt Fix For: 0.23.0 Attachments: MAPREDUCE-2716-20110903.txt, MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716-v5.patch, MAPREDUCE-2716-v6.patch, MAPREDUCE-2716-v7.patch, MAPREDUCE-2716-v8.patch, MAPREDUCE-2716.patch The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-USER/mapred/staging/USER/.staging/job_201107121640_0001/job.xml) Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to in TypeConverter.java): e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88) at org.apache.hadoop.fs.Path.init(Path.java:96) at org.apache.hadoop.mapred.JobConf.init(JobConf.java:445) at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104) at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218) at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757) at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741) at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219) at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133) at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke
[jira] [Commented] (MAPREDUCE-2716) MR279: MRReliabilityTest job fails because of missing job-file.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096595#comment-13096595 ] Hudson commented on MAPREDUCE-2716: --- Integrated in Hadoop-Mapreduce-trunk-Commit #837 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/837/]) MAPREDUCE-2716. MRReliabilityTest job fails because of missing job-file. Contributed by Jeffrey Naisbitt. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1164805 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/records/JobReport.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/records/impl/pb/JobReportPBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/proto/mr_protos.proto * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/NotRunningJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientRedirect.java MR279: MRReliabilityTest job fails because of missing job-file. --- Key: MAPREDUCE-2716 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Jeffrey Naisbitt Assignee: Jeffrey Naisbitt Fix For: 0.23.0 Attachments: MAPREDUCE-2716-20110903.txt, MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716-v5.patch, MAPREDUCE-2716-v6.patch, MAPREDUCE-2716-v7.patch, MAPREDUCE-2716-v8.patch, MAPREDUCE-2716.patch The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-USER/mapred/staging/USER/.staging/job_201107121640_0001/job.xml) Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to in TypeConverter.java): e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88) at org.apache.hadoop.fs.Path.init(Path.java:96) at org.apache.hadoop.mapred.JobConf.init(JobConf.java:445) at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104) at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218) at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757) at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741) at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219) at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133) at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method
[jira] [Updated] (MAPREDUCE-2716) MR279: MRReliabilityTest job fails because of missing job-file.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-2716: --- Resolution: Fixed Status: Resolved (was: Patch Available) I just committed this to trunk and branch 0.23. Thanks Jeffrey! MR279: MRReliabilityTest job fails because of missing job-file. --- Key: MAPREDUCE-2716 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Jeffrey Naisbitt Assignee: Jeffrey Naisbitt Fix For: 0.23.0 Attachments: MAPREDUCE-2716-20110903.txt, MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716-v5.patch, MAPREDUCE-2716-v6.patch, MAPREDUCE-2716-v7.patch, MAPREDUCE-2716-v8.patch, MAPREDUCE-2716.patch The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-USER/mapred/staging/USER/.staging/job_201107121640_0001/job.xml) Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to in TypeConverter.java): e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88) at org.apache.hadoop.fs.Path.init(Path.java:96) at org.apache.hadoop.mapred.JobConf.init(JobConf.java:445) at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104) at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218) at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757) at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741) at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219) at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133) at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144) at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111) at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:192) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-279) Map-Reduce 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated MAPREDUCE-279: Attachment: NodeManager.png NodeManager.gv Map-Reduce 2.0 -- Key: MAPREDUCE-279 URL: https://issues.apache.org/jira/browse/MAPREDUCE-279 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: Arun C Murthy Fix For: 0.23.0 Attachments: MR-279-script-20110817.sh, MR-279-script-final.sh, MR-279-script.sh, MR-279-script.sh, MR-279.patch, MR-279.patch, MR-279.sh, MR-279_MR_files_to_move-20110817.txt, MR-279_MR_files_to_move.txt, MR-279_MR_files_to_move.txt, MapReduce_NextGen_Architecture.pdf, NodeManager.gv, NodeManager.png, ResourceManager.gv, ResourceManager.png, capacity-scheduler-dark-theme.png, hadoop_contributors_meet_07_01_2011.pdf, multi-column-stable-sort-default-theme.png, post-move-patch-20110817.2.txt, post-move-patch-final.txt, post-move.patch, post-move.patch, post-move.patch, yarn-state-machine.job.dot, yarn-state-machine.job.png, yarn-state-machine.task-attempt.dot, yarn-state-machine.task-attempt.png, yarn-state-machine.task.dot, yarn-state-machine.task.png Re-factor MapReduce into a generic resource scheduler and a per-job, user-defined component that manages the application execution. Check it out by following [the instructions|http://goo.gl/rSJJC]. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-1347) Missing synchronization in MultipleOutputFormat
[ https://issues.apache.org/jira/browse/MAPREDUCE-1347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-1347: --- Attachment: MAPREDUCE-1347.r8.diff Updating patch for trunk. Hoping this goes in since its already been reviewed. Gets difficult updating patches since the trunk layouts are a storm. The r7 patch can apply to 0.22 I believe, if it is required. r8 is basically r7 adapted to trunk. Missing synchronization in MultipleOutputFormat --- Key: MAPREDUCE-1347 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1347 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.2, 0.21.0, 0.22.0 Reporter: Todd Lipcon Assignee: Harsh J Fix For: 0.23.0 Attachments: MAPREDUCE-1347.r2.diff, MAPREDUCE-1347.r3.diff, MAPREDUCE-1347.r4.diff, MAPREDUCE-1347.r5.diff, MAPREDUCE-1347.r6.diff, MAPREDUCE-1347.r7.diff, MAPREDUCE-1347.r8.diff, mapreduce.1347.r1.diff MultipleOutputFormat's RecordWriter implementation doesn't use synchronization when accessing the recordWriters member. When using multithreaded mappers or reducers, this can result in problems where two threads will both try to create the same file, causing AlreadyBeingCreatedException. Doing this more fine-grained than just synchronizing the whole method is probably a good idea, so that multithreaded mappers can actually achieve parallelism writing into separate output streams. From what I can tell, the new API's MultipleOutputs seems not to have this issue. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2384) Can MR make error response Immediately?
[ https://issues.apache.org/jira/browse/MAPREDUCE-2384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-2384: --- Attachment: MAPREDUCE-2384.r3.diff Previous patch dupe, updated for 0.23/trunk. Can MR make error response Immediately? --- Key: MAPREDUCE-2384 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2384 Project: Hadoop Map/Reduce Issue Type: Improvement Components: job submission Affects Versions: 0.21.0 Reporter: Denny Ye Assignee: Harsh J Fix For: 0.23.0 Attachments: MAPREDUCE-2384.r1.diff, MAPREDUCE-2384.r2.diff, MAPREDUCE-2384.r3.diff When I read the source code of MapReduce in Hadoop 0.21.0, sometimes it made me confused about error response. For example: 1. JobSubmitter checking output for each job. MapReduce makes rule to limit that each job output must be not exist to avoid fault overwrite. In my opinion, MR should verify output at the point of client submitting. Actually, it copies related files to specified target and then, doing the verifying. 2. JobTracker. Job has been submitted to JobTracker. In first step, JT create JIT object that is very huge . Next step, JT start to verify job queue authority and memory requirements. In normal case, verifying client input then response immediately if any cases in fault. Regular logic can be performed if all the inputs have passed. It seems like that those code does not make sense for understanding. Is only my personal opinion? Wish someone help me to explain the details. Thanks! -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-1347) Missing synchronization in MultipleOutputFormat
[ https://issues.apache.org/jira/browse/MAPREDUCE-1347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096620#comment-13096620 ] Hadoop QA commented on MAPREDUCE-1347: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12492886/MAPREDUCE-1347.r8.diff against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 8 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/596//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/596//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/596//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-api.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/596//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/596//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/596//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/596//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/596//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/596//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/596//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/596//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/596//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-jobclient.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/596//console This message is automatically generated. Missing synchronization in MultipleOutputFormat --- Key: MAPREDUCE-1347 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1347 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.2, 0.21.0, 0.22.0 Reporter: Todd Lipcon Assignee: Harsh J Fix For: 0.23.0 Attachments: MAPREDUCE-1347.r2.diff, MAPREDUCE-1347.r3.diff, MAPREDUCE-1347.r4.diff, MAPREDUCE-1347.r5.diff, MAPREDUCE-1347.r6.diff, MAPREDUCE-1347.r7.diff, MAPREDUCE-1347.r8.diff, mapreduce.1347.r1.diff MultipleOutputFormat's RecordWriter implementation doesn't use synchronization when accessing the recordWriters member. When using multithreaded mappers or reducers, this can result in problems where two threads will both try to create the same file, causing AlreadyBeingCreatedException. Doing this more fine-grained than just synchronizing the whole method is probably a good idea, so that multithreaded mappers can actually achieve parallelism writing into separate output streams. From what I can tell, the new API's MultipleOutputs seems not to have this issue. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2384) Can MR make error response Immediately?
[ https://issues.apache.org/jira/browse/MAPREDUCE-2384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096623#comment-13096623 ] Hadoop QA commented on MAPREDUCE-2384: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12492888/MAPREDUCE-2384.r3.diff against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 8 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/597//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/597//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/597//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/597//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/597//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/597//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-jobclient.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/597//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/597//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/597//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/597//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/597//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/597//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-api.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/597//console This message is automatically generated. Can MR make error response Immediately? --- Key: MAPREDUCE-2384 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2384 Project: Hadoop Map/Reduce Issue Type: Improvement Components: job submission Affects Versions: 0.21.0 Reporter: Denny Ye Assignee: Harsh J Fix For: 0.23.0 Attachments: MAPREDUCE-2384.r1.diff, MAPREDUCE-2384.r2.diff, MAPREDUCE-2384.r3.diff When I read the source code of MapReduce in Hadoop 0.21.0, sometimes it made me confused about error response. For example: 1. JobSubmitter checking output for each job. MapReduce makes rule to limit that each job output must be not exist to avoid fault overwrite. In my opinion, MR should verify output at the point of client submitting. Actually, it copies related files to specified target and then, doing the verifying. 2. JobTracker. Job has been submitted to JobTracker. In first step, JT create JIT object that is very huge . Next step, JT start to verify job queue authority and memory requirements. In normal case, verifying client input then response immediately if any cases in fault. Regular logic can be performed if all the inputs have passed. It seems like that those code does not make sense for understanding. Is only my personal opinion? Wish someone help me to explain the details. Thanks! -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2749) [MR-279] NM registers with RM even before it starts various servers
[
https://issues.apache.org/jira/browse/MAPREDUCE-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096628#comment-13096628
]
Vinod Kumar Vavilapalli commented on MAPREDUCE-2749:
Thanks for the patch Thomas!
One minor comment I have with this patch is the setting of keys via the event.
While this works, my concern is that if we do it this way, by the time
{{NodeManager.start()}} returns, the server isn't still guaranteed to be ready
for service. I think we can do this another way. If we move
{{ContainerTokenSecretManager}} to the top level {{NodeManager}} composite, we
can set the keys as the last step in {{NodeManager.start()}}. Thoughts?
Also, we can add a test for this - mock {{rpc.getServer()}} to throw an
exception and verify that ResourceManager doesn't have any registered NM.
[MR-279] NM registers with RM even before it starts various servers
---
Key: MAPREDUCE-2749
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2749
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Vinod Kumar Vavilapalli
Assignee: Thomas Graves
Fix For: 0.23.0
Attachments: MAPREDUCE-2749-v2.patch, MAPREDUCE-2749.patch
In case NM eventually fails to start the ContainerManager server because of
say a port clash, RM will have to wait for expiry to detect the NM crash.
It is desirable to make NM register with RM only after it can start all of
its components successfully.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2887) MR changes to match HADOOP-7524 (multiple RPC protocols)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096648#comment-13096648 ] Hudson commented on MAPREDUCE-2887: --- Integrated in Hadoop-Hdfs-trunk #781 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/781/]) HADOOP-7524 and MapReduce-2887 Change RPC to allow multiple protocols including multuple versions of the same protocol (sanjay Radia) MR changes to match HADOOP-7524 (multiple RPC protocols) Key: MAPREDUCE-2887 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2887 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Sanjay Radia Assignee: Sanjay Radia Fix For: 0.23.0, 0.24.0 Attachments: rpc6ForMR.patch, rpc7ForMR.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2716) MR279: MRReliabilityTest job fails because of missing job-file.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096645#comment-13096645 ] Hudson commented on MAPREDUCE-2716: --- Integrated in Hadoop-Hdfs-trunk #781 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/781/]) MAPREDUCE-2716. MRReliabilityTest job fails because of missing job-file. Contributed by Jeffrey Naisbitt. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1164805 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/records/JobReport.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/records/impl/pb/JobReportPBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/proto/mr_protos.proto * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/NotRunningJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientRedirect.java MR279: MRReliabilityTest job fails because of missing job-file. --- Key: MAPREDUCE-2716 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Jeffrey Naisbitt Assignee: Jeffrey Naisbitt Fix For: 0.23.0 Attachments: MAPREDUCE-2716-20110903.txt, MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716-v5.patch, MAPREDUCE-2716-v6.patch, MAPREDUCE-2716-v7.patch, MAPREDUCE-2716-v8.patch, MAPREDUCE-2716.patch The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-USER/mapred/staging/USER/.staging/job_201107121640_0001/job.xml) Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to in TypeConverter.java): e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88) at org.apache.hadoop.fs.Path.init(Path.java:96) at org.apache.hadoop.mapred.JobConf.init(JobConf.java:445) at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104) at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218) at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757) at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741) at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219) at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133) at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke
[jira] [Commented] (MAPREDUCE-2887) MR changes to match HADOOP-7524 (multiple RPC protocols)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096654#comment-13096654 ] Hudson commented on MAPREDUCE-2887: --- Integrated in Hadoop-Mapreduce-trunk #805 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/805/]) HADOOP-7524 and MapReduce-2887 Change RPC to allow multiple protocols including multuple versions of the same protocol (sanjay Radia) MR changes to match HADOOP-7524 (multiple RPC protocols) Key: MAPREDUCE-2887 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2887 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Sanjay Radia Assignee: Sanjay Radia Fix For: 0.23.0, 0.24.0 Attachments: rpc6ForMR.patch, rpc7ForMR.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2716) MR279: MRReliabilityTest job fails because of missing job-file.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096651#comment-13096651 ] Hudson commented on MAPREDUCE-2716: --- Integrated in Hadoop-Mapreduce-trunk #805 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/805/]) MAPREDUCE-2716. MRReliabilityTest job fails because of missing job-file. Contributed by Jeffrey Naisbitt. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1164805 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/records/JobReport.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/records/impl/pb/JobReportPBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/proto/mr_protos.proto * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/NotRunningJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientRedirect.java MR279: MRReliabilityTest job fails because of missing job-file. --- Key: MAPREDUCE-2716 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Jeffrey Naisbitt Assignee: Jeffrey Naisbitt Fix For: 0.23.0 Attachments: MAPREDUCE-2716-20110903.txt, MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716-v5.patch, MAPREDUCE-2716-v6.patch, MAPREDUCE-2716-v7.patch, MAPREDUCE-2716-v8.patch, MAPREDUCE-2716.patch The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-USER/mapred/staging/USER/.staging/job_201107121640_0001/job.xml) Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to in TypeConverter.java): e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88) at org.apache.hadoop.fs.Path.init(Path.java:96) at org.apache.hadoop.mapred.JobConf.init(JobConf.java:445) at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104) at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218) at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757) at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741) at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219) at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133) at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke
[jira] [Updated] (MAPREDUCE-2293) Enhance MultipleOutputs to allow additional characters in the named output name
[ https://issues.apache.org/jira/browse/MAPREDUCE-2293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-2293: --- Attachment: mapreduce.mo.removecheck.r3.diff Alejandro, Incorporated your requested conditions in the old API MO. New API has no concept of single/multi so the restriction is not required there. Updated patch for trunk. Enhance MultipleOutputs to allow additional characters in the named output name --- Key: MAPREDUCE-2293 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2293 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.21.0 Reporter: David Rosenstrauch Assignee: Harsh J Priority: Minor Fix For: 0.23.0 Attachments: mapreduce.mo.removecheck.r1.diff, mapreduce.mo.removecheck.r2.diff, mapreduce.mo.removecheck.r3.diff Currently you are only allowed to use alpha-numeric characters in a named output name in the MultipleOutputs class. This is a bit of an onerous restriction, as it would be extremely convenient to be able to use non alpha-numerics in the name too. (E.g., a '.' character would be very helpful, so that you can use the named output name for holding a file name/extension. Perhaps '-' and a '_' characters as well.) The restriction seems to be somewhat arbitrary - it appears to be only enforced in the checkTokenName method. (Though I don't know if there's any downstream impact by loosening this restriction.) Would be extremely helpful/useful to have this fixed though! -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2293) Enhance MultipleOutputs to allow additional characters in the named output name
[ https://issues.apache.org/jira/browse/MAPREDUCE-2293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-2293: --- Status: Open (was: Patch Available) Enhance MultipleOutputs to allow additional characters in the named output name --- Key: MAPREDUCE-2293 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2293 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.21.0 Reporter: David Rosenstrauch Assignee: Harsh J Priority: Minor Fix For: 0.23.0 Attachments: mapreduce.mo.removecheck.r1.diff, mapreduce.mo.removecheck.r2.diff, mapreduce.mo.removecheck.r3.diff Currently you are only allowed to use alpha-numeric characters in a named output name in the MultipleOutputs class. This is a bit of an onerous restriction, as it would be extremely convenient to be able to use non alpha-numerics in the name too. (E.g., a '.' character would be very helpful, so that you can use the named output name for holding a file name/extension. Perhaps '-' and a '_' characters as well.) The restriction seems to be somewhat arbitrary - it appears to be only enforced in the checkTokenName method. (Though I don't know if there's any downstream impact by loosening this restriction.) Would be extremely helpful/useful to have this fixed though! -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2293) Enhance MultipleOutputs to allow additional characters in the named output name
[ https://issues.apache.org/jira/browse/MAPREDUCE-2293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-2293: --- Status: Patch Available (was: Open) Enhance MultipleOutputs to allow additional characters in the named output name --- Key: MAPREDUCE-2293 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2293 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.21.0 Reporter: David Rosenstrauch Assignee: Harsh J Priority: Minor Fix For: 0.23.0 Attachments: mapreduce.mo.removecheck.r1.diff, mapreduce.mo.removecheck.r2.diff, mapreduce.mo.removecheck.r3.diff Currently you are only allowed to use alpha-numeric characters in a named output name in the MultipleOutputs class. This is a bit of an onerous restriction, as it would be extremely convenient to be able to use non alpha-numerics in the name too. (E.g., a '.' character would be very helpful, so that you can use the named output name for holding a file name/extension. Perhaps '-' and a '_' characters as well.) The restriction seems to be somewhat arbitrary - it appears to be only enforced in the checkTokenName method. (Though I don't know if there's any downstream impact by loosening this restriction.) Would be extremely helpful/useful to have this fixed though! -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2293) Enhance MultipleOutputs to allow additional characters in the named output name
[ https://issues.apache.org/jira/browse/MAPREDUCE-2293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1309#comment-1309 ] Hadoop QA commented on MAPREDUCE-2293: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12492900/mapreduce.mo.removecheck.r3.diff against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 8 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/598//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/598//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/598//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/598//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/598//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/598//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-jobclient.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/598//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/598//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/598//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/598//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/598//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/598//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-api.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/598//console This message is automatically generated. Enhance MultipleOutputs to allow additional characters in the named output name --- Key: MAPREDUCE-2293 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2293 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.21.0 Reporter: David Rosenstrauch Assignee: Harsh J Priority: Minor Fix For: 0.23.0 Attachments: mapreduce.mo.removecheck.r1.diff, mapreduce.mo.removecheck.r2.diff, mapreduce.mo.removecheck.r3.diff Currently you are only allowed to use alpha-numeric characters in a named output name in the MultipleOutputs class. This is a bit of an onerous restriction, as it would be extremely convenient to be able to use non alpha-numerics in the name too. (E.g., a '.' character would be very helpful, so that you can use the named output name for holding a file name/extension. Perhaps '-' and a '_' characters as well.) The restriction seems to be somewhat arbitrary - it appears to be only enforced in the checkTokenName method. (Though I don't know if there's any downstream impact by loosening this restriction.) Would be extremely helpful/useful to have this fixed though! -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2068) TaskInProgress contains duplicate idWithinJob() / getIdWithinJob() accessors
[ https://issues.apache.org/jira/browse/MAPREDUCE-2068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-2068: --- Attachment: MAPREDUCE-2068.r3.diff 0.23-MAPREDUCE-2068.r3.diff 0.22-MAPREDUCE-2068.r3.diff Rebased patches for 0.22, 0.23 and Trunk. TaskInProgress contains duplicate idWithinJob() / getIdWithinJob() accessors Key: MAPREDUCE-2068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2068 Project: Hadoop Map/Reduce Issue Type: Bug Components: task Affects Versions: 0.22.0 Reporter: Greg Roelofs Assignee: Harsh J Priority: Minor Fix For: 0.23.0 Attachments: 0.22-MAPREDUCE-2068.r3.diff, 0.23-MAPREDUCE-2068.r3.diff, MAPREDUCE-2068.r1.diff, MAPREDUCE-2068.r2.diff, MAPREDUCE-2068.r3.diff Both are public, both return partition;, and neither is currently deprecated. Main user seems to be JobInProgress, and it mostly uses the get flavor, but it also uses the other one in two places. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2068) TaskInProgress contains duplicate idWithinJob() / getIdWithinJob() accessors
[ https://issues.apache.org/jira/browse/MAPREDUCE-2068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-2068: --- Status: Open (was: Patch Available) TaskInProgress contains duplicate idWithinJob() / getIdWithinJob() accessors Key: MAPREDUCE-2068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2068 Project: Hadoop Map/Reduce Issue Type: Bug Components: task Affects Versions: 0.22.0 Reporter: Greg Roelofs Assignee: Harsh J Priority: Minor Fix For: 0.23.0 Attachments: 0.22-MAPREDUCE-2068.r3.diff, 0.23-MAPREDUCE-2068.r3.diff, MAPREDUCE-2068.r1.diff, MAPREDUCE-2068.r2.diff, MAPREDUCE-2068.r3.diff Both are public, both return partition;, and neither is currently deprecated. Main user seems to be JobInProgress, and it mostly uses the get flavor, but it also uses the other one in two places. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2068) TaskInProgress contains duplicate idWithinJob() / getIdWithinJob() accessors
[ https://issues.apache.org/jira/browse/MAPREDUCE-2068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-2068: --- Status: Patch Available (was: Open) This is highly trivial. Hoping for it to get some committer love soon. TaskInProgress contains duplicate idWithinJob() / getIdWithinJob() accessors Key: MAPREDUCE-2068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2068 Project: Hadoop Map/Reduce Issue Type: Bug Components: task Affects Versions: 0.22.0 Reporter: Greg Roelofs Assignee: Harsh J Priority: Minor Fix For: 0.23.0 Attachments: 0.22-MAPREDUCE-2068.r3.diff, 0.23-MAPREDUCE-2068.r3.diff, MAPREDUCE-2068.r1.diff, MAPREDUCE-2068.r2.diff, MAPREDUCE-2068.r3.diff Both are public, both return partition;, and neither is currently deprecated. Main user seems to be JobInProgress, and it mostly uses the get flavor, but it also uses the other one in two places. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2068) TaskInProgress contains duplicate idWithinJob() / getIdWithinJob() accessors
[ https://issues.apache.org/jira/browse/MAPREDUCE-2068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096720#comment-13096720 ] Hadoop QA commented on MAPREDUCE-2068: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12492910/MAPREDUCE-2068.r3.diff against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 8 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/599//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/599//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/599//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/599//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/599//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/599//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-jobclient.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/599//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/599//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/599//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/599//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/599//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/599//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-api.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/599//console This message is automatically generated. TaskInProgress contains duplicate idWithinJob() / getIdWithinJob() accessors Key: MAPREDUCE-2068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2068 Project: Hadoop Map/Reduce Issue Type: Bug Components: task Affects Versions: 0.22.0 Reporter: Greg Roelofs Assignee: Harsh J Priority: Minor Fix For: 0.23.0 Attachments: 0.22-MAPREDUCE-2068.r3.diff, 0.23-MAPREDUCE-2068.r3.diff, MAPREDUCE-2068.r1.diff, MAPREDUCE-2068.r2.diff, MAPREDUCE-2068.r3.diff Both are public, both return partition;, and neither is currently deprecated. Main user seems to be JobInProgress, and it mostly uses the get flavor, but it also uses the other one in two places. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2640) The maxRunningTasks property of the LimitTasksPerJob scheduler is ambiguous in its description, and must be updated
[
https://issues.apache.org/jira/browse/MAPREDUCE-2640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated MAPREDUCE-2640:
---
Status: Patch Available (was: Open)
The maxRunningTasks property of the LimitTasksPerJob scheduler is ambiguous
in its description, and must be updated
---
Key: MAPREDUCE-2640
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2640
Project: Hadoop Map/Reduce
Issue Type: Task
Components: documentation
Affects Versions: 0.20.2
Reporter: Harsh J
Assignee: Harsh J
Labels: doc
Fix For: 0.23.0
Attachments: 0.22-MAPREDUCE-2640.r1.diff,
0.23-MAPREDUCE-2640.r1.diff, MAPREDUCE-2640.r1.diff
The property's entry in mapred-default.xml is like so:
{code}
property
namemapreduce.jobtracker.taskscheduler.maxrunningtasks.perjob/name
value/value
descriptionThe maximum number of running tasks for a job before
it gets preempted. No limits if undefined.
/description
/property
{code}
There is no mention that this is a property exclusive to the LimitTasksPerJob
scheduler alone. The doc ought to be updated to note that unless there's a
plan of reusing such a property (I do not see fair or capacity schedulers
utilizing this, and they use their own configs).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2640) The maxRunningTasks property of the LimitTasksPerJob scheduler is ambiguous in its description, and must be updated
[
https://issues.apache.org/jira/browse/MAPREDUCE-2640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated MAPREDUCE-2640:
---
Attachment: MAPREDUCE-2640.r1.diff
0.23-MAPREDUCE-2640.r1.diff
0.22-MAPREDUCE-2640.r1.diff
Patches for trunk, 0.23, 0.22 that deprecates the misleading property and
provides a more relevant, new one.
The maxRunningTasks property of the LimitTasksPerJob scheduler is ambiguous
in its description, and must be updated
---
Key: MAPREDUCE-2640
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2640
Project: Hadoop Map/Reduce
Issue Type: Task
Components: documentation
Affects Versions: 0.20.2
Reporter: Harsh J
Assignee: Harsh J
Labels: doc
Fix For: 0.23.0
Attachments: 0.22-MAPREDUCE-2640.r1.diff,
0.23-MAPREDUCE-2640.r1.diff, MAPREDUCE-2640.r1.diff
The property's entry in mapred-default.xml is like so:
{code}
property
namemapreduce.jobtracker.taskscheduler.maxrunningtasks.perjob/name
value/value
descriptionThe maximum number of running tasks for a job before
it gets preempted. No limits if undefined.
/description
/property
{code}
There is no mention that this is a property exclusive to the LimitTasksPerJob
scheduler alone. The doc ought to be updated to note that unless there's a
plan of reusing such a property (I do not see fair or capacity schedulers
utilizing this, and they use their own configs).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2919) The JT web UI should show job start times
[ https://issues.apache.org/jira/browse/MAPREDUCE-2919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-2919: --- Attachment: Screen shot 2011-09-04 at 1.14.00 AM.png MAPREDUCE-2919.r1.diff Wish-granting patch attached, and also a screenshot after manual testing ;-) The JT web UI should show job start times -- Key: MAPREDUCE-2919 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2919 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobtracker Affects Versions: 0.20.203.0 Reporter: Eli Collins Assignee: Harsh J Priority: Minor Labels: newbie Attachments: MAPREDUCE-2919.r1.diff, Screen shot 2011-09-04 at 1.14.00 AM.png It would be helpful if the list of jobs in the main JT web UI (running, completed, failed..) had a column with the start time. Clicking into each job detail can get tedious. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2919) The JT web UI should show job start times
[ https://issues.apache.org/jira/browse/MAPREDUCE-2919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J reassigned MAPREDUCE-2919: -- Assignee: Harsh J The JT web UI should show job start times -- Key: MAPREDUCE-2919 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2919 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobtracker Affects Versions: 0.20.203.0 Reporter: Eli Collins Assignee: Harsh J Priority: Minor Labels: newbie Attachments: MAPREDUCE-2919.r1.diff, Screen shot 2011-09-04 at 1.14.00 AM.png It would be helpful if the list of jobs in the main JT web UI (running, completed, failed..) had a column with the start time. Clicking into each job detail can get tedious. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2919) The JT web UI should show job start times
[ https://issues.apache.org/jira/browse/MAPREDUCE-2919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-2919: --- Status: Patch Available (was: Open) The JT web UI should show job start times -- Key: MAPREDUCE-2919 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2919 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobtracker Affects Versions: 0.20.203.0 Reporter: Eli Collins Assignee: Harsh J Priority: Minor Labels: newbie Attachments: MAPREDUCE-2919.r1.diff, Screen shot 2011-09-04 at 1.14.00 AM.png It would be helpful if the list of jobs in the main JT web UI (running, completed, failed..) had a column with the start time. Clicking into each job detail can get tedious. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2910) Allow empty MapOutputFile segments
[ https://issues.apache.org/jira/browse/MAPREDUCE-2910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096747#comment-13096747 ] Harsh J commented on MAPREDUCE-2910: How much is the overhead of compressed, empty partition files? Allow empty MapOutputFile segments -- Key: MAPREDUCE-2910 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2910 Project: Hadoop Map/Reduce Issue Type: Improvement Components: task, tasktracker Affects Versions: 0.20.2, 0.23.0 Reporter: Binglin Chang Priority: Minor Fix For: 0.23.0 As the scale of cluster and job get larger, we see a lot of empty partitions in MapOutputFile due to large reduce numbers or partition skew. When map output compression is enabled, empty map output partitions gets larger has additional compressor/decompressor initialization overhead. This can be optimized by allowing empty MapOutputFile segments, where the rawLength partLength of IndexRecord all equal to 0. Corresponding support need to be added to IFile reader, writer, and reduce shuffle copier. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2845) Default replication level mapred.submit.replication=10 causes warnings on small clusters
[
https://issues.apache.org/jira/browse/MAPREDUCE-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096751#comment-13096751
]
Harsh J commented on MAPREDUCE-2845:
The property 'mapred.submit.replication' is configurable for that reason; or is
this a request to lower that value?
Indeed, this would also hamper decommissioning efforts when an admin doesn't
know about it.
I think making it dynamic makes sense, but adds RPC overheads possibly, per
submission.
Default replication level mapred.submit.replication=10 causes warnings on
small clusters
Key: MAPREDUCE-2845
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2845
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client, distributed-cache
Affects Versions: 0.20.1
Environment: Cloudera CDH 2 (hadoop-0.20 0.20.1+169.127-1~lenny-cdh2)
Reporter: Christoph Schmitz
Priority: Minor
By default, the replication level for job jars, libjars and the distributed
cache in general is mapred.submit.replication=10. This yields
under-replication warnings for these files on small clusters (less than 10
data nodes) when using fsck (hadoop fsck) on their HDFS.
Example on an 8-node cluster:
{quote}
/tmp/hadoop/mapred/system/job_201105191458_1857/job.jar: Under replicated
blk_-6996370258385460742_366223. Target Replicas is 10 but found 8 replica(s).
{quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2919) The JT web UI should show job start times
[ https://issues.apache.org/jira/browse/MAPREDUCE-2919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096750#comment-13096750 ] Hadoop QA commented on MAPREDUCE-2919: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12492929/Screen+shot+2011-09-04+at+1.14.00+AM.png against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/601//console This message is automatically generated. The JT web UI should show job start times -- Key: MAPREDUCE-2919 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2919 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobtracker Affects Versions: 0.20.203.0 Reporter: Eli Collins Assignee: Harsh J Priority: Minor Labels: newbie Attachments: MAPREDUCE-2919.r1.diff, Screen shot 2011-09-04 at 1.14.00 AM.png It would be helpful if the list of jobs in the main JT web UI (running, completed, failed..) had a column with the start time. Clicking into each job detail can get tedious. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2821) [MR-279] Missing fields in job summary logs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J reassigned MAPREDUCE-2821: -- Assignee: Harsh J [MR-279] Missing fields in job summary logs Key: MAPREDUCE-2821 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2821 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Assignee: Harsh J Priority: Minor Fix For: 0.23.0 The following fields are missing in the job summary logs in mrv2: - numSlotsPerMap - numSlotsPerReduce - clusterCapacity (Earlier known as clusterMapCapacity and clusterReduceCapacity in 0.20.x) The first two fields are important to know if the job was a High RAM job or not and the last field is important to know the total available resource in the cluster during job execution. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2821) [MR-279] Missing fields in job summary logs
[
https://issues.apache.org/jira/browse/MAPREDUCE-2821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096752#comment-13096752
]
Harsh J commented on MAPREDUCE-2821:
Per code comments, these appear to have been disabled for a reason:
{code}
// private int numSlotsPerMap; | Doesn't make sense with potentially different
// resource models
// private int numSlotsPerReduce; | Doesn't make sense with potentially
// different resource models
{code}
So is your reasoning atop these?
[MR-279] Missing fields in job summary logs
Key: MAPREDUCE-2821
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2821
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Ramya Sunil
Assignee: Harsh J
Priority: Minor
Fix For: 0.23.0
The following fields are missing in the job summary logs in mrv2:
- numSlotsPerMap
- numSlotsPerReduce
- clusterCapacity (Earlier known as clusterMapCapacity and
clusterReduceCapacity in 0.20.x)
The first two fields are important to know if the job was a High RAM job or
not and the last field is important to know the total available resource in
the cluster during job execution.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2817) MiniRMCluster hardcodes 'mapred.local.dir' configuration to 'build/test/mapred/local'
[
https://issues.apache.org/jira/browse/MAPREDUCE-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J reassigned MAPREDUCE-2817:
--
Assignee: Harsh J
MiniRMCluster hardcodes 'mapred.local.dir' configuration to
'build/test/mapred/local'
-
Key: MAPREDUCE-2817
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2817
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Environment: all
Reporter: Alejandro Abdelnur
Assignee: Harsh J
Priority: Minor
The {{mapred.local.dir}} configuration property for the {{MiniMRCluster}} is
forced to {{build/test/mapred/local}}
This is inconvenient in different situations. For example:
* When running multiple tests using {{MiniMRCluster}} is not possible to see
the end state of the dir for a particular test
* When using {{MiniMRCluster}} in another build system (i.e. Maven) that uses
a different output directory (target instead build)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2817) MiniRMCluster hardcodes 'mapred.local.dir' configuration to 'build/test/mapred/local'
[
https://issues.apache.org/jira/browse/MAPREDUCE-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated MAPREDUCE-2817:
---
Affects Version/s: 0.20.2
Fix Version/s: 0.20.205.0
(Patch is for 0.20, security branch.)
MiniRMCluster hardcodes 'mapred.local.dir' configuration to
'build/test/mapred/local'
-
Key: MAPREDUCE-2817
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2817
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 0.20.2
Environment: all
Reporter: Alejandro Abdelnur
Assignee: Harsh J
Priority: Minor
Fix For: 0.20.205.0
Attachments: MAPREDUCE-2817.r1.diff
The {{mapred.local.dir}} configuration property for the {{MiniMRCluster}} is
forced to {{build/test/mapred/local}}
This is inconvenient in different situations. For example:
* When running multiple tests using {{MiniMRCluster}} is not possible to see
the end state of the dir for a particular test
* When using {{MiniMRCluster}} in another build system (i.e. Maven) that uses
a different output directory (target instead build)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2817) MiniRMCluster hardcodes 'mapred.local.dir' configuration to 'build/test/mapred/local'
[
https://issues.apache.org/jira/browse/MAPREDUCE-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated MAPREDUCE-2817:
---
Attachment: MAPREDUCE-2817.r1.diff
Patch that incorporates Alejandro's suggestion with test.build.mapred sysprop.
Ran a sample MR cluster program to test change.
{code}
[junit] Running org.apache.hadoop.mapred.TestMiniMRChildTask
[junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 119.227 sec
{code}
MiniRMCluster hardcodes 'mapred.local.dir' configuration to
'build/test/mapred/local'
-
Key: MAPREDUCE-2817
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2817
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Environment: all
Reporter: Alejandro Abdelnur
Assignee: Harsh J
Priority: Minor
Attachments: MAPREDUCE-2817.r1.diff
The {{mapred.local.dir}} configuration property for the {{MiniMRCluster}} is
forced to {{build/test/mapred/local}}
This is inconvenient in different situations. For example:
* When running multiple tests using {{MiniMRCluster}} is not possible to see
the end state of the dir for a particular test
* When using {{MiniMRCluster}} in another build system (i.e. Maven) that uses
a different output directory (target instead build)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2910) Allow empty MapOutputFile segments
[ https://issues.apache.org/jira/browse/MAPREDUCE-2910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096818#comment-13096818 ] Binglin Chang commented on MAPREDUCE-2910: -- LzoCodec: 2byte EOF marker + 4 byte checksum - 14 byte compressed data + 4 byte checksum GzipCodec: 2byte EOF marker + 4 byte checksum - 26 byte compressed data + 4 byte checksum Empty segments don't have any bytes, thus the seek read in MapOutputServlet can also be saved. This optimization is only for extreme cases, I often see large proportion(90%) of empty segments in vary large jobs(particularly with map side filter) in our cluster, this is partially because of bad configuration or bad partitioner, but tuning a partitioner or key distribution sometimes is non trivial for user. Allow empty MapOutputFile segments -- Key: MAPREDUCE-2910 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2910 Project: Hadoop Map/Reduce Issue Type: Improvement Components: task, tasktracker Affects Versions: 0.20.2, 0.23.0 Reporter: Binglin Chang Priority: Minor Fix For: 0.23.0 As the scale of cluster and job get larger, we see a lot of empty partitions in MapOutputFile due to large reduce numbers or partition skew. When map output compression is enabled, empty map output partitions gets larger has additional compressor/decompressor initialization overhead. This can be optimized by allowing empty MapOutputFile segments, where the rawLength partLength of IndexRecord all equal to 0. Corresponding support need to be added to IFile reader, writer, and reduce shuffle copier. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
