[jira] [Commented] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13125767#comment-13125767 ] Hudson commented on MAPREDUCE-2702: --- Integrated in Hadoop-Hdfs-trunk #828 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/828/]) MAPREDUCE-3148. Ported MAPREDUCE-2702 to old mapred api for aiding task recovery. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1182008 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/dev-support/findbugs-exclude.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputCommitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MRConstants.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/OutputCommitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestFileOutputCommitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestFileOutputCommitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java [MR-279] OutputCommitter changes for MR Application Master recovery --- Key: MAPREDUCE-2702 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, mr2702_v1.patch, mr2702_v2.patch, mr2702_v3.patch, mr2702_v4.patch In MR AM recovers from a crash, it only reruns the non completed tasks. The completed tasks (along with their output, if any) needs to be recovered from the previous life. This would require some changes in OutputCommitter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13125833#comment-13125833 ] Hudson commented on MAPREDUCE-2702: --- Integrated in Hadoop-Mapreduce-trunk #858 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/858/]) MAPREDUCE-3148. Ported MAPREDUCE-2702 to old mapred api for aiding task recovery. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1182008 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/dev-support/findbugs-exclude.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputCommitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MRConstants.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/OutputCommitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestFileOutputCommitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestFileOutputCommitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java [MR-279] OutputCommitter changes for MR Application Master recovery --- Key: MAPREDUCE-2702 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, mr2702_v1.patch, mr2702_v2.patch, mr2702_v3.patch, mr2702_v4.patch In MR AM recovers from a crash, it only reruns the non completed tasks. The completed tasks (along with their output, if any) needs to be recovered from the previous life. This would require some changes in OutputCommitter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13125259#comment-13125259 ] Hudson commented on MAPREDUCE-2702: --- Integrated in Hadoop-Common-trunk-Commit #1057 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1057/]) MAPREDUCE-3148. Ported MAPREDUCE-2702 to old mapred api for aiding task recovery. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1182008 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/dev-support/findbugs-exclude.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputCommitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MRConstants.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/OutputCommitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestFileOutputCommitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestFileOutputCommitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java [MR-279] OutputCommitter changes for MR Application Master recovery --- Key: MAPREDUCE-2702 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, mr2702_v1.patch, mr2702_v2.patch, mr2702_v3.patch, mr2702_v4.patch In MR AM recovers from a crash, it only reruns the non completed tasks. The completed tasks (along with their output, if any) needs to be recovered from the previous life. This would require some changes in OutputCommitter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13125263#comment-13125263 ] Hudson commented on MAPREDUCE-2702: --- Integrated in Hadoop-Hdfs-trunk-Commit #1135 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1135/]) MAPREDUCE-3148. Ported MAPREDUCE-2702 to old mapred api for aiding task recovery. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1182008 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/dev-support/findbugs-exclude.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputCommitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MRConstants.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/OutputCommitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestFileOutputCommitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestFileOutputCommitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java [MR-279] OutputCommitter changes for MR Application Master recovery --- Key: MAPREDUCE-2702 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, mr2702_v1.patch, mr2702_v2.patch, mr2702_v3.patch, mr2702_v4.patch In MR AM recovers from a crash, it only reruns the non completed tasks. The completed tasks (along with their output, if any) needs to be recovered from the previous life. This would require some changes in OutputCommitter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13125277#comment-13125277 ] Hudson commented on MAPREDUCE-2702: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1077 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1077/]) MAPREDUCE-3148. Ported MAPREDUCE-2702 to old mapred api for aiding task recovery. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1182008 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/dev-support/findbugs-exclude.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputCommitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MRConstants.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/OutputCommitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestFileOutputCommitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestFileOutputCommitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java [MR-279] OutputCommitter changes for MR Application Master recovery --- Key: MAPREDUCE-2702 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, mr2702_v1.patch, mr2702_v2.patch, mr2702_v3.patch, mr2702_v4.patch In MR AM recovers from a crash, it only reruns the non completed tasks. The completed tasks (along with their output, if any) needs to be recovered from the previous life. This would require some changes in OutputCommitter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124020#comment-13124020 ] Hudson commented on MAPREDUCE-2702: --- Integrated in Hadoop-Hdfs-trunk #826 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/826/]) MAPREDUCE-3153. Fix TestFileOutputCommitter which was broken by MAPREDUCE-2702. Contributed by Mahadev Konar. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1180769 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java [MR-279] OutputCommitter changes for MR Application Master recovery --- Key: MAPREDUCE-2702 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, mr2702_v1.patch, mr2702_v2.patch, mr2702_v3.patch, mr2702_v4.patch In MR AM recovers from a crash, it only reruns the non completed tasks. The completed tasks (along with their output, if any) needs to be recovered from the previous life. This would require some changes in OutputCommitter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124084#comment-13124084 ] Hudson commented on MAPREDUCE-2702: --- Integrated in Hadoop-Mapreduce-trunk #856 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/856/]) MAPREDUCE-3153. Fix TestFileOutputCommitter which was broken by MAPREDUCE-2702. Contributed by Mahadev Konar. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1180769 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java [MR-279] OutputCommitter changes for MR Application Master recovery --- Key: MAPREDUCE-2702 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, mr2702_v1.patch, mr2702_v2.patch, mr2702_v3.patch, mr2702_v4.patch In MR AM recovers from a crash, it only reruns the non completed tasks. The completed tasks (along with their output, if any) needs to be recovered from the previous life. This would require some changes in OutputCommitter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13123858#comment-13123858 ] Hudson commented on MAPREDUCE-2702: --- Integrated in Hadoop-Hdfs-trunk-Commit #1126 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1126/]) MAPREDUCE-3153. Fix TestFileOutputCommitter which was broken by MAPREDUCE-2702. Contributed by Mahadev Konar. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1180769 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java [MR-279] OutputCommitter changes for MR Application Master recovery --- Key: MAPREDUCE-2702 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, mr2702_v1.patch, mr2702_v2.patch, mr2702_v3.patch, mr2702_v4.patch In MR AM recovers from a crash, it only reruns the non completed tasks. The completed tasks (along with their output, if any) needs to be recovered from the previous life. This would require some changes in OutputCommitter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13123862#comment-13123862 ] Hudson commented on MAPREDUCE-2702: --- Integrated in Hadoop-Common-trunk-Commit #1048 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1048/]) MAPREDUCE-3153. Fix TestFileOutputCommitter which was broken by MAPREDUCE-2702. Contributed by Mahadev Konar. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1180769 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java [MR-279] OutputCommitter changes for MR Application Master recovery --- Key: MAPREDUCE-2702 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, mr2702_v1.patch, mr2702_v2.patch, mr2702_v3.patch, mr2702_v4.patch In MR AM recovers from a crash, it only reruns the non completed tasks. The completed tasks (along with their output, if any) needs to be recovered from the previous life. This would require some changes in OutputCommitter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13123877#comment-13123877 ] Hudson commented on MAPREDUCE-2702: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1068 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1068/]) MAPREDUCE-3153. Fix TestFileOutputCommitter which was broken by MAPREDUCE-2702. Contributed by Mahadev Konar. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1180769 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java [MR-279] OutputCommitter changes for MR Application Master recovery --- Key: MAPREDUCE-2702 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, mr2702_v1.patch, mr2702_v2.patch, mr2702_v3.patch, mr2702_v4.patch In MR AM recovers from a crash, it only reruns the non completed tasks. The completed tasks (along with their output, if any) needs to be recovered from the previous life. This would require some changes in OutputCommitter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13121895#comment-13121895 ] Hudson commented on MAPREDUCE-2702: --- Integrated in Hadoop-Hdfs-0.23-Build #31 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/31/]) MAPREDUCE-2702. svn merge -c r1179188 --ignore-ancestry ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1179191 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/OutputCommitter.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java [MR-279] OutputCommitter changes for MR Application Master recovery --- Key: MAPREDUCE-2702 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, mr2702_v1.patch, mr2702_v2.patch, mr2702_v3.patch, mr2702_v4.patch In MR AM recovers from a crash, it only reruns the non completed tasks. The completed tasks (along with their output, if any) needs to be recovered from the previous life. This would require some changes in OutputCommitter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13121922#comment-13121922 ] Hudson commented on MAPREDUCE-2702: --- Integrated in Hadoop-Hdfs-trunk #822 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/822/]) MAPREDUCE-2702. Added a new API in OutputCommitter for recovering the outputs of tasks from a crashed job so as to support MR Application Master recovery. Contributed by Sharad Agarwal and Arun C Murthy. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1179188 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/OutputCommitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java [MR-279] OutputCommitter changes for MR Application Master recovery --- Key: MAPREDUCE-2702 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, mr2702_v1.patch, mr2702_v2.patch, mr2702_v3.patch, mr2702_v4.patch In MR AM recovers from a crash, it only reruns the non completed tasks. The completed tasks (along with their output, if any) needs to be recovered from the previous life. This would require some changes in OutputCommitter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13120709#comment-13120709 ] Hadoop QA commented on MAPREDUCE-2702: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12497751/MAPREDUCE-2702.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/945//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/945//console This message is automatically generated. [MR-279] OutputCommitter changes for MR Application Master recovery --- Key: MAPREDUCE-2702 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mrv2 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Priority: Blocker Attachments: MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, mr2702_v1.patch, mr2702_v2.patch, mr2702_v3.patch, mr2702_v4.patch In MR AM recovers from a crash, it only reruns the non completed tasks. The completed tasks (along with their output, if any) needs to be recovered from the previous life. This would require some changes in OutputCommitter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13120708#comment-13120708 ] Sharad Agarwal commented on MAPREDUCE-2702: --- bq. uniting isRecoverySupported and recoverTask into a single api combining the api has problem: - you won't know that recovery is supported or not until you use the recover task api. currently recover code path is different and only executed if recovery is supported. - recovery is at job level, while recover task is task level. semantically it is not very clear because recover task is invoked multiple times. [MR-279] OutputCommitter changes for MR Application Master recovery --- Key: MAPREDUCE-2702 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mrv2 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Priority: Blocker Attachments: MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, mr2702_v1.patch, mr2702_v2.patch, mr2702_v3.patch, mr2702_v4.patch In MR AM recovers from a crash, it only reruns the non completed tasks. The completed tasks (along with their output, if any) needs to be recovered from the previous life. This would require some changes in OutputCommitter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13120751#comment-13120751 ] Hadoop QA commented on MAPREDUCE-2702: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12497765/MAPREDUCE-2702.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/946//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/946//console This message is automatically generated. [MR-279] OutputCommitter changes for MR Application Master recovery --- Key: MAPREDUCE-2702 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mrv2 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, mr2702_v1.patch, mr2702_v2.patch, mr2702_v3.patch, mr2702_v4.patch In MR AM recovers from a crash, it only reruns the non completed tasks. The completed tasks (along with their output, if any) needs to be recovered from the previous life. This would require some changes in OutputCommitter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13120893#comment-13120893 ] Hudson commented on MAPREDUCE-2702: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1036 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1036/]) MAPREDUCE-2702. Added a new API in OutputCommitter for recovering the outputs of tasks from a crashed job so as to support MR Application Master recovery. Contributed by Sharad Agarwal and Arun C Murthy. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1179188 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/OutputCommitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java [MR-279] OutputCommitter changes for MR Application Master recovery --- Key: MAPREDUCE-2702 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702 Project: Hadoop Map/Reduce Issue Type: New Feature Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, mr2702_v1.patch, mr2702_v2.patch, mr2702_v3.patch, mr2702_v4.patch In MR AM recovers from a crash, it only reruns the non completed tasks. The completed tasks (along with their output, if any) needs to be recovered from the previous life. This would require some changes in OutputCommitter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13120907#comment-13120907 ] Hudson commented on MAPREDUCE-2702: --- Integrated in Hadoop-Hdfs-trunk-Commit #1098 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1098/]) MAPREDUCE-2702. Added a new API in OutputCommitter for recovering the outputs of tasks from a crashed job so as to support MR Application Master recovery. Contributed by Sharad Agarwal and Arun C Murthy. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1179188 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/OutputCommitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java [MR-279] OutputCommitter changes for MR Application Master recovery --- Key: MAPREDUCE-2702 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702 Project: Hadoop Map/Reduce Issue Type: New Feature Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, mr2702_v1.patch, mr2702_v2.patch, mr2702_v3.patch, mr2702_v4.patch In MR AM recovers from a crash, it only reruns the non completed tasks. The completed tasks (along with their output, if any) needs to be recovered from the previous life. This would require some changes in OutputCommitter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13120931#comment-13120931 ] Hudson commented on MAPREDUCE-2702: --- Integrated in Hadoop-Mapreduce-0.23-Build #37 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/37/]) MAPREDUCE-2702. svn merge -c r1179188 --ignore-ancestry ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1179191 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/OutputCommitter.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java [MR-279] OutputCommitter changes for MR Application Master recovery --- Key: MAPREDUCE-2702 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702 Project: Hadoop Map/Reduce Issue Type: New Feature Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, mr2702_v1.patch, mr2702_v2.patch, mr2702_v3.patch, mr2702_v4.patch In MR AM recovers from a crash, it only reruns the non completed tasks. The completed tasks (along with their output, if any) needs to be recovered from the previous life. This would require some changes in OutputCommitter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13120945#comment-13120945 ] Hudson commented on MAPREDUCE-2702: --- Integrated in Hadoop-Mapreduce-trunk #851 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/851/]) MAPREDUCE-2702. Added a new API in OutputCommitter for recovering the outputs of tasks from a crashed job so as to support MR Application Master recovery. Contributed by Sharad Agarwal and Arun C Murthy. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1179188 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/OutputCommitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java [MR-279] OutputCommitter changes for MR Application Master recovery --- Key: MAPREDUCE-2702 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702 Project: Hadoop Map/Reduce Issue Type: New Feature Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, mr2702_v1.patch, mr2702_v2.patch, mr2702_v3.patch, mr2702_v4.patch In MR AM recovers from a crash, it only reruns the non completed tasks. The completed tasks (along with their output, if any) needs to be recovered from the previous life. This would require some changes in OutputCommitter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13119935#comment-13119935 ] Arun C Murthy commented on MAPREDUCE-2702: -- Sharad, apologies for not getting around to this, but I finally spent sufficient time on this to do this important patch justice. Overall, it looks great! Some minor nits: # How about uniting isRecoverySupported and recoverTask into a single api: 'boolean recoverTask'. So, an OutputCommitter can return false from 'recoverTask' to indicate it doesn't support it. # I'm a little concerned about introducing the notion of 'appAttemptId' as an integer to recoverTask. This is probably one of the very few, if not the only place, where YARN leaks into 'classic' MapReduce apis - particularly since these are user-facing apis. Maybe we can just pull it out of TaskAttemptContext.configuration? # Other minor, cosmetic suggestions for function names: ## getJobTempDir - getJobTempDirName ## getTaskTempDir - getTaskTempDirName ## moveOutputs - moveJobOutputs Thoughts? [MR-279] OutputCommitter changes for MR Application Master recovery --- Key: MAPREDUCE-2702 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mrv2 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Priority: Blocker Attachments: mr2702_v1.patch, mr2702_v2.patch, mr2702_v3.patch, mr2702_v4.patch In MR AM recovers from a crash, it only reruns the non completed tasks. The completed tasks (along with their output, if any) needs to be recovered from the previous life. This would require some changes in OutputCommitter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13119972#comment-13119972 ] Hadoop QA commented on MAPREDUCE-2702: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12497598/MAPREDUCE-2702.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/934//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/934//console This message is automatically generated. [MR-279] OutputCommitter changes for MR Application Master recovery --- Key: MAPREDUCE-2702 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mrv2 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Priority: Blocker Attachments: MAPREDUCE-2702.patch, mr2702_v1.patch, mr2702_v2.patch, mr2702_v3.patch, mr2702_v4.patch In MR AM recovers from a crash, it only reruns the non completed tasks. The completed tasks (along with their output, if any) needs to be recovered from the previous life. This would require some changes in OutputCommitter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13120212#comment-13120212 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-2702: Looked at the patch. Some comments: - I think AppAttemptId is part of the API one way or the other. If we are not making the appAttemptId as part of the API explicitly, we should document recoverTask() about the Id being passed as part of the context/config so that folks with their own OuputCommitters can know. - We should probably drop the boolean return value also and rely explicitly on the exception, it isn't adding any value anyways. - We need to document that if recoverTask() throws an exception, that task's output won't be used in the new generation. - You missed changes to the old-api that you hinted at? - The method names got a little confusing with so many usages of 'temp', how about simply calling them getJobDirName() and getTaskDirName()? (Funny we never had a test-case for FileOutputCommitter, good test Sharad :) ) [MR-279] OutputCommitter changes for MR Application Master recovery --- Key: MAPREDUCE-2702 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mrv2 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Priority: Blocker Attachments: MAPREDUCE-2702.patch, mr2702_v1.patch, mr2702_v2.patch, mr2702_v3.patch, mr2702_v4.patch In MR AM recovers from a crash, it only reruns the non completed tasks. The completed tasks (along with their output, if any) needs to be recovered from the previous life. This would require some changes in OutputCommitter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117024#comment-13117024 ] Arun C Murthy commented on MAPREDUCE-2702: -- True, but too many folks are using the old api. Let me (or get someone to) just port it to the old api, ok? [MR-279] OutputCommitter changes for MR Application Master recovery --- Key: MAPREDUCE-2702 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mrv2 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Priority: Blocker Attachments: mr2702_v1.patch, mr2702_v2.patch, mr2702_v3.patch, mr2702_v4.patch In MR AM recovers from a crash, it only reruns the non completed tasks. The completed tasks (along with their output, if any) needs to be recovered from the previous life. This would require some changes in OutputCommitter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13116228#comment-13116228 ] Sharad Agarwal commented on MAPREDUCE-2702: --- can be done as separate jira for old api. anyway if the outputcommitter doesn't support recovery (the default), the MR AM falls back to rerun the job from beginning. [MR-279] OutputCommitter changes for MR Application Master recovery --- Key: MAPREDUCE-2702 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mrv2 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Priority: Blocker Attachments: mr2702_v1.patch, mr2702_v2.patch, mr2702_v3.patch, mr2702_v4.patch In MR AM recovers from a crash, it only reruns the non completed tasks. The completed tasks (along with their output, if any) needs to be recovered from the previous life. This would require some changes in OutputCommitter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13115297#comment-13115297 ] Arun C Murthy commented on MAPREDUCE-2702: -- Sharad, the approach and the patch look reasonable - I'll look more closely. The one nit is that we need to ensure the 'old' api i.e. o.a.h.mapred.(OutputCommitter|FileOutputCommitter) also supports this, can you please do that too? Thanks. [MR-279] OutputCommitter changes for MR Application Master recovery --- Key: MAPREDUCE-2702 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mrv2 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Priority: Blocker Attachments: mr2702_v1.patch, mr2702_v2.patch, mr2702_v3.patch, mr2702_v4.patch In MR AM recovers from a crash, it only reruns the non completed tasks. The completed tasks (along with their output, if any) needs to be recovered from the previous life. This would require some changes in OutputCommitter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13092027#comment-13092027 ] Hadoop QA commented on MAPREDUCE-2702: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12491601/mr2702_v3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 76 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 5 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/539//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/539//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/539//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-api.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/539//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/539//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/539//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/539//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/539//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/539//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/539//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/539//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/539//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-jobclient.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/539//console This message is automatically generated. [MR-279] OutputCommitter changes for MR Application Master recovery --- Key: MAPREDUCE-2702 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mrv2 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Attachments: mr2702_v1.patch, mr2702_v2.patch, mr2702_v3.patch In MR AM recovers from a crash, it only reruns the non completed tasks. The completed tasks (along with their output, if any) needs to be recovered from the previous life. This would require some changes in OutputCommitter. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079906#comment-13079906 ] Sharad Agarwal commented on MAPREDUCE-2702: --- Also as part of the last patch, fixed the directory structure from mapreduce/mr-client/hadoop-mapreduce-client-core/src/test/mapred to mapreduce/mr-client/hadoop-mapreduce-client-core/src/test/java [MR-279] OutputCommitter changes for MR Application Master recovery --- Key: MAPREDUCE-2702 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mrv2 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Attachments: mr2702_v1.patch In MR AM recovers from a crash, it only reruns the non completed tasks. The completed tasks (along with their output, if any) needs to be recovered from the previous life. This would require some changes in OutputCommitter. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
