[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13224121#comment-13224121 ] Hudson commented on HDFS-1580: -- Integrated in Hadoop-Hdfs-trunk-Commit #1921 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1921/]) Moving HDFS-2158, HDFS-2188, HDFS-2334, HDFS-2477, HDFS-2495, HDFS-2476, HDFS-1580, HDFS-1765 to 0.23.3 section in CHANGES.txt (Revision 1297903) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1297903 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Ivan Kelly Assignee: Jitendra Nath Pandey Fix For: HA branch (HDFS-1623), 0.24.0, 0.23.3 Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13224129#comment-13224129 ] Hudson commented on HDFS-1580: -- Integrated in Hadoop-Common-trunk-Commit #1847 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1847/]) Moving HDFS-2158, HDFS-2188, HDFS-2334, HDFS-2477, HDFS-2495, HDFS-2476, HDFS-1580, HDFS-1765 to 0.23.3 section in CHANGES.txt (Revision 1297903) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1297903 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Ivan Kelly Assignee: Jitendra Nath Pandey Fix For: HA branch (HDFS-1623), 0.24.0, 0.23.3 Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13224139#comment-13224139 ] Hudson commented on HDFS-1580: -- Integrated in Hadoop-Mapreduce-trunk-Commit #1855 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1855/]) Moving HDFS-2158, HDFS-2188, HDFS-2334, HDFS-2477, HDFS-2495, HDFS-2476, HDFS-1580, HDFS-1765 to 0.23.3 section in CHANGES.txt (Revision 1297903) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1297903 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Ivan Kelly Assignee: Jitendra Nath Pandey Fix For: HA branch (HDFS-1623), 0.24.0, 0.23.3 Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13224251#comment-13224251 ] Hudson commented on HDFS-1580: -- Integrated in Hadoop-Hdfs-0.23-Build #190 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/190/]) HDFS-1580. Merging change r1210749, r1210602 from trunk to 0.23 (Revision 1297864) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1297864 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupInputStream.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupOutputStream.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogInputStream.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogOutputStream.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalManager.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGenericJournalConf.java Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Ivan Kelly Assignee: Jitendra Nath Pandey Fix For: HA branch (HDFS-1623), 0.24.0, 0.23.3 Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13224275#comment-13224275 ] Hudson commented on HDFS-1580: -- Integrated in Hadoop-Hdfs-trunk #977 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/977/]) Moving HDFS-2158, HDFS-2188, HDFS-2334, HDFS-2477, HDFS-2495, HDFS-2476, HDFS-1580, HDFS-1765 to 0.23.3 section in CHANGES.txt (Revision 1297903) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1297903 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Ivan Kelly Assignee: Jitendra Nath Pandey Fix For: HA branch (HDFS-1623), 0.24.0, 0.23.3 Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13224292#comment-13224292 ] Hudson commented on HDFS-1580: -- Integrated in Hadoop-Mapreduce-0.23-Build #218 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/218/]) HDFS-1580. Merging change r1210749, r1210602 from trunk to 0.23 (Revision 1297864) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1297864 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupInputStream.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupOutputStream.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogInputStream.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogOutputStream.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalManager.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGenericJournalConf.java Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Ivan Kelly Assignee: Jitendra Nath Pandey Fix For: HA branch (HDFS-1623), 0.24.0, 0.23.3 Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13224314#comment-13224314 ] Hudson commented on HDFS-1580: -- Integrated in Hadoop-Mapreduce-trunk #1012 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1012/]) Moving HDFS-2158, HDFS-2188, HDFS-2334, HDFS-2477, HDFS-2495, HDFS-2476, HDFS-1580, HDFS-1765 to 0.23.3 section in CHANGES.txt (Revision 1297903) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1297903 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Ivan Kelly Assignee: Jitendra Nath Pandey Fix For: HA branch (HDFS-1623), 0.24.0, 0.23.3 Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13224046#comment-13224046 ] Hudson commented on HDFS-1580: -- Integrated in Hadoop-Hdfs-0.23-Commit #645 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/645/]) HDFS-1580. Merging change r1210749, r1210602 from trunk to 0.23 (Revision 1297864) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1297864 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupInputStream.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupOutputStream.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogInputStream.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogOutputStream.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalManager.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGenericJournalConf.java Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Ivan Kelly Assignee: Jitendra Nath Pandey Fix For: HA branch (HDFS-1623), 0.24.0, 0.23.3 Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13224052#comment-13224052 ] Hudson commented on HDFS-1580: -- Integrated in Hadoop-Common-0.23-Commit #654 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/654/]) HDFS-1580. Merging change r1210749, r1210602 from trunk to 0.23 (Revision 1297864) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1297864 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupInputStream.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupOutputStream.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogInputStream.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogOutputStream.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalManager.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGenericJournalConf.java Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Ivan Kelly Assignee: Jitendra Nath Pandey Fix For: HA branch (HDFS-1623), 0.24.0, 0.23.3 Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13224066#comment-13224066 ] Hudson commented on HDFS-1580: -- Integrated in Hadoop-Mapreduce-0.23-Commit #655 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/655/]) HDFS-1580. Merging change r1210749, r1210602 from trunk to 0.23 (Revision 1297864) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1297864 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupInputStream.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupOutputStream.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogInputStream.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogOutputStream.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalManager.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGenericJournalConf.java Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Ivan Kelly Assignee: Jitendra Nath Pandey Fix For: HA branch (HDFS-1623), 0.24.0, 0.23.3 Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13163565#comment-13163565 ] Hudson commented on HDFS-1580: -- Integrated in Hadoop-Hdfs-trunk #886 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/886/]) HDFS-1580. Committing the test case, it was missed in earlier commit. HDFS-1580. Add interface for generic Write Ahead Logging mechanisms. Contributed by Ivan Kelly. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1210749 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGenericJournalConf.java jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1210602 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Assignee: Jitendra Nath Pandey Fix For: HA branch (HDFS-1623), 0.24.0 Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13163600#comment-13163600 ] Hudson commented on HDFS-1580: -- Integrated in Hadoop-Mapreduce-trunk #919 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/919/]) HDFS-1580. Committing the test case, it was missed in earlier commit. HDFS-1580. Add interface for generic Write Ahead Logging mechanisms. Contributed by Ivan Kelly. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1210749 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGenericJournalConf.java jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1210602 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Assignee: Jitendra Nath Pandey Fix For: HA branch (HDFS-1623), 0.24.0 Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13163011#comment-13163011 ] Hudson commented on HDFS-1580: -- Integrated in Hadoop-Common-trunk-Commit #1366 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1366/]) HDFS-1580. Add interface for generic Write Ahead Logging mechanisms. Contributed by Ivan Kelly. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1210602 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Assignee: Jitendra Nath Pandey Fix For: HA branch (HDFS-1623), 0.24.0 Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13163033#comment-13163033 ] Hudson commented on HDFS-1580: -- Integrated in Hadoop-Mapreduce-trunk-Commit #1391 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1391/]) HDFS-1580. Add interface for generic Write Ahead Logging mechanisms. Contributed by Ivan Kelly. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1210602 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Assignee: Jitendra Nath Pandey Fix For: HA branch (HDFS-1623), 0.24.0 Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13163164#comment-13163164 ] Hudson commented on HDFS-1580: -- Integrated in Hadoop-Hdfs-trunk-Commit #1439 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1439/]) HDFS-1580. Add interface for generic Write Ahead Logging mechanisms. Contributed by Ivan Kelly. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1210602 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Assignee: Jitendra Nath Pandey Fix For: HA branch (HDFS-1623), 0.24.0 Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13163281#comment-13163281 ] Hudson commented on HDFS-1580: -- Integrated in Hadoop-Hdfs-trunk-Commit #1443 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1443/]) HDFS-1580. Committing the test case, it was missed in earlier commit. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1210749 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGenericJournalConf.java Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Assignee: Jitendra Nath Pandey Fix For: HA branch (HDFS-1623), 0.24.0 Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13163284#comment-13163284 ] Hudson commented on HDFS-1580: -- Integrated in Hadoop-Common-trunk-Commit #1371 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1371/]) HDFS-1580. Committing the test case, it was missed in earlier commit. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1210749 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGenericJournalConf.java Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Assignee: Jitendra Nath Pandey Fix For: HA branch (HDFS-1623), 0.24.0 Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13163288#comment-13163288 ] Hudson commented on HDFS-1580: -- Integrated in Hadoop-Mapreduce-trunk-Commit #1395 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1395/]) HDFS-1580. Committing the test case, it was missed in earlier commit. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1210749 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGenericJournalConf.java Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Assignee: Jitendra Nath Pandey Fix For: HA branch (HDFS-1623), 0.24.0 Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13161547#comment-13161547 ] Hadoop QA commented on HDFS-1580: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12505861/HDFS-1580.diff against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1628//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1628//console This message is automatically generated. Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Assignee: Jitendra Nath Pandey Fix For: HA branch (HDFS-1623), 0.24.0 Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13162029#comment-13162029 ] Jitendra Nath Pandey commented on HDFS-1580: +1. Patch looks good to me. Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Assignee: Jitendra Nath Pandey Fix For: HA branch (HDFS-1623), 0.24.0 Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13160860#comment-13160860 ] Hadoop QA commented on HDFS-1580: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12505753/HDFS-1580.diff against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestHDFSServerPorts org.apache.hadoop.hdfs.TestBlocksScheduledCounter org.apache.hadoop.hdfs.TestAppendDifferentChecksum +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1622//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1622//console This message is automatically generated. Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Assignee: Jitendra Nath Pandey Fix For: HA branch (HDFS-1623), 0.24.0 Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13161016#comment-13161016 ] Todd Lipcon commented on HDFS-1580: --- Looks like TestDecommission timed out... is that timing out on trunk? Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Assignee: Jitendra Nath Pandey Fix For: HA branch (HDFS-1623), 0.24.0 Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[
https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13161234#comment-13161234
]
Ivan Kelly commented on HDFS-1580:
--
I couldn't find any, but multiple reruns of it and the surrounding tests didnt
have any fails :/
{code}
Running org.apache.hadoop.hdfs.TestMissingBlocksAlert
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.422 sec
Running org.apache.hadoop.hdfs.TestDecommission
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 172.954 sec
Running org.apache.hadoop.hdfs.TestAppendDifferentChecksum
Tests run: 3, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 7.52 sec
Running org.apache.hadoop.hdfs.TestBlocksScheduledCounter
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.017 sec
Running org.apache.hadoop.hdfs.server.namenode.TestGenericJournalConf
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.41 sec
{code}
Im going to reupload the patch to kick off tests again.
Add interface for generic Write Ahead Logging mechanisms
Key: HDFS-1580
URL: https://issues.apache.org/jira/browse/HDFS-1580
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Ivan Kelly
Assignee: Jitendra Nath Pandey
Fix For: HA branch (HDFS-1623), 0.24.0
Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf,
EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff,
HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff,
HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff,
HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf,
generic_wal_iface.pdf, generic_wal_iface.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13161254#comment-13161254 ] Hadoop QA commented on HDFS-1580: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12505819/HDFS-1580.diff against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestDistributedFileSystem org.apache.hadoop.hdfs.TestConnCache org.apache.hadoop.hdfs.server.namenode.TestParallelImageWrite org.apache.hadoop.hdfs.TestSafeMode org.apache.hadoop.hdfs.server.datanode.TestDataNodeExit org.apache.hadoop.hdfs.server.namenode.TestHDFSConcat org.apache.hadoop.hdfs.TestFileAppend org.apache.hadoop.hdfs.TestDatanodeDeath org.apache.hadoop.hdfs.server.namenode.TestFileLimit org.apache.hadoop.hdfs.TestLease org.apache.hadoop.hdfs.TestShortCircuitLocalRead org.apache.hadoop.hdfs.TestDFSRename org.apache.hadoop.hdfs.TestPipelines org.apache.hadoop.hdfs.TestSetrepDecreasing org.apache.hadoop.hdfs.TestLeaseRecovery2 org.apache.hadoop.hdfs.TestFileAppend2 org.apache.hadoop.security.TestRefreshUserMappings org.apache.hadoop.hdfs.server.datanode.TestRefreshNamenodes org.apache.hadoop.hdfs.TestHFlush org.apache.hadoop.hdfs.TestLeaseRecovery org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract org.apache.hadoop.fs.loadGenerator.TestLoadGenerator org.apache.hadoop.hdfs.server.datanode.TestMulitipleNNDataBlockScanner org.apache.hadoop.hdfs.server.blockmanagement.TestNodeCount org.apache.hadoop.fs.TestGlobPaths org.apache.hadoop.hdfs.server.namenode.TestBlockUnderConstruction org.apache.hadoop.fs.TestResolveHdfsSymlink org.apache.hadoop.hdfs.TestFileCreationNamenodeRestart org.apache.hadoop.hdfs.server.namenode.TestListCorruptFileBlocks org.apache.hadoop.hdfs.server.namenode.TestCheckPointForSecurityTokens org.apache.hadoop.hdfs.TestDfsOverAvroRpc org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicy org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool org.apache.hadoop.hdfs.server.namenode.TestStorageRestore org.apache.hadoop.hdfs.server.namenode.TestNameNodeResourceChecker org.apache.hadoop.hdfs.TestAbandonBlock org.apache.hadoop.fs.TestUrlStreamHandler org.apache.hadoop.hdfs.server.namenode.TestTransferFsImage org.apache.hadoop.hdfs.TestBlockMissingException org.apache.hadoop.fs.viewfs.TestViewFsFileStatusHdfs org.apache.hadoop.hdfs.TestClientReportBadBlock org.apache.hadoop.hdfs.TestDFSShell org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure org.apache.hadoop.hdfs.server.namenode.TestProcessCorruptBlocks org.apache.hadoop.hdfs.TestFileCreation org.apache.hadoop.fs.TestFcHdfsPermission org.apache.hadoop.hdfs.TestCrcCorruption org.apache.hadoop.hdfs.server.namenode.TestNamenodeCapacityReport org.apache.hadoop.hdfs.TestListFilesInFileContext org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement org.apache.hadoop.hdfs.server.namenode.TestNameEditsConfigs org.apache.hadoop.hdfs.TestListPathServlet org.apache.hadoop.hdfs.TestMultiThreadedHflush org.apache.hadoop.hdfs.TestDFSFinalize org.apache.hadoop.hdfs.TestLocalDFS org.apache.hadoop.hdfs.server.namenode.TestBackupNode org.apache.hadoop.hdfs.TestSeekBug
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13160682#comment-13160682 ] Jitendra Nath Pandey commented on HDFS-1580: # Please annotate the public classes with InterfaceAudience.Private and also for stability. # Instead of using file for the file scheme, please declare it as a constant. Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Assignee: Jitendra Nath Pandey Fix For: HA branch (HDFS-1623), 0.24.0 Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143044#comment-13143044 ] jirapos...@reviews.apache.org commented on HDFS-1580: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2672/ --- (Updated 2011-11-03 11:25:09.526158) Review request for hadoop-hdfs. Summary --- This is the final piece to allow the loading of custom implementations of JournalManager. There is another change HDFS-2334 which adds closeable to JournalManager, but that may not be absolutely necessary for all journal types. (it is for bookkeeper) There's 2 changes: 1) I've changes the interfaces(JournalManager, EditLogInputStream EditLogOutputStream) so that they can be implemented outside of the org.apache.hadoop.hdfs.server.namenode. 2) Pluggable creation of journal managers. When FSEditLog is creating JournalManagers from dfs.namenode.edits.dir, and it encounters a URI with a schema different to file it loads the name of the implementing class from dfs.namenode.edits.journal-plugin.schema. This class must implement JournalManager and have a constructor which takes (Configuration, URI). This addresses bug HDFS-1580. http://issues.apache.org/jira/browse/HDFS-1580 Diffs (updated) - hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java 7630335 hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupInputStream.java 974697d hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupOutputStream.java 067990d hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java 9db7f8a hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java 4780d04 hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogInputStream.java c6f8505 hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogOutputStream.java 8681837 hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java f80f863 hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java 991fd08 hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java 3adb439 hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalManager.java 348e3ef hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java 45b5714 hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java a7fa7fb hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java 4d7cfd8 hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGenericJournalConf.java PRE-CREATION Diff: https://reviews.apache.org/r/2672/diff Testing --- Thanks, Ivan Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Assignee: Jitendra Nath Pandey Fix For: HA branch (HDFS-1623), 0.24.0 Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143166#comment-13143166 ] Hadoop QA commented on HDFS-1580: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12502129/HDFS-1580.diff against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestAbandonBlock +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1530//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1530//console This message is automatically generated. Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Assignee: Jitendra Nath Pandey Fix For: HA branch (HDFS-1623), 0.24.0 Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143541#comment-13143541 ] Ivan Kelly commented on HDFS-1580: -- Local reruns of TestAbandonBlock passes cleanly. Seems flakey in general, also failed on Build #1527, Build #1527, Build #1527 Build #1511. I didn't look back any further. Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Assignee: Jitendra Nath Pandey Fix For: HA branch (HDFS-1623), 0.24.0 Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13142188#comment-13142188 ] jirapos...@reviews.apache.org commented on HDFS-1580: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2672/ --- Review request for hadoop-hdfs. Summary --- This is the final piece to allow the loading of custom implementations of JournalManager. There is another change HDFS-2334 which adds closeable to JournalManager, but that may not be absolutely necessary for all journal types. (it is for bookkeeper) There's 2 changes: 1) I've changes the interfaces(JournalManager, EditLogInputStream EditLogOutputStream) so that they can be implemented outside of the org.apache.hadoop.hdfs.server.namenode. 2) Pluggable creation of journal managers. When FSEditLog is creating JournalManagers from dfs.namenode.edits.dir, and it encounters a URI with a schema different to file it loads the name of the implementing class from dfs.namenode.edits.journal-plugin.schema. This class must implement JournalManager and have a constructor which takes (Configuration, URI). This addresses bug HDFS-1580. http://issues.apache.org/jira/browse/HDFS-1580 Diffs - hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java dd39676 hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupInputStream.java 974697d hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupOutputStream.java 067990d hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java 9db7f8a hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java 4780d04 hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogInputStream.java c6f8505 hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogOutputStream.java 8681837 hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java f80f863 hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java 991fd08 hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java 3adb439 hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalManager.java 348e3ef hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java 45b5714 hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java a7fa7fb hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java 4d7cfd8 hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGenericJournalConf.java PRE-CREATION Diff: https://reviews.apache.org/r/2672/diff Testing --- Thanks, Ivan Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Assignee: Jitendra Nath Pandey Fix For: HA branch (HDFS-1623), 0.24.0 Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13142230#comment-13142230 ] Hadoop QA commented on HDFS-1580: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501965/HDFS-1580.diff against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestFileAppend2 org.apache.hadoop.hdfs.server.datanode.TestMulitipleNNDataBlockScanner org.apache.hadoop.hdfs.TestBalancerBandwidth +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1519//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1519//console This message is automatically generated. Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Assignee: Jitendra Nath Pandey Fix For: HA branch (HDFS-1623), 0.24.0 Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13142317#comment-13142317 ] jirapos...@reviews.apache.org commented on HDFS-1580: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2672/#review3014 --- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java https://reviews.apache.org/r/2672/#comment6727 we use _PREFIX instead of _BASE elsewhere for key prefixes hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java https://reviews.apache.org/r/2672/#comment6728 why not just use conf.getClass here and return a Class? And throw the exception right here instead of returning null and throwing below hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java https://reviews.apache.org/r/2672/#comment6729 this is the wrong layer - better to filter for file:// URLs where this is called, I think. hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGenericJournalConf.java https://reviews.apache.org/r/2672/#comment6730 no need to have any datanodes for any of these tests - will run faster without. hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGenericJournalConf.java https://reviews.apache.org/r/2672/#comment6731 our convention is to use american spelling (initialized) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGenericJournalConf.java https://reviews.apache.org/r/2672/#comment6732 our style is to not have multiple classes per .java file unless they're inner classes. You can make this a static inner class of the test. hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGenericJournalConf.java https://reviews.apache.org/r/2672/#comment6733 just return Mockito.mock(EditLogOutputStream.class) and you don't need to have the whole implementation below - Todd On 2011-11-02 14:33:47, Ivan Kelly wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2672/ bq. --- bq. bq. (Updated 2011-11-02 14:33:47) bq. bq. bq. Review request for hadoop-hdfs. bq. bq. bq. Summary bq. --- bq. bq. This is the final piece to allow the loading of custom implementations of JournalManager. There is another change HDFS-2334 which adds closeable to JournalManager, but that may not be absolutely necessary for all journal types. (it is for bookkeeper) bq. bq. There's 2 changes: bq. 1) I've changes the interfaces(JournalManager, EditLogInputStream EditLogOutputStream) so that they can be implemented outside of the org.apache.hadoop.hdfs.server.namenode. bq. bq. 2) Pluggable creation of journal managers. bq. When FSEditLog is creating JournalManagers from dfs.namenode.edits.dir, and it encounters a URI with a schema different to file it loads the name of the implementing class from dfs.namenode.edits.journal-plugin.schema. This class must implement JournalManager and have a constructor which takes (Configuration, URI). bq. bq. bq. This addresses bug HDFS-1580. bq. http://issues.apache.org/jira/browse/HDFS-1580 bq. bq. bq. Diffs bq. - bq. bq. hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java dd39676 bq. hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupInputStream.java 974697d bq. hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupOutputStream.java 067990d bq. hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java 9db7f8a bq. hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java 4780d04 bq. hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogInputStream.java c6f8505 bq. hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogOutputStream.java 8681837 bq. hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java f80f863 bq. hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java 991fd08 bq. hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java 3adb439
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13133686#comment-13133686 ] Hadoop QA commented on HDFS-1580: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12500367/HDFS-1580.diff against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.server.namenode.TestBackupNode org.apache.hadoop.hdfs.TestFileAppend2 org.apache.hadoop.hdfs.TestBalancerBandwidth org.apache.hadoop.hdfs.TestRestartDFS org.apache.hadoop.hdfs.TestDistributedFileSystem org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1422//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1422//console This message is automatically generated. Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Assignee: Jitendra Nath Pandey Fix For: HA branch (HDFS-1623), 0.24.0 Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13132937#comment-13132937 ] Hadoop QA commented on HDFS-1580: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12500207/HDFS-1580.diff against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestDistributedFileSystem org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool org.apache.hadoop.hdfs.TestAbandonBlock org.apache.hadoop.hdfs.server.namenode.TestBackupNode org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer org.apache.hadoop.hdfs.TestRestartDFS +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1406//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1406//console This message is automatically generated. Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Assignee: Jitendra Nath Pandey Fix For: HA branch (HDFS-1623), 0.24.0 Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13061358#comment-13061358 ] Ivan Kelly commented on HDFS-1580: -- @dhruba 1. I agree. See 21st March comments on this JIRA. Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Fix For: Edit log branch (HDFS-1073) Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13058708#comment-13058708 ] dhruba borthakur commented on HDFS-1580: 1. it would be nice to get rid of mark() method 2. possibly rename InputStream.readNext() to InputStream.seekNext() just to make it clear that we are operating on some sort of an underlying stream Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Fix For: Edit log branch (HDFS-1073) Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13042313#comment-13042313 ] Ivan Kelly commented on HDFS-1580: -- @Jitendra (1) should work for checkpointing, as if you journal A has more edits than journal B while counting the in_progress file, it will have more or an equal number not counting the in_progress file. More in the case that B has gaps in which case it throws an exception, equal otherwise. So we finalise inprogress when we open a write and spot an inprogress file. I guess this should only happen on startup after a crash. The writer shouldn't finalise an inprogress if something else is writing to it. We have nothing to prevent this now, but if this is happening, your system is broken. Fencing could be implemented later to explicitly exclude this possibility. Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Fix For: Edit log branch (HDFS-1073) Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13041621#comment-13041621 ] Sanjay Radia commented on HDFS-1580: Finalizing the edits at initialization makes sense. It would require reading the edits but this should not be bad if we roll the edits at reasonable intervals. One possibility is for this to be part of init() that returns the highest txId. Jitendra had mentioned to me why he preferred the getNumTransaction(sinceTx) but I forget the reason. Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Fix For: Edit log branch (HDFS-1073) Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13041627#comment-13041627 ] Ivan Kelly commented on HDFS-1580: -- Finalizing in getNumTransactions is a bit messy, because its difficult to tell if a inprogress file is actually inprogress or the result of a crashed namenode. I tried it in HDFS-2018, but ended up creating an explicit recoverUnclosedStreams() call, which only gets called on loading (getNumTransactions is called in other places, such as checkpointing). It would be possible to hide this inside the JournalManager, but then we'd have to make some sort of locking mechanism to ensure an inprogress stream doesn't get finalised while it is writing. This is simple for bookkeeper, but for filesystems we'd need some sort of lease mechanism. Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Fix For: Edit log branch (HDFS-1073) Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13041637#comment-13041637 ] Todd Lipcon commented on HDFS-1580: --- I think we need that kind of recover interface anyway to do IO fencing on shared storage, when it comes to HA. Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Fix For: Edit log branch (HDFS-1073) Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[
https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039766#comment-13039766
]
Ivan Kelly commented on HDFS-1580:
--
{quote}Would it be better instead to ask getHighestTxid()?{quote}
#getNumberOfTransactions() is useful for deciding which journal to use.
Presumably sinceTxnId refers to the last transaction Id loaded from the image.
So sinceTxnId + getNumberOfTransactions() gives you the maximum prefix of the
NameNode history a particular journal can give you.
#getHighestTxid() would, however, give you the highest transaction id without
taking into consideration that gaps may exist.
Add interface for generic Write Ahead Logging mechanisms
Key: HDFS-1580
URL: https://issues.apache.org/jira/browse/HDFS-1580
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Ivan Kelly
Fix For: Edit log branch (HDFS-1073)
Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf,
HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff,
generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf,
generic_wal_iface.txt
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[
https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039769#comment-13039769
]
Ivan Kelly commented on HDFS-1580:
--
I've been working on implementing the design for HDFS-1580 on top of the
HDFS-1073 branch and have run into a problem with
#getNumberOfTransactions().
Specifically, I've been working on the input code in FSImage:
{code}
protected boolean loadEdits(JournalManager journal) throws IOException {
LOG.debug(About to load edits:\n + journal);
FSEditLogLoader loader = new FSEditLogLoader(namesystem);
long startingTxId = storage.getMostRecentCheckpointTxId() + 1;
int numLoaded = 0;
// Load latest edits
long numTransactionsToLoad =
journal.getNumberOfTransactions(startingTxId);
while (numLoaded numTransactionsToLoad) {
EditLogInputStream editIn = journal.getInputStream(startingTxId);
LOG.debug(Reading + editIn + expecting start txid # +
startingTxId);
int thisNumLoaded = loader.loadFSEdits(editIn, startingTxId);
startingTxId += thisNumLoaded;
numLoaded += thisNumLoaded;
editIn.close();
}
// update the counts
getFSNamesystem().dir.updateCountForINodeWithQuota();
// update the txid for the edit log
editLog.setNextTxId(storage.getMostRecentCheckpointTxId() +
numLoaded + 1);
// If we loaded any edits, need to save.
return numLoaded 0;
}
{code}
The load is in a loop now, as the output is still in LogSegment form,
but even in a single stream implementation getNumberOfTransactions()
presents a problem.
The problem is that sometimes it is impossible to return a number for
getNumberOfTransactions(). This case is when NameNode has crashed in the
middle of an edit log. The editlog is named edits_inprogress_N where N
is the first transaction id in the edit log. But since NN crashed, we
dont know the last transaction so we cannot possibly return the number
of transactions in the journal without scanning the file from the start.
Without getNumberOfTransactions() its difficult to choose which journal
has the most edits. HDFS-1073 uses the number of bytes in the file, but
this doesn't feel very safe for anything that isn't a file. Whats more,
if the start transaction of two journal snippets are out of sync, then
it becomes impossible to choose which journal has the most transactions
using just filesize(This is an argument for log segments).
The simplest solution I see is to actually scan the _inprogress file
from the start to get the last transaction written. As this should only
happen in NameNode crashes, the delay for doing this shouldn't be
prohibitive.
Add interface for generic Write Ahead Logging mechanisms
Key: HDFS-1580
URL: https://issues.apache.org/jira/browse/HDFS-1580
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Ivan Kelly
Fix For: Edit log branch (HDFS-1073)
Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf,
HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff,
generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf,
generic_wal_iface.txt
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039775#comment-13039775 ] Ivan Kelly commented on HDFS-1580: -- I've created HDFS-2003 to split FSEditLogLoader into reading and memory state building. This will make it easier to count the number of transactions in a journal. Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Fix For: Edit log branch (HDFS-1073) Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039478#comment-13039478 ] Sanjay Radia commented on HDFS-1580: Please add some text doc to motivate the Journal#getNumberOfTransactions(long 
sinceTxnId)
. From what I understand this method gives the # of transactions since tX in a particular storage dir. Would it be better instead to ask getHighestTxid()? Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Fix For: Edit log branch (HDFS-1073) Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[
https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13026675#comment-13026675
]
Todd Lipcon commented on HDFS-1580:
---
Hi Jitendra. Here are some thoughts on your latest document:
- While I appreciate that this work will probably make snapshots a little
easier down the road, it's by far not the most difficult part of supporting
snapshots, nor is it really the goal we're trying to address. So I think it's
premature to mention snapshots in the design.
- The concept of layout version I think has been overloaded way too much. We
currently use a single version number to indicate (a) the file and
serialziation format for image dumps, (b) the file and serialization format for
edit logs, and (c) the actual layout of files within the {{current/}}
directory. I would like to advocate splitting this out into
IMAGE_FORMAT_VERSION, EDITS_FORMAT_VERSION, and LAYOUT_VERSION. To be clear,
this jira is mostly concerned with what I would call EDITS_FORMAT_VERSION (e.g.
the way in which we turn a mkdirs into bytes). Do you agree with this
interpretation?
- The idea of a {{purgeTransactions}} call makes sense -- after a checkpoint
has been uploaded for txid N, we don't need edits prior to N. However, there
are some policies that make sense to me like keep edits for at least a week.
Would you assume these retention policies would be the responsibility of the
edit log implementation? ie that, even if told to purge transactions older than
txid N, it might keep them around for some time, or take care of archiving them
to a NAS/HDFS?
- For the {{getInputStream}} call, is there any restriction on valid values of
{{sinceTxId}} that it be on any kind of boundary? e.g that it must correspond
to a mark call? See more about this below regarding the idea of log segments
- I don't entirely understand the usage of the {{setVersion}} call. When would
the version of a log change mid-stream?
- I'm not entirely clear on mark as well. The semantics described in the
Discussion section are what I would normally call {{sync}}, but in other
parts of the document it's described as a {{roll}} equivalent. If it's not
sync, then we're missing sync altogether, and that implies that each {{write}}
call will have to sync on its own, thus breaking group commit. I think we
should maintain the existing buffering/syncing calls {{write}},
{{setReadyToFlush}}, and {{flushAndSync}}.
- The {{EditLogInputStream}} interface is strange - it's called InputStream but
doesn't follow a normal InputStream API. It's something sort of like an
Iterator, but also doesn't implement that interface. Could we add a wrapper
class {{EditTransaction}}, and make EditLogInputStream an
InterableEditTransaction? EditTransaction would then take the {{getTxnId}}
call.
- The API {{getTxn}} shouldn't return {{byte[]}} since that implies an extra
buffer copy to get a transaction into its own array. Instead it should be able
to point into an existing byte array. Alternatively, the input stream could
continue to implement InputStream so we can use the existing editlog loading
code.
As I've proposed over in some other JIRAs, I think we should do away with the
{{roll}} call, and instead make the concept of _log segments_ a first class
citizen. In the file-based storage case, a log segment is an individual file.
In the BK case, it may be that a log segment is a ledger (I don't know BK's API
well).
Thus, rolling the logs becomes a sequence like:
{code}
endCurrentLogSegment();
long nextTxId = getLastWrittenTxId() + 1;
LOG.info(Rolling edit logs. Next txid after roll will be + nextTxId);
startLogSegment(nextTxId);
{code}
where {{endCurrentLogSegment}} closes off the current segment across all
journals, and {{startLogSegment}} starts a new output stream across all
journals.
The advantages I see of this approach are:
- elsewhere we have discussed that we want to keep the property that logs
always roll together across all parts of the system, and thus that the storage
directories have parallel contents with identical names and identical file
contents. It's possible to achieve this with just the roll API, but it becomes
more obvious how to do it with the segment concept. As one example, consider
what happens when one journal fails (eg due to an NFS mount going down
temporarily). While it's down, we don't write txns to this journal. But, after
some time we may notice that the mount is available again. Rather than just
calling {{roll}} here, it makes sense to be explicit that we're starting a new
segment, and be explicit about the starting txid of that new segment.
- We generally want the property that, while saving a namespace or in safe
mode, we don't accept edits. Thus, it would be nice to have the edit log
actually be closed during this operation. Splitting {{roll}} into a
{{endCurrent}} and
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13026742#comment-13026742 ] Jitendra Nath Pandey commented on HDFS-1580: - The design doesn't go in any detail regarding snapshots concurring to your view. However, I mentioned about it because it is one of the requirements we will have to address eventually. - This jira doesn't change any semantics related to the layout version. The version is a piece of metadata that needs to be stored with edit logs so that namenode can understand and load edit logs. I am open to making it a byte array instead of just an integer so that namenode can store any metadata it wants to store, which is relevant for understanding the edit logs. I agree that version is a little overloaded but that can be addressed in a different jira. - I think retention policy for edit logs should be namenode's responsibility, because retention of edit logs will be closely tied with retention of old checkpoint images. If namenode has called purgeTransactions it should never ask for older transaction ids. - mark means that the last written transaction is available for reading including all previous transactions. sinceTxnId in getInputStream can be any transaction Id before the last call of mark or close of the output stream. Apart from that, sinceTxnId doesn't assume any boundary. - The motivation for mark method was that BK has this limitation that open ledgers cannot be read, mark will give a cue to a BK implementation that the current ledger should be made available for reading. If an implementation doesn't have this limitation it can just ignore mark, that is why I didn't call it roll. That also explains that it is different from sync. - I assumed that a write also syncs, because in most operations we sync immediately after writing the log, and in this design we are writing the entire transaction as a unit. Management of buffers and flush, should be the responsibility of the implementation. - In EditLogInputStream, I think we can rename next to readNext, it will look less like iterator. One way to avoid extra array copy would be that readNext() reads the version and txnId and synchronizes the underlying inputstream to the begining of transaction record and then getTxn can directly return the underlying inputstream for reading the transaction bytes. Does that make sense? LogSegements: LogSegments gets rid of roll method but exposes the underlying units of storage to the namenode which I don't think is required. .. elsewhere we have discussed that we want to keep the property that logs always roll together across all parts of the system. Do we really want this property? Isn't it better that we don't expose any boundaries between transactions to the namenode? We generally want the property that, while saving a namespace or in safe mode, we don't accept edits. This can be achieved by just closing the EditLogOutputStream. Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Fix For: Edit log branch (HDFS-1073) Attachments: EditlogInterface.1.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[
https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13026772#comment-13026772
]
Todd Lipcon commented on HDFS-1580:
---
bq. which is relevant for understanding the edit logs. I agree that version is
a little overloaded but that can be addressed in a different jira
Agreed that's a separate JIRA -- I just wanted to clarify that the version
you're talking about here is the edits log serialization format version
rather than something about actual layout.
bq. If namenode has called purgeTransactions it should never ask for older
transaction ids
Fair enough.
bq. Apart from that, sinceTxnId doesn't assume any boundary
I think that will really complicate things like edits transfer in the 2NN. In
the file-based storage there's no clean way to seek to a particular transaction
ID, meaning we'd have to add in this facility into EditLogInputStream, etc.
That's a lot of complexity for little benefit that I can see.
bq. The motivation for mark method was that BK has this limitation that open
ledgers cannot be read, mark will give a cue to a BK implementation that the
current ledger should be made available for reading
This seems like a somewhat serious flaw. If we anticipate using BK for HA, I
was under the impression that the hot backup would be following along on the
edits as they're written into BK. What you're saying here implies that the
primary NN would have to be rolling its logs every few seconds if you want the
standby to be truly hot.
bq. If an implementation doesn't have this limitation it can just ignore mark,
that is why I didn't call it roll
Another way of doing this is to say that, if an implementation _does_ have this
limitation, it can choose to mark whenever it likes. No?
bq. I assumed that a write also syncs, because in most operations we sync
immediately after writing the log, and in this design we are writing the entire
transaction as a unit.
In fact this is not at all how the current design works. Most operations write
the edit to the log while holding the FSN lock (to ensure serialized order
between ops) and then drop the FSN lock to sync. This allows group commit and
is crucial for reasonable throughput.
bq. Management of buffers and flush, should be the responsibility of the
implementation.
But flush needs to be coordinated as a separate action from writing in order to
achieve lock release and group commit.
bq. readNext() reads the version and txnId and synchronizes the underlying
inputstream to the begining of transaction record and then getTxn can directly
return the underlying inputstream for reading the transaction bytes
Yep, that makes sense.
bq. LogSegments gets rid of roll method but exposes the underlying units of
storage to the namenode which I don't think is required
It's not absolutely required in the theoretical sense, but in the sense that
we'd like to keep the code as simple as possible, I think it helps that goal.
For example, edit log transfer right now is based around the concept of
discrete files which can be entirely fetched, with an associated md5sum. If we
have to support fetching arbitrary ranges of transactions, these safety checks
become more difficult to implement. And, we need to split the file transfer
code into two different code paths, one for files (fsimage) and another for
edits (arbitrary transaction ranges)
bq. Do we really want this property? Isn't it better that we don't expose any
boundaries between transactions to the namenode?
Yes, this property is very useful for operations. Refer to the discussion on
HDFS-1073 about this property. The fact that I can run md5sum
/data/{1..4}/dfs/name/current/* and verify that the files are all identical
gives me great peace of mind.
Add interface for generic Write Ahead Logging mechanisms
Key: HDFS-1580
URL: https://issues.apache.org/jira/browse/HDFS-1580
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Ivan Kelly
Fix For: Edit log branch (HDFS-1073)
Attachments: EditlogInterface.1.pdf, HDFS-1580+1521.diff,
HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf,
generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[
https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13026802#comment-13026802
]
Jitendra Nath Pandey commented on HDFS-1580:
In the file-based storage there's no clean way to seek to a particular
transaction ID
A savenamespace will be preceded by a call to mark (like current roll). A file
implementation can close the current file and start a new file at that point.
Therefore in usual operation, when a namenode starts up it will load an fsimage
and requests to read transactions after that point, it will most likely find a
file that starts from next transaction id.
Alternatively, a file implementation can ignore mark and close a file every
10 transactions. Now if it has to seek to 5th transaction it can just
read and ignore previous transactions. Since transaction files will be read
only for checkpointing or at namenode startup or by backup at failover, it is
not very expensive. In a recent measurement we found that namenode could load
1.4M transactions in 27 seconds.
Also if we store edit logs to book keeper, 2NN can read from book keeper and
there won't be a need for edit transfer, that is another attraction for using
book keeper.
This seems like a somewhat serious flaw. If we anticipate using BK for HA..
Agreed that the backup will lag behind the primary but when failover happens
it can quickly read the additional transactions before declaring itself active.
Won't that be an acceptable delay? There is some discussion on this in
ZOOKEEPER-1016.
Another way of doing this is to say that, if an implementation does have this
limitation, it can choose to mark whenever it likes.
That is correct, however mark will be useful in the interface to be called
before a savenamespace.
Most operations write the edit to the log while holding the FSN lock (to
ensure serialized order between ops) and then drop the FSN lock to sync
Good catch! A sync method is needed in EditLogOutputStream to be called after
releasing the lock.
edit log transfer right now is based around the concept of discrete files
which can be entirely fetched, with an associated md5sum
I think it should be File storage implemenation's responsibility to keep an
md5sum with every file, therefore the safety check while transferring files can
still be supported.
This interface doesn't manage transfer of edit logs. It only talks about
reading/writing the transactions from/to a storage. When 2NN wants to do a
checkpoint, it will download the files from primary, it will then get an
EditLogInputStream object using this interface for the edit log files, and read
the transactions.
For Book-keeper storage, transfer will not be required.
md5sum /data/{1..4}/dfs/name/current/
If we use a system like Book-keeper, we won't have the ability to perform
this sanity check anyway. For different file storages, this ability will
continue to exist, because a) mark will be called for all journal instances at
the same time, and b) even if file storage implementation closes file every
10 transactions it will be consistent for all files.
Refer to the discussion on HDFS-1073 about this property.
Sure, I will look at it.
Add interface for generic Write Ahead Logging mechanisms
Key: HDFS-1580
URL: https://issues.apache.org/jira/browse/HDFS-1580
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Ivan Kelly
Fix For: Edit log branch (HDFS-1073)
Attachments: EditlogInterface.1.pdf, HDFS-1580+1521.diff,
HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf,
generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13026852#comment-13026852 ] dhruba borthakur commented on HDFS-1580: Nice document. can you pl describe the error situations in more detail: 1. what happens when the write to the outputstream throws an expcetion? can the client continue to write new transactions to the stream? 2. what happens when the InputStream.read() encounters a partial transaction? 3. from the InputStream, is it possible to know how many transactions are available (without actually reading till the end of the stream)? Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Fix For: Edit log branch (HDFS-1073) Attachments: EditlogInterface.1.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13015162#comment-13015162 ] Ivan Kelly commented on HDFS-1580: -- Applies on top of 1073 branch. Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Attachments: HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13012898#comment-13012898 ] Ivan Kelly commented on HDFS-1580: -- I'd like to keep the pluggable bit (JournalFactory, or JournalWriterJournalReader) to one interface for ease of configuration. Otherwise when a user specifies a custom journal type in their configuration, they'll have to specify the class for the Writer and for the Reader. The EditLogOutputStream i proposed won't coexist with the current EditLogOutputStream, it will replace it. We need a single interface for writing the edit log, so that FSEditLog can loop through all the streams and treat them the same. I like the idea of TxnRecord. Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Attachments: HDFS-1580+1521.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13013534#comment-13013534 ] Todd Lipcon commented on HDFS-1580: --- I'm not sure about TxnRecord - if not done carefully it will cause an extra memory copy on all the writes, which isn't free. I like modularity, but we will have to carefully benchmark if we make that change. Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Attachments: HDFS-1580+1521.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[
https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13012817#comment-13012817
]
Jitendra Nath Pandey commented on HDFS-1580:
Ivan,
I am thinking of something like following code. I am using name
TxnLogOutputStream instead of EditLogOutputStream because the later confuses
with existing EditLogOutputStream in Hdfs. This undoes some of my previous
suggestions e.g. regarding putting transaction id in the URI. It seems better
that the Journal is viewed as a single input or output stream and URI just
identifies the namespace.
I think we should use TxnRecord object which encapsulate a transaction as a
byte-array instead of using Writables, its better to keep interface unaware of
serialization protocol.
{code}
abstract class JournalWriter {
//The URI identifies the journal location namespace
JournalWriter(URI u);
//Initializes the layoutVersion
void format(int layoutVersion);
//After this call, any subsequent transactions should
//map to newLayoutVersion, to be used in upgrade or rollback.
void setLayoutVersion(int newLayoutVersion);
TxnLogOutputStream getOutputStream();
//Deletes all transactions older than uptoTxnId (inclusive)
void delete(long uptoTxnId);
}
class TxnRecord {
long txnId;
byte[] txn;
}
interface TxnLogOutputStream {
write(long txnId, TxnRecord rec);
void roll();
close();
}
abstract class JournalReader {
JournalReader(URI u);
TxnLogInputStream getInputStream(long sinceTxnId);
}
interface TxnLogInputStream {
TxnRecord next();
//Returns layout version of the next record.
int getLayoutVersion();
void close();
}
{code}
Add interface for generic Write Ahead Logging mechanisms
Key: HDFS-1580
URL: https://issues.apache.org/jira/browse/HDFS-1580
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Ivan Kelly
Attachments: HDFS-1580+1521.diff, HDFS-1580.diff,
generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf,
generic_wal_iface.txt
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13010767#comment-13010767 ] Ivan Kelly commented on HDFS-1580: -- @Todd Ah ok, since rolling has to be exposed, we should expose it fully. Your segment suggestion sounds good. It will require a separate array in FSEditLog though, one for JournalManagers, one for EditLogOutputStreams. I'll have to think about this. Perhaps we could get rid of the journal manager abstraction completely and extend EditLogOutputStream to handle rolling, as rolling is the only operation that changes for output with transactions. Then we could have a separate class that handles stuff like format and getting a list of the logs. I'll think about this more and put up a new design. @Jitendra 1. I think I mentioned the second alternative earlier in the thread, but yes, #transfer will go away. 2. A single bookkeeper setup can be shared by many writers. The more writers you have the slower the reads, as ledgers are interleaved when written to disk. 3. There's two layouts to think about, the storage layout and the data layout. The first will be entirely internal to the JournalManager. The second will need to be saved though. 4. Very good point. 5. Ok 6. Will do Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Attachments: HDFS-1580+1521.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13009138#comment-13009138 ] Ivan Kelly commented on HDFS-1580: -- This depends on whether we want the roll operation exposed to the user of the JournalManager. In the past this was required as roll had to happen to allow a checkpoint to take place. However, once HDFS-1580 goes in, its not strictly necessary because rolling is stateless. The JournalManager could roll in the background without the user of the class having any knowledge of the roll. To the user, the JournalManager is simply something to open, write to and close. Do you think this simplification is worthwhile? I agree that getOutputStream(txid) everywhere is a bit messy. It also doesnt work. If a roll happens while the OutputStream has been given out the editlog may end up writing to two different streams which could mess up transaction ids. This isn't in any way unsolvable, but I need to think about it more. Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Attachments: HDFS-1580+1521.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13009291#comment-13009291 ] Todd Lipcon commented on HDFS-1580: --- I'm not sure I agree with your point that rolling should be a concern of the JournalManager and not of the higher-level frameworks. In HDFS-1073, Sanjay and I had come to the conclusion that we should maintain the invariant that different edits logs should roll at the same time. This makes it possible to take two edits directories (even across the BN and NN) and the files inside should be md5sum-identical to each other -- very important for ops and peace of mind. Perhaps some WAL implementions may want to ignore the roll boundaries, but we at least need coordination between different File logs. Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Attachments: HDFS-1580+1521.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13009512#comment-13009512 ] Jitendra Nath Pandey commented on HDFS-1580: A few comments. 1. JournalManager#transfer() Is this a push method to be executed on primary or a pull from secondary? The mechanism as we have today, which is more like a pull, can be provided by the interface using getLogs and getInputStream methods, which, in my opinion, is sufficient. 2. We could use single BookKeeper installation (or any shared storage solution) for storing edit logs from multiple namenodes (e.g. federation). In that case a secondary/backup should be able to instantiate a JournalManager and request logs for a specific namenode. Similar could be argued about the files on namenodes, where backup or secondary wants to specify the namenode where it wants to download the files from. As I understand the steps to get edit logs at the backup node or secondary would be - Instantiate a JournalManager - specify the source of edit logs. This could be specified in the JournalManager constructor. In case of BK the source would be the set of ledgers for a specific namenode. - getLogs from sinceTransactionId. It returns a list of URIs. - For each uri get the input stream and read the logs. 3. For each edit log we need to store some metadata, particularly the LAYOUT_VERSION. I think it would make sense to provide for that in JournalManager interface. 4. For edit logs do we really need upgrade, rollback or finalizeUpgrade? For edit log files, it seems to me that all that matters is whether the namenode has right version to understand them or load them, which can be determined from LAYOUT_VERSION. 5. I will recommend putting JournalManager and its derived classes in separate packages. 6. Please add javadoc comments on JournalManager class and its methods. Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Attachments: HDFS-1580+1521.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13008886#comment-13008886 ] Todd Lipcon commented on HDFS-1580: --- Hi Ivan, Trying to integrate this with my HDFS-1073 work, running into some things that feel wrong about the JournalManager API... It seems strange that every edit makes a call to getOutputStream(txid) rather than simply calling it once at each roll point. I think it makes the most sense to introduce the idea of a log segment - in the file case it corresponds to a single edit file. In the BK case it might be a ledger. In other cases it might be a no-op. The state machine for logging then becomes something like: - open - beginLogSegment(txid 1): returns an EditLogOutputStream - write(...) 5 times - roll: -- endLogSegment(1, 6) -- startLogSegment(7) - write more data, etc Does this seem to make sense to you? Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Attachments: HDFS-1580+1521.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13003127#comment-13003127 ] Ivan Kelly commented on HDFS-1580: -- To read in a log you pass in a URI to the getInputStream method. For BK, this will allow us to have no explicit transfer stage. Files are a different matter. I haven't thought much about it yet, but file URI do have a space for host. file://host/path. The FileJournalManager could return a URI with the host part set for getLogs. This could be passed to the getInputStream call, which would download it from the remote host if nescessary. So, the steps would be as you outlined except for the iterates over each of the RemoteLogs, and fetches them. Instead you would just iterate over them and load them using the JournalManager.getInputStream method. The implementation of the method would take care of whether the file is remote or local. Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Attachments: HDFS-1580+1521.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13003106#comment-13003106 ] Todd Lipcon commented on HDFS-1580: --- Hi Ivan. I'm trying to figure out how this work will fit in with the log-transfer step between checkpointers and the primary NN. Do you anticipate that these JournalManager implementations will be instantiated on the checkpointer side as well? Or is the transfer method orthogonal to the storage method, and this JIRA is only about the NN side of things? It seems to me they have to be orthogonal concerns, at least initially, if you plan to be able to have two separate types of edit logs on the NN. So, even if the NN is using both BK and files, a 2NN would still use HTTP to fetch logs at checkpoint time. In working on HDFS-1073, I'm thinking the interface for log transfer will look something like: - add a new RPC method to the NN corresponding to your getLogs(long sinceTxId). This would return something like ListRemoteLog, where RemoteLog has some method which takes care of fetching. - for now, when the 2NN wants to checkpoint, it initiates rollEdits() which returns CheckpointSignature. This would be modified to include the lastCheckpointTxId field - the 2NN then issues getLogs(sig.lastCheckpointTxId), iterates over each of the RemoteLogs, and fetches them Does this gel with what you're planning? Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Attachments: HDFS-1580+1521.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[
https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12991314#comment-12991314
]
Ivan Kelly commented on HDFS-1580:
--
{quote}
ShouldWALStreamCustodians control rolling themselves?
As I understand, book-keeper doesn't allow reading logs from open ledger, is
that correct? If that is the case it would be better to first roll and then do
the checkpoint to capture most latest edits.
{quote}
That's true. Therefore there must be a method for having an external entity
call roll. However, for the usecase of rolling logs periodically (to keep under
a certain size), is there any requirement that en external entity knows
anything about it?
{quote}
WALStreamCustodianNotifier
It seems to me this will be implemented only by fsedit log. If that is the case
we could handle errors just by exceptions i.e. if an operation on
WALStreamCustodian interface fails an exception is thrown and fsedit log can
decide to remove the custodian depending on the kind of exception.
{quote}
Also true. In addition to point one, this means we could get rid of
WALStreamCustodianNotifier completely.
{quote}
namenode crash
If the namenode comes back before the znode disappears from zookeeper, the
ledger will be open. In that case, will the FSEditLog.load be able to load
transactions from the open ledger as well?
{quote}
When coming back up, the Bookkeeper WAL implementation will see that there is a
ledger open, but no namenode alive, so it will manually close the ledger. At
that point FSEditLog will be able to read all the updates.
{quote}
interface JournalStream
The document defines this interface but doesn't describe its purpose or
use-case.
{quote}
This interface already exists. I should take these methods out of this design
though, as I think 1073 will be adding something like them anyhow.
{quote}
ListURI getLogs(long sinceTransactionId);
The list returned must be ordered w.r.t the transactions contained. It might be
a good idea to encode the ordering attribute in the url itself, so that the
caller of this method can also verify that order is correct. The uri naming
convention could mimic this aspect from the convention proposed in 1073.
{quote}
Agreed.
{quote}
void startRoll()
void endRoll()
I can only imagine a single roll method, that cuts a log, and starts a new one.
I believe the naming convention or the ordering attribute for the logs should
be controlled by the application and not the storage, therefore the roll method
should take a parameter which becomes part of the log metadata and is used to
order the logs. Again this also depends on how 1073 does it for file logs.
{quote}
Making the ordering controlled by the application and not the storage makes it
hard to encapsulate periodic rolling inside the storage. Rolling requires the
current transaction id (i assume this would be the usual parameter) to open a
new log.
However, what we could do in this case, is remove the roll call completely.
When you want to roll, you just call close on the WALStreamCustodian. Then the
next call to getOutputStream() would open a new stream. getOutputStream() would
require the current transaction id to know where how to call the new stream,
but this shouldn't be a problem.
Add interface for generic Write Ahead Logging mechanisms
Key: HDFS-1580
URL: https://issues.apache.org/jira/browse/HDFS-1580
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Ivan Kelly
Attachments: generic_wal_iface.pdf, generic_wal_iface.pdf,
generic_wal_iface.txt
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12990760#comment-12990760 ] Jitendra Nath Pandey commented on HDFS-1580: ShouldWALStreamCustodians control rolling themselves? As I understand, book-keeper doesn't allow reading logs from open ledger, is that correct? If that is the case it would be better to first roll and then do the checkpoint to capture most latest edits. WALStreamCustodianNotifier It seems to me this will be implemented only by fsedit log. If that is the case we could handle errors just by exceptions i.e. if an operation on WALStreamCustodian interface fails an exception is thrown and fsedit log can decide to remove the custodian depending on the kind of exception. namenode crash If the namenode comes back before the znode disappears from zookeeper, the ledger will be open. In that case, will the FSEditLog.load be able to load transactions from the open ledger as well? interface JournalStream The document defines this interface but doesn't describe its purpose or use-case. ListURI getLogs(long sinceTransactionId); The list returned must be ordered w.r.t the transactions contained. It might be a good idea to encode the ordering attribute in the url itself, so that the caller of this method can also verify that order is correct. The uri naming convention could mimic this aspect from the convention proposed in 1073. void startRoll() void endRoll() I can only imagine a single roll method, that cuts a log, and starts a new one. I believe the naming convention or the ordering attribute for the logs should be controlled by the application and not the storage, therefore the roll method should take a parameter which becomes part of the log metadata and is used to order the logs. Again this also depends on how 1073 does it for file logs. Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Ivan Kelly Attachments: generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12989338#comment-12989338 ] Benjamin Reed commented on HDFS-1580: - looks good ivan. i like section 1. i see a couple of small problems in section 2: 1) you have a constructor in the interface. you should either make that a comment (meaning that is how you will construct those objects using reflection), or you should call it init(...) 2) why is transfer() in WriteAheadLogStreamFactory? shouldn't the transfer happen outside of the logging? other logic should be reading from the local WAL and transferring to the secondary. 3) can you comment a bit more on upgrade(), rollback(), and finalize()? Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Ivan Kelly Attachments: generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12986709#action_12986709 ] Jitendra Nath Pandey commented on HDFS-1580: The interface also needs to have a counterpart of roll-edits method. Currently, for checkpointing, first thing being done is to roll the edit logs i.e. an edits.new is created. As I understand, in hdfs-1073 instead of edits.new the edit files will be numbered. At least FileWriteAheadLog will need to roll to keep edit files from getting too big, even if it is not required for checkpointing. The interface should also provide methods to get all previously rotated edit log files (or ledgers) and also current in-progress edit log file or ledger. As a suggestion, the interface could have a concept of log handles, where each handle uniquely corresponds to single edit log file or ledger. Thus, we could have a method getAllLogs and it will return a list of log-handles. I think ordered handles will fit with hdfs-1073 model (need to confirm). LogHandle can also have some meta data for example first transaction id, whether its current or old etc. Hdfs-1073 is proposing to store first transaction-id in the edit-file name itself, which could be used to populate the log-handle in case of FileWriteAheadLog. The input and output streams should be in the LogHandle, so that any log-file can be read. Log-Handle for older files should not let one create an output stream. A method to purge the editlogs might also be needed, i.e. given a handle remove the corresponding log-file (or ledger). Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Ivan Kelly Attachments: generic_wal_iface.txt -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12986718#action_12986718 ] Todd Lipcon commented on HDFS-1580: --- Above sounds reasonable with respect to 1073. Is HDFS-1557 almost ready to go? (should I take a look at it?) Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Ivan Kelly Attachments: generic_wal_iface.txt -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
