[jira] [Commented] (HDFS-2718) Optimize OP_ADD in edits loading
[ https://issues.apache.org/jira/browse/HDFS-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199670#comment-13199670 ] Hudson commented on HDFS-2718: -- Integrated in Hadoop-Hdfs-trunk #945 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/945/]) HDFS-2718. Optimize OP_ADD in edits loading. Contributed by Konstantin Shvachko. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1239760 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFileUnderConstruction.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestAbandonBlock.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java Optimize OP_ADD in edits loading Key: HDFS-2718 URL: https://issues.apache.org/jira/browse/HDFS-2718 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.22.0, 0.24.0, 1.0.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Fix For: 0.24.0, 0.23.1, 0.22.1 Attachments: editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch During loading the edits journal FSEditLog.loadEditRecords() processes OP_ADD inefficiently. It first removes the existing INodeFile from the directory tree, then adds it back as a regular INodeFile, and then replaces it with INodeFileUnderConstruction if files is not closed. This slows down edits loading. OP_ADD should be done in one shot and retain previously existing data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2718) Optimize OP_ADD in edits loading
[ https://issues.apache.org/jira/browse/HDFS-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199681#comment-13199681 ] Hudson commented on HDFS-2718: -- Integrated in Hadoop-Hdfs-0.23-Build #158 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/158/]) HDFS-2718. Optimize OP_ADD in edits loading. Contributed by Konstantin Shvachko. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1239769 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFileUnderConstruction.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestAbandonBlock.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java Optimize OP_ADD in edits loading Key: HDFS-2718 URL: https://issues.apache.org/jira/browse/HDFS-2718 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.22.0, 0.24.0, 1.0.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Fix For: 0.24.0, 0.23.1, 0.22.1 Attachments: editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch During loading the edits journal FSEditLog.loadEditRecords() processes OP_ADD inefficiently. It first removes the existing INodeFile from the directory tree, then adds it back as a regular INodeFile, and then replaces it with INodeFileUnderConstruction if files is not closed. This slows down edits loading. OP_ADD should be done in one shot and retain previously existing data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2718) Optimize OP_ADD in edits loading
[ https://issues.apache.org/jira/browse/HDFS-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199720#comment-13199720 ] Hudson commented on HDFS-2718: -- Integrated in Hadoop-Mapreduce-0.23-Build #180 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/180/]) HDFS-2718. Optimize OP_ADD in edits loading. Contributed by Konstantin Shvachko. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1239769 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFileUnderConstruction.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestAbandonBlock.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java Optimize OP_ADD in edits loading Key: HDFS-2718 URL: https://issues.apache.org/jira/browse/HDFS-2718 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.22.0, 0.24.0, 1.0.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Fix For: 0.24.0, 0.23.1, 0.22.1 Attachments: editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch During loading the edits journal FSEditLog.loadEditRecords() processes OP_ADD inefficiently. It first removes the existing INodeFile from the directory tree, then adds it back as a regular INodeFile, and then replaces it with INodeFileUnderConstruction if files is not closed. This slows down edits loading. OP_ADD should be done in one shot and retain previously existing data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2718) Optimize OP_ADD in edits loading
[ https://issues.apache.org/jira/browse/HDFS-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199731#comment-13199731 ] Hudson commented on HDFS-2718: -- Integrated in Hadoop-Mapreduce-trunk #978 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/978/]) HDFS-2718. Optimize OP_ADD in edits loading. Contributed by Konstantin Shvachko. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1239760 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFileUnderConstruction.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestAbandonBlock.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java Optimize OP_ADD in edits loading Key: HDFS-2718 URL: https://issues.apache.org/jira/browse/HDFS-2718 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.22.0, 0.24.0, 1.0.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Fix For: 0.24.0, 0.23.1, 0.22.1 Attachments: editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch During loading the edits journal FSEditLog.loadEditRecords() processes OP_ADD inefficiently. It first removes the existing INodeFile from the directory tree, then adds it back as a regular INodeFile, and then replaces it with INodeFileUnderConstruction if files is not closed. This slows down edits loading. OP_ADD should be done in one shot and retain previously existing data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2718) Optimize OP_ADD in edits loading
[ https://issues.apache.org/jira/browse/HDFS-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199104#comment-13199104 ] Hudson commented on HDFS-2718: -- Integrated in Hadoop-Common-trunk-Commit #1645 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1645/]) HDFS-2718. Optimize OP_ADD in edits loading. Contributed by Konstantin Shvachko. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1239760 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFileUnderConstruction.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestAbandonBlock.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java Optimize OP_ADD in edits loading Key: HDFS-2718 URL: https://issues.apache.org/jira/browse/HDFS-2718 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.22.0, 0.24.0, 1.0.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Attachments: editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch During loading the edits journal FSEditLog.loadEditRecords() processes OP_ADD inefficiently. It first removes the existing INodeFile from the directory tree, then adds it back as a regular INodeFile, and then replaces it with INodeFileUnderConstruction if files is not closed. This slows down edits loading. OP_ADD should be done in one shot and retain previously existing data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2718) Optimize OP_ADD in edits loading
[ https://issues.apache.org/jira/browse/HDFS-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199101#comment-13199101 ] Hudson commented on HDFS-2718: -- Integrated in Hadoop-Hdfs-trunk-Commit #1716 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1716/]) HDFS-2718. Optimize OP_ADD in edits loading. Contributed by Konstantin Shvachko. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1239760 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFileUnderConstruction.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestAbandonBlock.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java Optimize OP_ADD in edits loading Key: HDFS-2718 URL: https://issues.apache.org/jira/browse/HDFS-2718 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.22.0, 0.24.0, 1.0.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Attachments: editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch During loading the edits journal FSEditLog.loadEditRecords() processes OP_ADD inefficiently. It first removes the existing INodeFile from the directory tree, then adds it back as a regular INodeFile, and then replaces it with INodeFileUnderConstruction if files is not closed. This slows down edits loading. OP_ADD should be done in one shot and retain previously existing data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2718) Optimize OP_ADD in edits loading
[ https://issues.apache.org/jira/browse/HDFS-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199129#comment-13199129 ] Hudson commented on HDFS-2718: -- Integrated in Hadoop-Hdfs-0.23-Commit #462 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/462/]) HDFS-2718. Optimize OP_ADD in edits loading. Contributed by Konstantin Shvachko. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1239769 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFileUnderConstruction.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestAbandonBlock.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java Optimize OP_ADD in edits loading Key: HDFS-2718 URL: https://issues.apache.org/jira/browse/HDFS-2718 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.22.0, 0.24.0, 1.0.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Fix For: 0.24.0, 0.23.1, 0.22.1 Attachments: editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch During loading the edits journal FSEditLog.loadEditRecords() processes OP_ADD inefficiently. It first removes the existing INodeFile from the directory tree, then adds it back as a regular INodeFile, and then replaces it with INodeFileUnderConstruction if files is not closed. This slows down edits loading. OP_ADD should be done in one shot and retain previously existing data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2718) Optimize OP_ADD in edits loading
[ https://issues.apache.org/jira/browse/HDFS-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199132#comment-13199132 ] Hudson commented on HDFS-2718: -- Integrated in Hadoop-Common-0.23-Commit #472 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/472/]) HDFS-2718. Optimize OP_ADD in edits loading. Contributed by Konstantin Shvachko. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1239769 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFileUnderConstruction.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestAbandonBlock.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java Optimize OP_ADD in edits loading Key: HDFS-2718 URL: https://issues.apache.org/jira/browse/HDFS-2718 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.22.0, 0.24.0, 1.0.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Fix For: 0.24.0, 0.23.1, 0.22.1 Attachments: editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch During loading the edits journal FSEditLog.loadEditRecords() processes OP_ADD inefficiently. It first removes the existing INodeFile from the directory tree, then adds it back as a regular INodeFile, and then replaces it with INodeFileUnderConstruction if files is not closed. This slows down edits loading. OP_ADD should be done in one shot and retain previously existing data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2718) Optimize OP_ADD in edits loading
[ https://issues.apache.org/jira/browse/HDFS-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199149#comment-13199149 ] Hudson commented on HDFS-2718: -- Integrated in Hadoop-Mapreduce-trunk-Commit #1661 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1661/]) HDFS-2718. Optimize OP_ADD in edits loading. Contributed by Konstantin Shvachko. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1239760 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFileUnderConstruction.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestAbandonBlock.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java Optimize OP_ADD in edits loading Key: HDFS-2718 URL: https://issues.apache.org/jira/browse/HDFS-2718 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.22.0, 0.24.0, 1.0.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Fix For: 0.24.0, 0.23.1, 0.22.1 Attachments: editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch During loading the edits journal FSEditLog.loadEditRecords() processes OP_ADD inefficiently. It first removes the existing INodeFile from the directory tree, then adds it back as a regular INodeFile, and then replaces it with INodeFileUnderConstruction if files is not closed. This slows down edits loading. OP_ADD should be done in one shot and retain previously existing data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2718) Optimize OP_ADD in edits loading
[ https://issues.apache.org/jira/browse/HDFS-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199180#comment-13199180 ] Hudson commented on HDFS-2718: -- Integrated in Hadoop-Mapreduce-0.23-Commit #486 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/486/]) HDFS-2718. Optimize OP_ADD in edits loading. Contributed by Konstantin Shvachko. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1239769 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFileUnderConstruction.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestAbandonBlock.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java Optimize OP_ADD in edits loading Key: HDFS-2718 URL: https://issues.apache.org/jira/browse/HDFS-2718 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.22.0, 0.24.0, 1.0.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Fix For: 0.24.0, 0.23.1, 0.22.1 Attachments: editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch During loading the edits journal FSEditLog.loadEditRecords() processes OP_ADD inefficiently. It first removes the existing INodeFile from the directory tree, then adds it back as a regular INodeFile, and then replaces it with INodeFileUnderConstruction if files is not closed. This slows down edits loading. OP_ADD should be done in one shot and retain previously existing data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2718) Optimize OP_ADD in edits loading
[ https://issues.apache.org/jira/browse/HDFS-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199309#comment-13199309 ] Todd Lipcon commented on HDFS-2718: --- Looking at what was committed to trunk, I found a couple issues. Why was this committed without any +1? Optimize OP_ADD in edits loading Key: HDFS-2718 URL: https://issues.apache.org/jira/browse/HDFS-2718 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.22.0, 0.24.0, 1.0.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Fix For: 0.24.0, 0.23.1, 0.22.1 Attachments: editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch During loading the edits journal FSEditLog.loadEditRecords() processes OP_ADD inefficiently. It first removes the existing INodeFile from the directory tree, then adds it back as a regular INodeFile, and then replaces it with INodeFileUnderConstruction if files is not closed. This slows down edits loading. OP_ADD should be done in one shot and retain previously existing data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2718) Optimize OP_ADD in edits loading
[ https://issues.apache.org/jira/browse/HDFS-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199330#comment-13199330 ] Konstantin Shvachko commented on HDFS-2718: --- Glad to have your attention, Todd. What are the issues? Optimize OP_ADD in edits loading Key: HDFS-2718 URL: https://issues.apache.org/jira/browse/HDFS-2718 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.22.0, 0.24.0, 1.0.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Fix For: 0.24.0, 0.23.1, 0.22.1 Attachments: editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch During loading the edits journal FSEditLog.loadEditRecords() processes OP_ADD inefficiently. It first removes the existing INodeFile from the directory tree, then adds it back as a regular INodeFile, and then replaces it with INodeFileUnderConstruction if files is not closed. This slows down edits loading. OP_ADD should be done in one shot and retain previously existing data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2718) Optimize OP_ADD in edits loading
[
https://issues.apache.org/jira/browse/HDFS-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199367#comment-13199367
]
Todd Lipcon commented on HDFS-2718:
---
No major issues, but this is fairly critical code and committing without a
committer +1 worries me. Here's my feedback on the patch:
- Rather than adjusting the code to handle the invalid edit log case of first
OP_ADD having multiple blocks, we should fix the CreateEditLogs tool to create
a realistic edit log. That would allow us to remove code from the loader, as
well as avoid having to add the new INodeFileUnderConstruction constructor.
- The {{updateFile}} call takes a number of parameters which aren't used -- in
particular permissions, replication, and preferredBlockSize. Having these
parameters present implies that they'll also be used to update the file, which
is not the case.
- I think the construction of filenames in TestEditLog may be buggy -- I think
you mean to do {{/filename + (startIndex + i)}} since otherwise it will do
two string-appends, rather than a string append of an addition.
Optimize OP_ADD in edits loading
Key: HDFS-2718
URL: https://issues.apache.org/jira/browse/HDFS-2718
Project: Hadoop HDFS
Issue Type: Bug
Components: name-node
Affects Versions: 0.22.0, 0.24.0, 1.0.0
Reporter: Konstantin Shvachko
Assignee: Konstantin Shvachko
Fix For: 0.24.0, 0.23.1, 0.22.1
Attachments: editsLoader-0.22.patch, editsLoader-0.22.patch,
editsLoader-0.22.patch, editsLoader-trunk.patch, editsLoader-trunk.patch,
editsLoader-trunk.patch, editsLoader-trunk.patch
During loading the edits journal FSEditLog.loadEditRecords() processes OP_ADD
inefficiently. It first removes the existing INodeFile from the directory
tree, then adds it back as a regular INodeFile, and then replaces it with
INodeFileUnderConstruction if files is not closed. This slows down edits
loading. OP_ADD should be done in one shot and retain previously existing
data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2718) Optimize OP_ADD in edits loading
[ https://issues.apache.org/jira/browse/HDFS-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199371#comment-13199371 ] Aaron T. Myers commented on HDFS-2718: -- bq. I think the construction of filenames in TestEditLog may be buggy – I think you mean to do /filename + (startIndex + i) since otherwise it will do two string-appends, rather than a string append of an addition. To address this issue, why not use the exact same code as was used in the patch for HDFS-2602? It doesn't have this bug, and will make merges a little bit easier. Optimize OP_ADD in edits loading Key: HDFS-2718 URL: https://issues.apache.org/jira/browse/HDFS-2718 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.22.0, 0.24.0, 1.0.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Fix For: 0.24.0, 0.23.1, 0.22.1 Attachments: editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch During loading the edits journal FSEditLog.loadEditRecords() processes OP_ADD inefficiently. It first removes the existing INodeFile from the directory tree, then adds it back as a regular INodeFile, and then replaces it with INodeFileUnderConstruction if files is not closed. This slows down edits loading. OP_ADD should be done in one shot and retain previously existing data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2718) Optimize OP_ADD in edits loading
[ https://issues.apache.org/jira/browse/HDFS-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199405#comment-13199405 ] Hudson commented on HDFS-2718: -- Integrated in Hadoop-Hdfs-22-branch #125 (See [https://builds.apache.org/job/Hadoop-Hdfs-22-branch/125/]) HDFS-2718. Optimize OP_ADD in edits loading. Contributed by Konstantin Shvachko. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1239762 Files : * /hadoop/common/branches/branch-0.22/hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/BlockInfoUnderConstruction.java * /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/BlockManager.java * /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/INodeFileUnderConstruction.java * /hadoop/common/branches/branch-0.22/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestAbandonBlock.java * /hadoop/common/branches/branch-0.22/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java Optimize OP_ADD in edits loading Key: HDFS-2718 URL: https://issues.apache.org/jira/browse/HDFS-2718 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.22.0, 0.24.0, 1.0.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Fix For: 0.24.0, 0.23.1, 0.22.1 Attachments: editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch During loading the edits journal FSEditLog.loadEditRecords() processes OP_ADD inefficiently. It first removes the existing INodeFile from the directory tree, then adds it back as a regular INodeFile, and then replaces it with INodeFileUnderConstruction if files is not closed. This slows down edits loading. OP_ADD should be done in one shot and retain previously existing data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2718) Optimize OP_ADD in edits loading
[ https://issues.apache.org/jira/browse/HDFS-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199535#comment-13199535 ] Konstantin Shvachko commented on HDFS-2718: --- Intended to fix CreateEditLogs in a subsequent jira. Got the patch ready. Will cleanup the parameters in there as well. As for (startIndex + i), my patch does exactly the same as HDFS-2602, except the latter adds - between the stringified numbers. If you insist I'll reuse the code from your patch for TestEditLog. But I don't see a bug here as long as the file names generated by different threads are disjoint, which was a bug in the original code. Thanks for the feedback. Optimize OP_ADD in edits loading Key: HDFS-2718 URL: https://issues.apache.org/jira/browse/HDFS-2718 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.22.0, 0.24.0, 1.0.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Fix For: 0.24.0, 0.23.1, 0.22.1 Attachments: editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch During loading the edits journal FSEditLog.loadEditRecords() processes OP_ADD inefficiently. It first removes the existing INodeFile from the directory tree, then adds it back as a regular INodeFile, and then replaces it with INodeFileUnderConstruction if files is not closed. This slows down edits loading. OP_ADD should be done in one shot and retain previously existing data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2718) Optimize OP_ADD in edits loading
[ https://issues.apache.org/jira/browse/HDFS-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197962#comment-13197962 ] Konstantin Shvachko commented on HDFS-2718: --- If there are no more comments I'll commit it later today. Optimize OP_ADD in edits loading Key: HDFS-2718 URL: https://issues.apache.org/jira/browse/HDFS-2718 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.22.0, 0.24.0, 1.0.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Attachments: editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch During loading the edits journal FSEditLog.loadEditRecords() processes OP_ADD inefficiently. It first removes the existing INodeFile from the directory tree, then adds it back as a regular INodeFile, and then replaces it with INodeFileUnderConstruction if files is not closed. This slows down edits loading. OP_ADD should be done in one shot and retain previously existing data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2718) Optimize OP_ADD in edits loading
[ https://issues.apache.org/jira/browse/HDFS-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198065#comment-13198065 ] Aaron T. Myers commented on HDFS-2718: -- Hey Konst, the patch largely looks good. A few comments: # Given that there's no protected analog to unprotectedUpdateFile, I think it should be renamed updateFile. The unprotected term is usually used when there's another method which calls the unprotected method, and also gets the write lock and logs to the edit log. # Given that FSDirectory#unprotectedUpdateFile is only called from FSEditLogLoader, let's move this code to FSEditLogLoader. Then, we can also change it to take just the AddCloseOp as a parameter, instead of all the members of AddCloseOp as individual parameters. # Nit: there's a few spots in the new patch where you have if(. Please put a space between if and ( per the style guidelines. Optimize OP_ADD in edits loading Key: HDFS-2718 URL: https://issues.apache.org/jira/browse/HDFS-2718 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.22.0, 0.24.0, 1.0.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Attachments: editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch During loading the edits journal FSEditLog.loadEditRecords() processes OP_ADD inefficiently. It first removes the existing INodeFile from the directory tree, then adds it back as a regular INodeFile, and then replaces it with INodeFileUnderConstruction if files is not closed. This slows down edits loading. OP_ADD should be done in one shot and retain previously existing data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2718) Optimize OP_ADD in edits loading
[ https://issues.apache.org/jira/browse/HDFS-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198534#comment-13198534 ] Hadoop QA commented on HDFS-2718: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12512905/editsLoader-trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1832//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1832//console This message is automatically generated. Optimize OP_ADD in edits loading Key: HDFS-2718 URL: https://issues.apache.org/jira/browse/HDFS-2718 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.22.0, 0.24.0, 1.0.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Attachments: editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch During loading the edits journal FSEditLog.loadEditRecords() processes OP_ADD inefficiently. It first removes the existing INodeFile from the directory tree, then adds it back as a regular INodeFile, and then replaces it with INodeFileUnderConstruction if files is not closed. This slows down edits loading. OP_ADD should be done in one shot and retain previously existing data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2718) Optimize OP_ADD in edits loading
[ https://issues.apache.org/jira/browse/HDFS-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196796#comment-13196796 ] Konstantin Shvachko commented on HDFS-2718: --- Findbugs is the same GetConf. TestFSInputChecker fails because of HADOOP-8006. Both unrelated. Optimize OP_ADD in edits loading Key: HDFS-2718 URL: https://issues.apache.org/jira/browse/HDFS-2718 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.22.0, 0.24.0, 1.0.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Attachments: editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch During loading the edits journal FSEditLog.loadEditRecords() processes OP_ADD inefficiently. It first removes the existing INodeFile from the directory tree, then adds it back as a regular INodeFile, and then replaces it with INodeFileUnderConstruction if files is not closed. This slows down edits loading. OP_ADD should be done in one shot and retain previously existing data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2718) Optimize OP_ADD in edits loading
[
https://issues.apache.org/jira/browse/HDFS-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196018#comment-13196018
]
Jitendra Nath Pandey commented on HDFS-2718:
Is it ok to add lease in all cases of OP_ADD, what if a lease already exists?
It seems to me it should be fine as the existing lease will just be renewed,
please confirm.
{code}
diskspace = ((INodeFile)newNode).diskspaceConsumed(blocks);
{code}
If blocks are null diskspace will be zero. In the existing code it is -1
(UNKNOWN_DISK_SPACE).
{code}
if(getBlockUCState() != BlockUCState.COMMITTED)
throw new IOException(
Cannot complete block: block has not been COMMITTED by the client);
{code}
Please clarify why this code was removed from convertToCompleteBlock and moved
to the caller.
Optimize OP_ADD in edits loading
Key: HDFS-2718
URL: https://issues.apache.org/jira/browse/HDFS-2718
Project: Hadoop HDFS
Issue Type: Bug
Components: name-node
Affects Versions: 0.22.0, 0.24.0, 1.0.0
Reporter: Konstantin Shvachko
Assignee: Konstantin Shvachko
Attachments: editsLoader-0.22.patch, editsLoader-trunk.patch,
editsLoader-trunk.patch
During loading the edits journal FSEditLog.loadEditRecords() processes OP_ADD
inefficiently. It first removes the existing INodeFile from the directory
tree, then adds it back as a regular INodeFile, and then replaces it with
INodeFileUnderConstruction if files is not closed. This slows down edits
loading. OP_ADD should be done in one shot and retain previously existing
data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2718) Optimize OP_ADD in edits loading
[ https://issues.apache.org/jira/browse/HDFS-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196234#comment-13196234 ] Aaron T. Myers commented on HDFS-2718: -- Hey Kosnt, given that this change is being done as a performance optimization, do you have any numbers on the performance gain of edits loading attained from this change? It seems like it should speed things up, and it's at least a good refactor that should make the code less obtuse, but I'd still like to make sure edits loading actually is faster after this change. Optimize OP_ADD in edits loading Key: HDFS-2718 URL: https://issues.apache.org/jira/browse/HDFS-2718 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.22.0, 0.24.0, 1.0.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Attachments: editsLoader-0.22.patch, editsLoader-trunk.patch, editsLoader-trunk.patch During loading the edits journal FSEditLog.loadEditRecords() processes OP_ADD inefficiently. It first removes the existing INodeFile from the directory tree, then adds it back as a regular INodeFile, and then replaces it with INodeFileUnderConstruction if files is not closed. This slows down edits loading. OP_ADD should be done in one shot and retain previously existing data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2718) Optimize OP_ADD in edits loading
[
https://issues.apache.org/jira/browse/HDFS-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196647#comment-13196647
]
Konstantin Shvachko commented on HDFS-2718:
---
Thanks guys for the reviews.
JP it should be fine as the existing lease will just be renewed, please
confirm.
Correct.
JP If blocks are null diskspace will be zero. In the existing code it is -1
(UNKNOWN_DISK_SPACE).
Good point. Just realized that explicit calculation of diskspace is not
necessary here at all, because it is calculated in addNode() if childDiskspace
0. Removing diskspace calculation completely.
JP COMMITTED code was removed from convertToCompleteBlock and moved to the
caller.
COMMITTED state is a confirmation from client that it finished writing the
block.
During edits loading block will never be in committed state, since there are no
clients. So we force it to COMPLETE state. And checking if the block is
COMMITTED should be performed only if block completion is not forced. Therefore
had to move verification to the caller.
ATM do you have any numbers on the performance gain of edits loading attained
from this change?
I created 500,000 files using CreateEditsLog utility. And then started NameNode
before and after applying the patch. The results show just over 20% improvement.
I expect that with addBlock transactions in place the gain will be higher.
{code}
Before
12/01/27 16:05:21 INFO common.Storage: Edits file
/hadoop-data/hdfs/name/current/edits of size 286856143 edits # 1005001 loaded
in 18 seconds.
After
12/01/27 16:21:17 INFO common.Storage: Edits file
/hadoop-data/hdfs/name/current/edits of size 203356143 edits # 1005001 loaded
in 14 seconds.
{code}
As a side effect it turned out that CreateEditsLog generates non-standard
transactions. In real life first transaction that creates a file does not
contain blocks. While CreateEditsLog adds blocks to this transaction. I had to
introduced additional constructor for INodeFileUnderConstruction in order to
cover this use case.
Attaching files shortly.
Optimize OP_ADD in edits loading
Key: HDFS-2718
URL: https://issues.apache.org/jira/browse/HDFS-2718
Project: Hadoop HDFS
Issue Type: Bug
Components: name-node
Affects Versions: 0.22.0, 0.24.0, 1.0.0
Reporter: Konstantin Shvachko
Assignee: Konstantin Shvachko
Attachments: editsLoader-0.22.patch, editsLoader-trunk.patch,
editsLoader-trunk.patch
During loading the edits journal FSEditLog.loadEditRecords() processes OP_ADD
inefficiently. It first removes the existing INodeFile from the directory
tree, then adds it back as a regular INodeFile, and then replaces it with
INodeFileUnderConstruction if files is not closed. This slows down edits
loading. OP_ADD should be done in one shot and retain previously existing
data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2718) Optimize OP_ADD in edits loading
[ https://issues.apache.org/jira/browse/HDFS-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196683#comment-13196683 ] Hadoop QA commented on HDFS-2718: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12512526/editsLoader-trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestFSInputChecker +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1826//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1826//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1826//console This message is automatically generated. Optimize OP_ADD in edits loading Key: HDFS-2718 URL: https://issues.apache.org/jira/browse/HDFS-2718 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.22.0, 0.24.0, 1.0.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Attachments: editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch During loading the edits journal FSEditLog.loadEditRecords() processes OP_ADD inefficiently. It first removes the existing INodeFile from the directory tree, then adds it back as a regular INodeFile, and then replaces it with INodeFileUnderConstruction if files is not closed. This slows down edits loading. OP_ADD should be done in one shot and retain previously existing data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2718) Optimize OP_ADD in edits loading
[ https://issues.apache.org/jira/browse/HDFS-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13191591#comment-13191591 ] Hadoop QA commented on HDFS-2718: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12511578/editsLoader-trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1800//console This message is automatically generated. Optimize OP_ADD in edits loading Key: HDFS-2718 URL: https://issues.apache.org/jira/browse/HDFS-2718 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.22.0, 0.24.0, 1.0.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Attachments: editsLoader-0.22.patch, editsLoader-trunk.patch During loading the edits journal FSEditLog.loadEditRecords() processes OP_ADD inefficiently. It first removes the existing INodeFile from the directory tree, then adds it back as a regular INodeFile, and then replaces it with INodeFileUnderConstruction if files is not closed. This slows down edits loading. OP_ADD should be done in one shot and retain previously existing data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2718) Optimize OP_ADD in edits loading
[ https://issues.apache.org/jira/browse/HDFS-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13191698#comment-13191698 ] Hadoop QA commented on HDFS-2718: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12511588/editsLoader-trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 21 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1801//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1801//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1801//console This message is automatically generated. Optimize OP_ADD in edits loading Key: HDFS-2718 URL: https://issues.apache.org/jira/browse/HDFS-2718 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.22.0, 0.24.0, 1.0.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Attachments: editsLoader-0.22.patch, editsLoader-trunk.patch, editsLoader-trunk.patch During loading the edits journal FSEditLog.loadEditRecords() processes OP_ADD inefficiently. It first removes the existing INodeFile from the directory tree, then adds it back as a regular INodeFile, and then replaces it with INodeFileUnderConstruction if files is not closed. This slows down edits loading. OP_ADD should be done in one shot and retain previously existing data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2718) Optimize OP_ADD in edits loading
[ https://issues.apache.org/jira/browse/HDFS-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13191789#comment-13191789 ] Konstantin Shvachko commented on HDFS-2718: --- Checked findBugs warning. It is because GetConf#Command as enum is serializable, but contains non-serializable field handler. This patch does not change GetConf. Checked JavaDoc warnings: all are in files that are not affected by the patch. Optimize OP_ADD in edits loading Key: HDFS-2718 URL: https://issues.apache.org/jira/browse/HDFS-2718 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.22.0, 0.24.0, 1.0.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Attachments: editsLoader-0.22.patch, editsLoader-trunk.patch, editsLoader-trunk.patch During loading the edits journal FSEditLog.loadEditRecords() processes OP_ADD inefficiently. It first removes the existing INodeFile from the directory tree, then adds it back as a regular INodeFile, and then replaces it with INodeFileUnderConstruction if files is not closed. This slows down edits loading. OP_ADD should be done in one shot and retain previously existing data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2718) Optimize OP_ADD in edits loading
[ https://issues.apache.org/jira/browse/HDFS-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13175114#comment-13175114 ] Todd Lipcon commented on HDFS-2718: --- This is very similar to HDFS-2602 - we should at least share code from the HA branch so we don't have a nasty conflict at merge time. Optimize OP_ADD in edits loading Key: HDFS-2718 URL: https://issues.apache.org/jira/browse/HDFS-2718 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.22.0, 0.24.0, 1.0.0 Reporter: Konstantin Shvachko During loading the edits journal FSEditLog.loadEditRecords() processes OP_ADD inefficiently. It first removes the existing INodeFile from the directory tree, then adds it back as a regular INodeFile, and then replaces it with INodeFileUnderConstruction if files is not closed. This slows down edits loading. OP_ADD should be done in one shot and retain previously existing data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
