[jira] [Updated] (HDFS-2718) Optimize OP_ADD in edits loading
[ https://issues.apache.org/jira/browse/HDFS-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-2718: -- Resolution: Fixed Fix Version/s: 0.22.1 0.23.1 0.24.0 Target Version/s: 0.24.0, 0.23.1, 0.22.1 (was: 0.22.1, 0.23.1, 0.24.0) Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I just committed this. Optimize OP_ADD in edits loading Key: HDFS-2718 URL: https://issues.apache.org/jira/browse/HDFS-2718 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.22.0, 0.24.0, 1.0.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Fix For: 0.24.0, 0.23.1, 0.22.1 Attachments: editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch During loading the edits journal FSEditLog.loadEditRecords() processes OP_ADD inefficiently. It first removes the existing INodeFile from the directory tree, then adds it back as a regular INodeFile, and then replaces it with INodeFileUnderConstruction if files is not closed. This slows down edits loading. OP_ADD should be done in one shot and retain previously existing data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2718) Optimize OP_ADD in edits loading
[ https://issues.apache.org/jira/browse/HDFS-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-2718: -- Attachment: editsLoader-trunk.patch editsLoader-0.22.patch I tried to move updateFiles() into EditLogLoader as Aaron suggested. It is possible, but I didn't like how it looked. Yes this method is called only in EditLogLoader, but by functionality it clearly belongs to FSDirectory, as it updates the file in the directory tree, same as addFile() or delete(). And methods should be assigned to classes based on the functionality rather than usage. So I left it in FSDirectory. Unprotected means as I understand it that the method does not hold the lock, so the naming was consistent, but I agree we have too many unprotected methods for no apparent reason, so I renamed it as suggested. If-s are half that way and half another. I am probably guilty of a lot of them, but don't see the point of fighting it now. Optimize OP_ADD in edits loading Key: HDFS-2718 URL: https://issues.apache.org/jira/browse/HDFS-2718 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.22.0, 0.24.0, 1.0.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Attachments: editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch During loading the edits journal FSEditLog.loadEditRecords() processes OP_ADD inefficiently. It first removes the existing INodeFile from the directory tree, then adds it back as a regular INodeFile, and then replaces it with INodeFileUnderConstruction if files is not closed. This slows down edits loading. OP_ADD should be done in one shot and retain previously existing data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2718) Optimize OP_ADD in edits loading
[ https://issues.apache.org/jira/browse/HDFS-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-2718: -- Attachment: editsLoader-0.22.patch Optimize OP_ADD in edits loading Key: HDFS-2718 URL: https://issues.apache.org/jira/browse/HDFS-2718 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.22.0, 0.24.0, 1.0.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Attachments: editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-trunk.patch, editsLoader-trunk.patch During loading the edits journal FSEditLog.loadEditRecords() processes OP_ADD inefficiently. It first removes the existing INodeFile from the directory tree, then adds it back as a regular INodeFile, and then replaces it with INodeFileUnderConstruction if files is not closed. This slows down edits loading. OP_ADD should be done in one shot and retain previously existing data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2718) Optimize OP_ADD in edits loading
[ https://issues.apache.org/jira/browse/HDFS-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-2718: -- Target Version/s: 0.24.0, 0.23.1, 0.22.1 (was: 0.22.1, 0.23.1, 0.24.0) Status: Open (was: Patch Available) Optimize OP_ADD in edits loading Key: HDFS-2718 URL: https://issues.apache.org/jira/browse/HDFS-2718 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 1.0.0, 0.22.0, 0.24.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Attachments: editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-trunk.patch, editsLoader-trunk.patch During loading the edits journal FSEditLog.loadEditRecords() processes OP_ADD inefficiently. It first removes the existing INodeFile from the directory tree, then adds it back as a regular INodeFile, and then replaces it with INodeFileUnderConstruction if files is not closed. This slows down edits loading. OP_ADD should be done in one shot and retain previously existing data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2718) Optimize OP_ADD in edits loading
[ https://issues.apache.org/jira/browse/HDFS-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-2718: -- Target Version/s: 0.24.0, 0.23.1, 0.22.1 (was: 0.22.1, 0.23.1, 0.24.0) Status: Patch Available (was: Open) Optimize OP_ADD in edits loading Key: HDFS-2718 URL: https://issues.apache.org/jira/browse/HDFS-2718 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 1.0.0, 0.22.0, 0.24.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Attachments: editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch During loading the edits journal FSEditLog.loadEditRecords() processes OP_ADD inefficiently. It first removes the existing INodeFile from the directory tree, then adds it back as a regular INodeFile, and then replaces it with INodeFileUnderConstruction if files is not closed. This slows down edits loading. OP_ADD should be done in one shot and retain previously existing data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2718) Optimize OP_ADD in edits loading
[ https://issues.apache.org/jira/browse/HDFS-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-2718: -- Attachment: editsLoader-0.22.patch Optimize OP_ADD in edits loading Key: HDFS-2718 URL: https://issues.apache.org/jira/browse/HDFS-2718 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.22.0, 0.24.0, 1.0.0 Reporter: Konstantin Shvachko Attachments: editsLoader-0.22.patch During loading the edits journal FSEditLog.loadEditRecords() processes OP_ADD inefficiently. It first removes the existing INodeFile from the directory tree, then adds it back as a regular INodeFile, and then replaces it with INodeFileUnderConstruction if files is not closed. This slows down edits loading. OP_ADD should be done in one shot and retain previously existing data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2718) Optimize OP_ADD in edits loading
[ https://issues.apache.org/jira/browse/HDFS-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-2718: -- Attachment: editsLoader-trunk.patch Here are the patches for trunk and 0.22 branch. The patch for trunk is applicable to 0.23 branch as well. I tried to make the two patches as close to one another as possible. No new tests included, but I modified existing tests so that they tested the new functionality. I ran all tests for 0.22, Everything passed. For trunk I ran some key tests and will let Jenkins validate everything else. Merging with HA branch shouldn't be hard. In 0.22 I tested this patch with block synchronization turned on - also works fine. Can somebody please review this. Optimize OP_ADD in edits loading Key: HDFS-2718 URL: https://issues.apache.org/jira/browse/HDFS-2718 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.22.0, 0.24.0, 1.0.0 Reporter: Konstantin Shvachko Attachments: editsLoader-0.22.patch, editsLoader-trunk.patch During loading the edits journal FSEditLog.loadEditRecords() processes OP_ADD inefficiently. It first removes the existing INodeFile from the directory tree, then adds it back as a regular INodeFile, and then replaces it with INodeFileUnderConstruction if files is not closed. This slows down edits loading. OP_ADD should be done in one shot and retain previously existing data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2718) Optimize OP_ADD in edits loading
[ https://issues.apache.org/jira/browse/HDFS-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-2718: -- Target Version/s: 0.24.0, 0.23.1, 0.22.1 (was: 0.22.1) Assignee: Konstantin Shvachko Optimize OP_ADD in edits loading Key: HDFS-2718 URL: https://issues.apache.org/jira/browse/HDFS-2718 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.22.0, 0.24.0, 1.0.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Attachments: editsLoader-0.22.patch, editsLoader-trunk.patch During loading the edits journal FSEditLog.loadEditRecords() processes OP_ADD inefficiently. It first removes the existing INodeFile from the directory tree, then adds it back as a regular INodeFile, and then replaces it with INodeFileUnderConstruction if files is not closed. This slows down edits loading. OP_ADD should be done in one shot and retain previously existing data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2718) Optimize OP_ADD in edits loading
[ https://issues.apache.org/jira/browse/HDFS-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-2718: -- Target Version/s: 0.24.0, 0.23.1, 0.22.1 (was: 0.22.1, 0.23.1, 0.24.0) Status: Patch Available (was: Open) Optimize OP_ADD in edits loading Key: HDFS-2718 URL: https://issues.apache.org/jira/browse/HDFS-2718 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 1.0.0, 0.22.0, 0.24.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Attachments: editsLoader-0.22.patch, editsLoader-trunk.patch During loading the edits journal FSEditLog.loadEditRecords() processes OP_ADD inefficiently. It first removes the existing INodeFile from the directory tree, then adds it back as a regular INodeFile, and then replaces it with INodeFileUnderConstruction if files is not closed. This slows down edits loading. OP_ADD should be done in one shot and retain previously existing data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2718) Optimize OP_ADD in edits loading
[ https://issues.apache.org/jira/browse/HDFS-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-2718: -- Attachment: editsLoader-trunk.patch Optimize OP_ADD in edits loading Key: HDFS-2718 URL: https://issues.apache.org/jira/browse/HDFS-2718 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.22.0, 0.24.0, 1.0.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Attachments: editsLoader-0.22.patch, editsLoader-trunk.patch, editsLoader-trunk.patch During loading the edits journal FSEditLog.loadEditRecords() processes OP_ADD inefficiently. It first removes the existing INodeFile from the directory tree, then adds it back as a regular INodeFile, and then replaces it with INodeFileUnderConstruction if files is not closed. This slows down edits loading. OP_ADD should be done in one shot and retain previously existing data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2718) Optimize OP_ADD in edits loading
[ https://issues.apache.org/jira/browse/HDFS-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-2718: -- Target Version/s: 0.22.1 I remember this was discussed on several occasions but couldn't find a jira for that, let me know if you know one. This exists in all HDFS versions. Optimize OP_ADD in edits loading Key: HDFS-2718 URL: https://issues.apache.org/jira/browse/HDFS-2718 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.22.0, 0.24.0, 1.0.0 Reporter: Konstantin Shvachko During loading the edits journal FSEditLog.loadEditRecords() processes OP_ADD inefficiently. It first removes the existing INodeFile from the directory tree, then adds it back as a regular INodeFile, and then replaces it with INodeFileUnderConstruction if files is not closed. This slows down edits loading. OP_ADD should be done in one shot and retain previously existing data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
