[jira] Commented: (HDFS-1489) breaking the dependency between FSEditLog and FSImage
[ https://issues.apache.org/jira/browse/HDFS-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12974585#action_12974585 ] Ivan Kelly commented on HDFS-1489: -- I've moved the splitting out of NNStorage into a separate subtask, HDFS-1557 breaking the dependency between FSEditLog and FSImage - Key: HDFS-1489 URL: https://issues.apache.org/jira/browse/HDFS-1489 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 0.21.0 Reporter: Diego Marron Attachments: HDFS-1489.diff, HDFS-1489.diff, HDFS-1489.pdf, NNStorage.diff, NNStorage.diff, NNStorage.diff This is a refactor patch which its main concerns are: - breaking the dependency between FSEditLog and FSImage - Splitting the abstracting the error handling and directory management, - Decoupling Storage from FSImage. In order to accomplish the above goal, we will need to introduce new classes: - NNStorage: Will care about the storage. It extends Storage class, and will contain the StorageDirectories. - NNUtils: Some utility static methods on FSImage and FSEditLog will be moved here. - PersistenceManager: FSNameSystem will now be responsible for managing the FSImage FSEditLog objects. There will be some logic that will have to moved out of FSImage to facilite this. For this we propose a PersistanceManager? object as follows. For more deep details, see the design document uploaded. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1489) breaking the dependency between FSEditLog and FSImage
[ https://issues.apache.org/jira/browse/HDFS-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12974657#action_12974657 ] Ivan Kelly commented on HDFS-1489: -- Just to notify you guys, HDFS-1557 is in patch submitted state. breaking the dependency between FSEditLog and FSImage - Key: HDFS-1489 URL: https://issues.apache.org/jira/browse/HDFS-1489 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 0.21.0 Reporter: Diego Marron Attachments: HDFS-1489.diff, HDFS-1489.diff, HDFS-1489.pdf, NNStorage.diff, NNStorage.diff, NNStorage.diff This is a refactor patch which its main concerns are: - breaking the dependency between FSEditLog and FSImage - Splitting the abstracting the error handling and directory management, - Decoupling Storage from FSImage. In order to accomplish the above goal, we will need to introduce new classes: - NNStorage: Will care about the storage. It extends Storage class, and will contain the StorageDirectories. - NNUtils: Some utility static methods on FSImage and FSEditLog will be moved here. - PersistenceManager: FSNameSystem will now be responsible for managing the FSImage FSEditLog objects. There will be some logic that will have to moved out of FSImage to facilite this. For this we propose a PersistanceManager? object as follows. For more deep details, see the design document uploaded. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1489) breaking the dependency between FSEditLog and FSImage
[ https://issues.apache.org/jira/browse/HDFS-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12973549#action_12973549 ] Konstantin Shvachko commented on HDFS-1489: --- once you stop FSImage from deriving from Storage all hell breaks loose. If NNStorage extends Storage now, what is the role of FSImage then? breaking the dependency between FSEditLog and FSImage - Key: HDFS-1489 URL: https://issues.apache.org/jira/browse/HDFS-1489 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 0.21.0 Reporter: Diego Marron Attachments: HDFS-1489.diff, HDFS-1489.diff, HDFS-1489.pdf, NNStorage.diff This is a refactor patch which its main concerns are: - breaking the dependency between FSEditLog and FSImage - Splitting the abstracting the error handling and directory management, - Decoupling Storage from FSImage. In order to accomplish the above goal, we will need to introduce new classes: - NNStorage: Will care about the storage. It extends Storage class, and will contain the StorageDirectories. - NNUtils: Some utility static methods on FSImage and FSEditLog will be moved here. - PersistenceManager: FSNameSystem will now be responsible for managing the FSImage FSEditLog objects. There will be some logic that will have to moved out of FSImage to facilite this. For this we propose a PersistanceManager? object as follows. For more deep details, see the design document uploaded. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1489) breaking the dependency between FSEditLog and FSImage
[ https://issues.apache.org/jira/browse/HDFS-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12973581#action_12973581 ] Ivan Kelly commented on HDFS-1489: -- NNStorage holds the functionality for managing namenode storage directories which is common to both FSImage and FSEditLog. FSImage handles loading, saving, upgrading, rollback etc of image files. It also owns FSEditLog in the latest patch. The logic behind this whole JIRA is that FSImage FSEditLog shouldn't be mutually dependent. The big patch removes the dependency completely. The NNStorage patch leave the dependency of FSImage on FSEditLog, but works towards removing the dependency of FSEditLog on FSImage. breaking the dependency between FSEditLog and FSImage - Key: HDFS-1489 URL: https://issues.apache.org/jira/browse/HDFS-1489 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 0.21.0 Reporter: Diego Marron Attachments: HDFS-1489.diff, HDFS-1489.diff, HDFS-1489.pdf, NNStorage.diff This is a refactor patch which its main concerns are: - breaking the dependency between FSEditLog and FSImage - Splitting the abstracting the error handling and directory management, - Decoupling Storage from FSImage. In order to accomplish the above goal, we will need to introduce new classes: - NNStorage: Will care about the storage. It extends Storage class, and will contain the StorageDirectories. - NNUtils: Some utility static methods on FSImage and FSEditLog will be moved here. - PersistenceManager: FSNameSystem will now be responsible for managing the FSImage FSEditLog objects. There will be some logic that will have to moved out of FSImage to facilite this. For this we propose a PersistanceManager? object as follows. For more deep details, see the design document uploaded. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1489) breaking the dependency between FSEditLog and FSImage
[ https://issues.apache.org/jira/browse/HDFS-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12973303#action_12973303 ] Konstantin Shvachko commented on HDFS-1489: --- Ivan, would also consider making the patch against trunk rather than 0.22. breaking the dependency between FSEditLog and FSImage - Key: HDFS-1489 URL: https://issues.apache.org/jira/browse/HDFS-1489 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 0.21.0 Reporter: Diego Marron Attachments: HDFS-1489.diff, HDFS-1489.diff, HDFS-1489.pdf This is a refactor patch which its main concerns are: - breaking the dependency between FSEditLog and FSImage - Splitting the abstracting the error handling and directory management, - Decoupling Storage from FSImage. In order to accomplish the above goal, we will need to introduce new classes: - NNStorage: Will care about the storage. It extends Storage class, and will contain the StorageDirectories. - NNUtils: Some utility static methods on FSImage and FSEditLog will be moved here. - PersistenceManager: FSNameSystem will now be responsible for managing the FSImage FSEditLog objects. There will be some logic that will have to moved out of FSImage to facilite this. For this we propose a PersistanceManager? object as follows. For more deep details, see the design document uploaded. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1489) breaking the dependency between FSEditLog and FSImage
[ https://issues.apache.org/jira/browse/HDFS-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12972944#action_12972944 ] Todd Lipcon commented on HDFS-1489: --- Hi Ivan. I like the general ideas in the patch, but as is it's too big to review, and seems to partially revert some other patches that it conflicted with over the last few weeks. Do you think we could work together to split it into two or three smaller pieces? For example, maybe we can start with just the refactor for error handling? breaking the dependency between FSEditLog and FSImage - Key: HDFS-1489 URL: https://issues.apache.org/jira/browse/HDFS-1489 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 0.21.0 Reporter: Diego Marron Attachments: HDFS-1489.diff, HDFS-1489.pdf This is a refactor patch which its main concerns are: - breaking the dependency between FSEditLog and FSImage - Splitting the abstracting the error handling and directory management, - Decoupling Storage from FSImage. In order to accomplish the above goal, we will need to introduce new classes: - NNStorage: Will care about the storage. It extends Storage class, and will contain the StorageDirectories. - NNUtils: Some utility static methods on FSImage and FSEditLog will be moved here. - PersistenceManager: FSNameSystem will now be responsible for managing the FSImage FSEditLog objects. There will be some logic that will have to moved out of FSImage to facilite this. For this we propose a PersistanceManager? object as follows. For more deep details, see the design document uploaded. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1489) breaking the dependency between FSEditLog and FSImage
[ https://issues.apache.org/jira/browse/HDFS-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12973027#action_12973027 ] Ivan Kelly commented on HDFS-1489: -- @Todd I'll try to have a look this week at possibly splitting it into smaller patches, but I'm not sure how possible it is given the interconnection between FSImage, FSEditLog FSNamesystem. Perhaps NNStorage could be submitted as a separate patch. At least that would get rid of the majority of the dependencies from FSEditLog to FSImage, and we could work from there. The comment about reverting conflicting changes worries me. Which changes in particular are you referring to? breaking the dependency between FSEditLog and FSImage - Key: HDFS-1489 URL: https://issues.apache.org/jira/browse/HDFS-1489 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 0.21.0 Reporter: Diego Marron Attachments: HDFS-1489.diff, HDFS-1489.diff, HDFS-1489.pdf This is a refactor patch which its main concerns are: - breaking the dependency between FSEditLog and FSImage - Splitting the abstracting the error handling and directory management, - Decoupling Storage from FSImage. In order to accomplish the above goal, we will need to introduce new classes: - NNStorage: Will care about the storage. It extends Storage class, and will contain the StorageDirectories. - NNUtils: Some utility static methods on FSImage and FSEditLog will be moved here. - PersistenceManager: FSNameSystem will now be responsible for managing the FSImage FSEditLog objects. There will be some logic that will have to moved out of FSImage to facilite this. For this we propose a PersistanceManager? object as follows. For more deep details, see the design document uploaded. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1489) breaking the dependency between FSEditLog and FSImage
[ https://issues.apache.org/jira/browse/HDFS-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12972577#action_12972577 ] Ivan Kelly commented on HDFS-1489: -- I forgot to mention. The patch applies on top of branch-0.22. breaking the dependency between FSEditLog and FSImage - Key: HDFS-1489 URL: https://issues.apache.org/jira/browse/HDFS-1489 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 0.21.0 Reporter: Diego Marron Attachments: HDFS-1489.diff, HDFS-1489.pdf This is a refactor patch which its main concerns are: - breaking the dependency between FSEditLog and FSImage - Splitting the abstracting the error handling and directory management, - Decoupling Storage from FSImage. In order to accomplish the above goal, we will need to introduce new classes: - NNStorage: Will care about the storage. It extends Storage class, and will contain the StorageDirectories. - NNUtils: Some utility static methods on FSImage and FSEditLog will be moved here. - PersistenceManager: FSNameSystem will now be responsible for managing the FSImage FSEditLog objects. There will be some logic that will have to moved out of FSImage to facilite this. For this we propose a PersistanceManager? object as follows. For more deep details, see the design document uploaded. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1489) breaking the dependency between FSEditLog and FSImage
[ https://issues.apache.org/jira/browse/HDFS-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12928768#action_12928768 ] Todd Lipcon commented on HDFS-1489: --- big +1, I was trying to work on this last week as well, but it gets really hairy. HDFS-1473 is also related - I had a patch for that one, but HDFS-1435 complicated stuff a bit more. breaking the dependency between FSEditLog and FSImage - Key: HDFS-1489 URL: https://issues.apache.org/jira/browse/HDFS-1489 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 0.21.0 Reporter: Diego Marron Attachments: HDFS-1489.pdf This is a refactor patch which its main concerns are: - breaking the dependency between FSEditLog and FSImage - Splitting the abstracting the error handling and directory management, - Decoupling Storage from FSImage. In order to accomplish the above goal, we will need to introduce new classes: - NNStorage: Will care about the storage. It extends Storage class, and will contain the StorageDirectories. - NNUtils: Some utility static methods on FSImage and FSEditLog will be moved here. - PersistenceManager: FSNameSystem will now be responsible for managing the FSImage FSEditLog objects. There will be some logic that will have to moved out of FSImage to facilite this. For this we propose a PersistanceManager? object as follows. For more deep details, see the design document uploaded. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
