[jira] [Updated] (HDFS-3190) Simple refactors in existing NN code to assist QuorumJournalManager extension
[ https://issues.apache.org/jira/browse/HDFS-3190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-3190: -- Fix Version/s: 2.2.0-alpha Backported this to branch-2, since it was causing some conflicts in other backports, and it's a straight refactor. Simple refactors in existing NN code to assist QuorumJournalManager extension - Key: HDFS-3190 URL: https://issues.apache.org/jira/browse/HDFS-3190 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Affects Versions: 2.0.0-alpha Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Fix For: 3.0.0, 2.2.0-alpha Attachments: hdfs-3190.txt, hdfs-3190.txt, hdfs-3190.txt, hdfs-3190.txt, hdfs-3190.txt This JIRA is for some simple refactors in the NN: - refactor the code which writes the seen_txid file in NNStorage into a new LongContainingFile utility class. This is useful for the JournalNode to atomically/durably record its last promised epoch - refactor the interface from FileJournalManager back to StorageDirectory to use a StorageErrorReport interface. This allows FileJournalManager to be used in isolation of a full StorageDirectory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3190) Simple refactors in existing NN code to assist QuorumJournalManager extension
[
https://issues.apache.org/jira/browse/HDFS-3190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated HDFS-3190:
--
Attachment: hdfs-3190.txt
Thanks for the review, Suresh. Here's an updated patch.
- Fixed the 80+ long lines.
- Improved javadoc on StorageErrorReporter
- Renamed PersistentLong to PersistentLongFile
I also removed {{reportErrorOnDirectory}} from the error reporting interface,
in order to simplify it. The only caller was FileJournalManager, and it was
easy to change it to just report on the actual files that had an error. Seemed
worth simplifying the API here.
Simple refactors in existing NN code to assist QuorumJournalManager extension
-
Key: HDFS-3190
URL: https://issues.apache.org/jira/browse/HDFS-3190
Project: Hadoop HDFS
Issue Type: Sub-task
Components: name-node
Affects Versions: 2.0.0-alpha
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Minor
Attachments: hdfs-3190.txt, hdfs-3190.txt, hdfs-3190.txt,
hdfs-3190.txt, hdfs-3190.txt
This JIRA is for some simple refactors in the NN:
- refactor the code which writes the seen_txid file in NNStorage into a new
LongContainingFile utility class. This is useful for the JournalNode to
atomically/durably record its last promised epoch
- refactor the interface from FileJournalManager back to StorageDirectory to
use a StorageErrorReport interface. This allows FileJournalManager to be used
in isolation of a full StorageDirectory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3190) Simple refactors in existing NN code to assist QuorumJournalManager extension
[ https://issues.apache.org/jira/browse/HDFS-3190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-3190: -- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Committed to trunk only for now. We can backport the HDFS-3077-related changes en masse when it's all ready in trunk. Simple refactors in existing NN code to assist QuorumJournalManager extension - Key: HDFS-3190 URL: https://issues.apache.org/jira/browse/HDFS-3190 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Affects Versions: 2.0.0-alpha Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Fix For: 3.0.0 Attachments: hdfs-3190.txt, hdfs-3190.txt, hdfs-3190.txt, hdfs-3190.txt, hdfs-3190.txt This JIRA is for some simple refactors in the NN: - refactor the code which writes the seen_txid file in NNStorage into a new LongContainingFile utility class. This is useful for the JournalNode to atomically/durably record its last promised epoch - refactor the interface from FileJournalManager back to StorageDirectory to use a StorageErrorReport interface. This allows FileJournalManager to be used in isolation of a full StorageDirectory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3190) Simple refactors in existing NN code to assist QuorumJournalManager extension
[
https://issues.apache.org/jira/browse/HDFS-3190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated HDFS-3190:
--
Attachment: hdfs-3190.txt
Attaching new draft of this patch:
- I agreed with Colin that LongContainingFile was a bad name. I ended up
renaming to PersistentLong, and clarified with JavaDoc what its purpose is. I
also made it instantiable as a wrapper which holds a persisted long value --
this was useful in development of HDFS-3077 in order to hold the promised
epoch persistent across restarts.
- Instead of introducing StorageErrorReporter, I just moved the error reporting
functionality up into {{Storage}} instead of {{NNStorage}}. It seems like a
generally useful thing -- in the future we may want to consolidate the
error-tracking functionality between the DN and NN using this mechanism, for
example. For now, the {{Storage}} implementation just logs the errors.
- Change TransferFsImage to now take a Storage instead of NNStorage. This is so
that in HDFS-3077, we can download logs into a new {{JNStorage}} class.
- Move {{getFiles()}} from {{NNStorage}} into {{Storage}} since it's also
generally useful and not NN-specific.
- Some minor refactor in {{TransferFsImage]} to make code more re-usable (also
used for edits transfer in HDFS-3077).
While extracting PersistentLong, I also noticed a bug that, if there were an
IOE while writing the file, it would still attempt to close the
AtomicFileOutputStream. This could cause the incompletely written value to get
incorrectly committed. I added a simple abort function for this.
Simple refactors in existing NN code to assist QuorumJournalManager extension
-
Key: HDFS-3190
URL: https://issues.apache.org/jira/browse/HDFS-3190
Project: Hadoop HDFS
Issue Type: Sub-task
Components: name-node
Affects Versions: 2.0.0-alpha
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Minor
Attachments: hdfs-3190.txt, hdfs-3190.txt, hdfs-3190.txt
This JIRA is for some simple refactors in the NN:
- refactor the code which writes the seen_txid file in NNStorage into a new
LongContainingFile utility class. This is useful for the JournalNode to
atomically/durably record its last promised epoch
- refactor the interface from FileJournalManager back to StorageDirectory to
use a StorageErrorReport interface. This allows FileJournalManager to be used
in isolation of a full StorageDirectory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3190) Simple refactors in existing NN code to assist QuorumJournalManager extension
[ https://issues.apache.org/jira/browse/HDFS-3190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-3190: -- Attachment: hdfs-3190.txt Actually, went back to the original idea of a StorageErrorReporter interface. It makes the dependencies clearer between JournalManager and the Storage, which is nice when trying to unit-test a JournalManager in isolation. Otherwise this patch should be the same. Simple refactors in existing NN code to assist QuorumJournalManager extension - Key: HDFS-3190 URL: https://issues.apache.org/jira/browse/HDFS-3190 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Affects Versions: 2.0.0-alpha Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Attachments: hdfs-3190.txt, hdfs-3190.txt, hdfs-3190.txt, hdfs-3190.txt This JIRA is for some simple refactors in the NN: - refactor the code which writes the seen_txid file in NNStorage into a new LongContainingFile utility class. This is useful for the JournalNode to atomically/durably record its last promised epoch - refactor the interface from FileJournalManager back to StorageDirectory to use a StorageErrorReport interface. This allows FileJournalManager to be used in isolation of a full StorageDirectory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3190) Simple refactors in existing NN code to assist QuorumJournalManager extension
[ https://issues.apache.org/jira/browse/HDFS-3190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-3190: -- Attachment: hdfs-3190.txt Re-attaching to get a fresh QA run, since it's been a couple months. Simple refactors in existing NN code to assist QuorumJournalManager extension - Key: HDFS-3190 URL: https://issues.apache.org/jira/browse/HDFS-3190 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Affects Versions: 2.0.0-alpha Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Attachments: hdfs-3190.txt, hdfs-3190.txt This JIRA is for some simple refactors in the NN: - refactor the code which writes the seen_txid file in NNStorage into a new LongContainingFile utility class. This is useful for the JournalNode to atomically/durably record its last promised epoch - refactor the interface from FileJournalManager back to StorageDirectory to use a StorageErrorReport interface. This allows FileJournalManager to be used in isolation of a full StorageDirectory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3190) Simple refactors in existing NN code to assist QuorumJournalManager extension
[ https://issues.apache.org/jira/browse/HDFS-3190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-3190: -- Attachment: hdfs-3190.txt Simple patch implements the above. Does not add unit tests since it's a straight refactor of existing code, and that code is covered by many existing tests. Simple refactors in existing NN code to assist QuorumJournalManager extension - Key: HDFS-3190 URL: https://issues.apache.org/jira/browse/HDFS-3190 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Affects Versions: 2.0.0 Reporter: Todd Lipcon Priority: Minor Attachments: hdfs-3190.txt This JIRA is for some simple refactors in the NN: - refactor the code which writes the seen_txid file in NNStorage into a new LongContainingFile utility class. This is useful for the JournalNode to atomically/durably record its last promised epoch - refactor the interface from FileJournalManager back to StorageDirectory to use a StorageErrorReport interface. This allows FileJournalManager to be used in isolation of a full StorageDirectory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3190) Simple refactors in existing NN code to assist QuorumJournalManager extension
[ https://issues.apache.org/jira/browse/HDFS-3190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-3190: -- Assignee: Todd Lipcon Status: Patch Available (was: Open) Simple refactors in existing NN code to assist QuorumJournalManager extension - Key: HDFS-3190 URL: https://issues.apache.org/jira/browse/HDFS-3190 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Affects Versions: 2.0.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Attachments: hdfs-3190.txt This JIRA is for some simple refactors in the NN: - refactor the code which writes the seen_txid file in NNStorage into a new LongContainingFile utility class. This is useful for the JournalNode to atomically/durably record its last promised epoch - refactor the interface from FileJournalManager back to StorageDirectory to use a StorageErrorReport interface. This allows FileJournalManager to be used in isolation of a full StorageDirectory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
