[
https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13009512#comment-13009512
]
Jitendra Nath Pandey commented on HDFS-1580:
--------------------------------------------
A few comments.
1. > JournalManager#transfer()
Is this a push method to be executed on primary or a pull from secondary? The
mechanism as we have today, which is more like a pull, can be provided by the
interface using getLogs and getInputStream methods, which, in my opinion, is
sufficient.
2. We could use single BookKeeper installation (or any shared storage solution)
for storing edit logs from multiple namenodes (e.g. federation). In that case a
secondary/backup should be able to instantiate a JournalManager and request
logs for a specific namenode. Similar could be argued about the files on
namenodes, where backup or secondary wants to specify the namenode where it
wants to download the files from.
As I understand the steps to get edit logs at the backup node or secondary
would be
- Instantiate a JournalManager
- specify the source of edit logs. This could be specified in the
JournalManager constructor. In case of BK the source would be the set of
ledgers for a specific namenode.
- getLogs from sinceTransactionId. It returns a list of URIs.
- For each uri get the input stream and read the logs.
3. For each edit log we need to store some metadata, particularly the
LAYOUT_VERSION. I think it would make sense to provide for that in
JournalManager interface.
4. For edit logs do we really need upgrade, rollback or finalizeUpgrade? For
edit log files, it seems to me that all that matters is whether the namenode
has right version to understand them or load them, which can be determined from
LAYOUT_VERSION.
5. I will recommend putting JournalManager and its derived classes in separate
packages.
6. Please add javadoc comments on JournalManager class and its methods.
> Add interface for generic Write Ahead Logging mechanisms
> --------------------------------------------------------
>
> Key: HDFS-1580
> URL: https://issues.apache.org/jira/browse/HDFS-1580
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Ivan Kelly
> Attachments: HDFS-1580+1521.diff, HDFS-1580.diff,
> generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
