[
https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13003106#comment-13003106
]
Todd Lipcon commented on HDFS-1580:
-----------------------------------
Hi Ivan. I'm trying to figure out how this work will fit in with the
log-transfer step between checkpointers and the primary NN. Do you anticipate
that these JournalManager implementations will be instantiated on the
checkpointer side as well? Or is the transfer method orthogonal to the storage
method, and this JIRA is only about the NN side of things?
It seems to me they have to be orthogonal concerns, at least initially, if you
plan to be able to have two separate "types" of edit logs on the NN. So, even
if the NN is using both BK and files, a 2NN would still use HTTP to fetch logs
at checkpoint time.
In working on HDFS-1073, I'm thinking the interface for log transfer will look
something like:
- add a new RPC method to the NN corresponding to your getLogs(long sinceTxId).
This would return something like List<RemoteLog>, where RemoteLog has some
method which takes care of fetching.
- for now, when the 2NN wants to checkpoint, it initiates rollEdits() which
returns CheckpointSignature. This would be modified to include the
lastCheckpointTxId field
- the 2NN then issues getLogs(sig.lastCheckpointTxId), iterates over each of
the RemoteLogs, and fetches them
Does this gel with what you're planning?
> Add interface for generic Write Ahead Logging mechanisms
> --------------------------------------------------------
>
> Key: HDFS-1580
> URL: https://issues.apache.org/jira/browse/HDFS-1580
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Ivan Kelly
> Attachments: HDFS-1580+1521.diff, HDFS-1580.diff,
> generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt
>
>
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
