[ https://issues.apache.org/jira/browse/HDFS-1073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13071462#comment-13071462 ]
Todd Lipcon commented on HDFS-1073: ----------------------------------- I see your point now about the protocol naming. I'll changing it JournalProtocol. bq. Document the architecture is important as it is the proof of correctness of the approach If writing documents about code guaranteed the code were correct, our jobs would be a lot easier, wouldn't they? :) But yes, I'll clean up the doc after merging to make sure there's nothing inaccurate. bq. I hope "longer" does not mean file length? In the case that the only logs available starting at a given transaction ID are named edits_inprogress_N, then we read through them to determine the "valid length" -- ie the number of valid transactions. A transaction is valid if it has a valid checksum, sequential transaction ID, etc. The one with the most valid transactions is chosen. So, extra 0s or FFs on the end of a file won't affect the "valid length". bq. Do you attempt to restore bad streams on rollEdits() as done by attemptRestoreRemovedStorage() in current implementation? Yes -- each JournalManager creats a new OutputStream object when edits are rolled. bq. ...OP_JSPOOL_START... Yep, this is entirely eliminated now. {quote} Is it possible in your implementation that a) BN already processed transactions with higher id than segmentTxId b) BN hasn't seen yet transaction preceding segmentTxId According to Precondition this should not be possible. What guarantees that? {quote} Because all of the calls to the BN go through JournalManager, and all of the calls are synchronous, the ordering won't get interleaved. That is to say, when an edit log is rolled, the startLogSegment() RPC call must respond before the next transaction can be journaled. And, before calling startLogSegment(), the previous log segment is flushed, guaranteeing that all previous edits "made it". The Precondition is there just in case there's a bug that we missed -- this way we'll get a BN crash rather than something worse like silent data loss. > Simpler model for Namenode's fs Image and edit Logs > ---------------------------------------------------- > > Key: HDFS-1073 > URL: https://issues.apache.org/jira/browse/HDFS-1073 > Project: Hadoop HDFS > Issue Type: Improvement > Affects Versions: 0.23.0 > Reporter: Sanjay Radia > Assignee: Todd Lipcon > Fix For: 0.23.0 > > Attachments: hdfs-1073-editloading-algos.txt, hdfs-1073-merge.patch, > hdfs-1073-merge.patch, hdfs-1073.txt, hdfs1073.pdf, hdfs1073.pdf, > hdfs1073.pdf, hdfs1073.tex > > > The naming and handling of NN's fsImage and edit logs can be significantly > improved resulting simpler and more robust code. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira