[
https://issues.apache.org/jira/browse/HDFS-1073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12930023#action_12930023
]
Konstantin Shvachko commented on HDFS-1073:
-------------------------------------------
Rob, I seems we cannot have the "I'm quitting!" record just because there is no
"quit" or "shutdown" command. I agree the "rolled" transaction can be useful
for a sanity check for the edits files that are not in-progress.
Todd, based on the design doc (I should have read first thing) I don't see much
difference between the current and your new implementation ascept that you
don't need a side file to write the edits while spooling. Currently
BN.startCheckpoint() causes NN.rollEdits(), which in turn sends back to BN the
SPOOL_START record. This is when BN starts spooling. You seem to be trying to
call the process of spooling (writing into edits file but not applying to
memory) by journal. That is how the state is called in your design, right?
Which may be confusing as BN continues journaling (writing to edits file)
whether it is in synchronized or in spooling mode.
Also I don't see how you can get by with only 2 states for BN you need 3. While
spooling there are 2 active threads: one (writer) is writing edits from NN
directly to the edits_K, another (reader) is reading formerly written records
from edits_K. At the end we need to switch the writer thread from writing to
applying the records to in memory state and shut down the second thread. This
is where you need the third state, currently called WAIT. When the reader
thread reaches the end of file it sets the WAIT state. The writer may still be
writing before it sees the WAIT state. After seeing WAIT it blocks and waits
until spooling is OFF. The reader read the remaining records and turns the
spooling OFF.
Let me summarize the current journal spool states meaning
- JSpoolState.OFF - spooling is off, apply edits to memory state and write into
journal (edits file)
- JSpoolState.INPROGRESS - spooling in progress, do not apply to memory, just
journal
- JSpoolState.WAIT - stop, do nothing wait until spooling is OFF
Does that make sense?
> Simpler model for Namenode's fs Image and edit Logs
> ----------------------------------------------------
>
> Key: HDFS-1073
> URL: https://issues.apache.org/jira/browse/HDFS-1073
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Sanjay Radia
> Assignee: Todd Lipcon
> Attachments: hdfs-1073.txt, hdfs1073.pdf
>
>
> The naming and handling of NN's fsImage and edit logs can be significantly
> improved resulting simpler and more robust code.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
