[
https://issues.apache.org/jira/browse/HDFS-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13223934#comment-13223934
]
Colin Patrick McCabe commented on HDFS-3004:
--------------------------------------------
One thing that I would like some suggestions about is how to handle the
assertion in FSEditLog: "Safety check: we should never start a segment if there
are newer txids readable."
As you can see, this is a problem after we have done a successful recovery
where we lost some edits. We will have edit logs in the namenode directory
that "seem" to have a higher txid than the current txid. This would trigger
the assert (if I hadn't commented it out.)
I commented out this assert in the latest patch, because I wasn't sure exactly
how to handle it. Here are some ways I thought of to do it:
1. Bump up the current FSImage txid to be equal to the highest txid the edit
logs appear to contain (based on their filenames). This will have the side
effect that there appears to be a hole in the edit log history. However, there
is probably already a hole in the history because of the corrupted file.
2. Delete the old corrupted edit logs after recovery, so that we can start
without encountering this assert
3. Rename the old corrupted edit logs after recovery, so that we can start
without encountering this assert
I'm leaning towards solution #1 right now.
> Implement Recovery Mode
> -----------------------
>
> Key: HDFS-3004
> URL: https://issues.apache.org/jira/browse/HDFS-3004
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: tools
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Attachments: HDFS-3004.005.patch, HDFS-3004.patch,
> HDFS-3004__namenode_recovery_tool.txt
>
>
> When the NameNode metadata is corrupt for some reason, we want to be able to
> fix it. Obviously, we would prefer never to get in this case. In a perfect
> world, we never would. However, bad data on disk can happen from time to
> time, because of hardware errors or misconfigurations. In the past we have
> had to correct it manually, which is time-consuming and which can result in
> downtime.
> Recovery mode is initialized by the system administrator. When the NameNode
> starts up in Recovery Mode, it will try to load the FSImage file, apply all
> the edits from the edits log, and then write out a new image. Then it will
> shut down.
> Unlike in the normal startup process, the recovery mode startup process will
> be interactive. When the NameNode finds something that is inconsistent, it
> will prompt the operator as to what it should do. The operator can also
> choose to take the first option for all prompts by starting up with the '-f'
> flag, or typing 'a' at one of the prompts.
> I have reused as much code as possible from the NameNode in this tool.
> Hopefully, the effort that was spent developing this will also make the
> NameNode editLog and image processing even more robust than it already is.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
