[
https://issues.apache.org/jira/browse/HDFS-3540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443002#comment-13443002
]
Tsz Wo (Nicholas), SZE commented on HDFS-3540:
----------------------------------------------
If the edit log is not corrupted, both recovery mode and edit log toleration
are not useful. Note that recovery mode here means recovery mode in branch-1
but not the one in trunk.
When an edit log is corrupted, NN cannot start up normally. We compare
recovery mode and edit log toleration below.
*Recovery Mode*
- "Recovery" here means starting NN with a corrupted edit log. It is unable to
recover the corrupted edit log or transactions.
- There is a namenode command option "hadoop namenode -recover" to enter
recovery mode.
- When reading the first corrupted transaction in the edit log, it prompts the
admin to either "stop reading" or "quit without saving".
- If "stop reading" is selected, NN ignores the remaining edit log (from the
first corrupted transaction to the end of the edit log) and then starts up as
usual.
- There is a "-force" option to FORCE_FIRST_CHOICE, i.e. it is a
non-interactive mode.
- If there is a stray OP_INVALID byte, it could be misinterpreted as an
end-of-log and lead to silent data loss. Recovery Mode does not help.
(Please help out if I have missed anything.)
*Edit Log Toleration*
- It has a conf property "dfs.namenode.edits.toleration.length" for setting the
toleration length.
- The default toleration length is -1, i.e. disable it. The feature is enabled
when the value >= 0.
- When the feature is enabled, it always reads the entire edit log, computes
read length, corruption length and padding length and shows the following
summary
{noformat}
2012-08-27 22:04:38,625 INFO - Checked the bytes after the end of edit log
(/Users/szetszwo/hadoop/b-1/build/test/data/dfs/name1/current/edits):
2012-08-27 22:04:38,625 INFO - Padding position = 876 (-1 means padding not
found)
2012-08-27 22:04:38,625 INFO - Edit log length = 1065
2012-08-27 22:04:38,625 INFO - Read length = 168
2012-08-27 22:04:38,625 INFO - Corruption length = 708
2012-08-27 22:04:38,625 INFO - Toleration length = 1024 (=
dfs.namenode.edits.toleration.length)
2012-08-27 22:04:38,626 INFO - Summary: |---------- Read=168 ----------|--
Corrupt=708 --|-- Pad=189 --|
2012-08-27 22:04:38,626 WARN - Edit log corruption detected:
corruption length = 708 <= toleration length = 1024; the corruption is
tolerable.
{noformat}
- When toleration length is set to 0, it makes sure that there is no corruption
in the entire log, including the padding. A stray OP_INVALID byte won't be
misinterpreted as an end-of-log.
- When toleration length is set to >0, NN starts up only if corruption length
<= toleration length. If corruption length > toleration length, it throws an
exception as below
{noformat}
2012-08-27 22:04:39,123 INFO - Start checking end of edit log
(/Users/szetszwo/hadoop/b-1/build/test/data/dfs/name1/current/edits) ...
2012-08-27 22:04:39,123 DEBUG - found: bytes[0]=0xFF=pad, firstPadPos=169
2012-08-27 22:04:39,123 DEBUG - reset: bytes[1410]=0xAB, pad=0xFF
2012-08-27 22:04:39,124 DEBUG - found: bytes[1411]=0xFF=pad, firstPadPos=1580
2012-08-27 22:04:39,124 INFO - Checked the bytes after the end of edit log
(/Users/szetszwo/hadoop/b-1/build/test/data/dfs/name1/current/edits):
2012-08-27 22:04:39,124 INFO - Padding position = 1580 (-1 means padding
not found)
2012-08-27 22:04:39,124 INFO - Edit log length = 2638
2012-08-27 22:04:39,124 INFO - Read length = 169
2012-08-27 22:04:39,124 INFO - Corruption length = 1411
2012-08-27 22:04:39,124 INFO - Toleration length = 1024 (=
dfs.namenode.edits.toleration.length)
2012-08-27 22:04:39,125 INFO - Summary: |---------- Read=169 ----------|--
Corrupt=1411 --|-- Pad=1058 --|
2012-08-27 22:04:39,125 ERROR - FSNamesystem initialization failed.
java.io.IOException: Edit log corruption detected:
corruption length = 1411 > toleration length = 1024; the corruption is
intolerable.
at
org.apache.hadoop.hdfs.server.namenode.FSEditLog.checkEndOfLog(FSEditLog.java:609)
...
{noformat}
- Therefore, the recommanded setting is to set the conf to 0 (or a small
number). When corruption is detected (i.e. NN cannot start up), the corruption
length can be read from the log. Then, admin could decide whether they are
willing to tolerate the corruption or they could try to recover the edit log by
other means.
> Further improvement on recovery mode and edit log toleration in branch-1
> ------------------------------------------------------------------------
>
> Key: HDFS-3540
> URL: https://issues.apache.org/jira/browse/HDFS-3540
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 1.2.0
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
>
> *Recovery Mode*: HDFS-3479 backported HDFS-3335 to branch-1. However, the
> recovery mode feature in branch-1 is dramatically different from the recovery
> mode in trunk since the edit log implementations in these two branch are
> different. For example, there is UNCHECKED_REGION_LENGTH in branch-1 but not
> in trunk.
> *Edit Log Toleration*: HDFS-3521 added this feature to branch-1 to remedy
> UNCHECKED_REGION_LENGTH and to tolerate edit log corruption.
> There are overlaps between these two features. We study potential further
> improvement in this issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
