[
https://issues.apache.org/jira/browse/HDFS-2709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13176098#comment-13176098
]
Aaron T. Myers commented on HDFS-2709:
--------------------------------------
Another option I'd like to put forth would be to separate the reading of edit
log ops off disk from the actual application of those ops to the in-memory FS
state. The first stage of the process would just read all edit log ops from
disk and put them in a queue. The second stage would go through the queue and
apply all the edits.
If an error occurs during this stage, the standby NN would log a warning and
continue on, since this is a potentially normal operating condition. The first
stage is idempotent, and can safely be retried at a later time. This will allow
us to queue up an entire file's worth of edits, guaranteed. If an error occurs
in the second stage, we abort the standby NN, since this is indicative of a
corrupt FS, and should not occur in practice.
Thoughts?
> HA: Appropriately handle error conditions in EditLogTailer
> ----------------------------------------------------------
>
> Key: HDFS-2709
> URL: https://issues.apache.org/jira/browse/HDFS-2709
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Todd Lipcon
> Assignee: Aaron T. Myers
> Priority: Critical
>
> Currently if the edit log tailer experiences an error replaying edits in the
> middle of a file, it will go back to retrying from the beginning of the file
> on the next tailing iteration. This is incorrect since many of the edits will
> have already been replayed, and not all edits are idempotent.
> Instead, we either need to (a) support reading from the middle of a finalized
> file (ie skip those edits already applied), or (b) abort the standby if it
> hits an error while tailing. If "a" isn't simple, let's do "b" for now and
> come back to 'a' later since this is a rare circumstance and better to abort
> than be incorrect.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
