[
https://issues.apache.org/jira/browse/HDFS-2874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13199031#comment-13199031
]
Todd Lipcon commented on HDFS-2874:
-----------------------------------
bq. Adding any type of write ordering to edit log updates will increase commit
latency. For example, writing to shared edit log first, waiting for this to
complete and then writing to local dirs will double the write latency.
We already _do_ write to each volume serially, so the latency is no different.
I agree there is a potential optimization to do them in parallel, but since
we've been living fine without for years, I don't think it's on the critical
path for most installations.
bq. A better design is to write all the locations in parallel and then make a
policy decision during recovery as to go with higher txid or the lower one.
Except that the remote NN can't see the local edit dirs, so no such policy
exists.
> HA: edit log should log to shared dirs before local dirs
> --------------------------------------------------------
>
> Key: HDFS-2874
> URL: https://issues.apache.org/jira/browse/HDFS-2874
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: hdfs-2874.txt
>
>
> Currently, the NN logs its edits to each of its edits directories in
> sequence. This can produce the following bad sequence:
> - NN accumulates 100 edits (tx 1-100) in the buffer. Writes and syncs to
> local drive, then crashes
> - Failover occurs. SBN takes over at txid=1, since txid 1 never got writen.
> - First NN restarts. It reads up to txid 100 from its local directories. It
> is now "ahead" of the active NN with inconsistent state.
> The solution is to write to the shared edits dir, and sync that, before
> writing to any local drives.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
