[
https://issues.apache.org/jira/browse/HDFS-3077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13230432#comment-13230432
]
Andrew Purtell commented on HDFS-3077:
--------------------------------------
>From a user perspective.
bq. [Todd] I think a quorum commit is vastly superior for HA, especially given
we'd like to collocate the log replicas on machines doing other work. When
those machines have latency hiccups, or crash, we don't want the active NN to
have to wait for long timeout periods before continuing.
I think this is a promising direction. See next:
bq. [Eli] BK has two of the same main issues that we have depending on a an HA
filer: (1) many users don't want to admin a separate storage system (even if
you "embed" BK it will be discrete, fail independently etc)
Perhaps we can go so far as to suggest the loggers be an additional thread
added to the DataNodes. Perhaps some subset of the DN pool is elected for the
purpose. (Need we waste a whole disk just for the transaction log? Maybe the
log can be shared with DN storage. Or using a SSD device for this purpose seems
reasonable but the average user should not be expected to have nodes with such
on hand.) On the one hand, this would increase the internal complexity of the
DataNode implementation, even if the functionality can be pretty well
partitioned -- separate package, separate thread, etc. On the other hand, there
would be not yet another moving part to consider when deploying components
around the cluster: ZooKeeper quorum peers, NameNodes, DataNodes, the YARN AM,
the Yarn NMs, HBase Masters, HBase RegionServers etc. etc. etc.
This idea may go too far, but IMHO embedding BookKeeper goes enough in the
other direction to give me heartburn thinking about HA cluster ops.
> Quorum-based protocol for reading and writing edit logs
> -------------------------------------------------------
>
> Key: HDFS-3077
> URL: https://issues.apache.org/jira/browse/HDFS-3077
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: ha, name-node
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
>
> Currently, one of the weak points of the HA design is that it relies on
> shared storage such as an NFS filer for the shared edit log. One alternative
> that has been proposed is to depend on BookKeeper, a ZooKeeper subproject
> which provides a highly available replicated edit log on commodity hardware.
> This JIRA is to implement another alternative, based on a quorum commit
> protocol, integrated more tightly in HDFS and with the requirements driven
> only by HDFS's needs rather than more generic use cases. More details to
> follow.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
