[
https://issues.apache.org/jira/browse/HDFS-5477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13931095#comment-13931095
]
Edward Bortnikov commented on HDFS-5477:
----------------------------------------
All great questions. Our previous documentation was pretty substandard.
New design PDF attached at https://issues.apache.org/jira/browse/HDFS-5732 - it
clarifies many things about the remote NN operation.
Regarding Todd's question specifically. Yes, it's impossible to guarantee the
100% atomicity of transactions in the face of failures. This atomicity is also
not necessarily required as long as no data is lost. The distributed state must
eventually converge.
Our solution is to treat communication failures and process failures
identically. If an RPC times out, we re-establish the NN-BM connection and
re-synchronize the state. (There are many ways to optimize this process, e.g.,
Merkle trees). Since timeouts should be very rare in a datacenter network (in
the absence of bugs), this treatment is not too harsh.
> Block manager as a service
> --------------------------
>
> Key: HDFS-5477
> URL: https://issues.apache.org/jira/browse/HDFS-5477
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Affects Versions: 2.0.0-alpha, 3.0.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: Proposal.pdf, Proposal.pdf, Standalone BM.pdf,
> Standalone BM.pdf, patches.tar.gz
>
>
> The block manager needs to evolve towards having the ability to run as a
> standalone service to improve NN vertical and horizontal scalability. The
> goal is reducing the memory footprint of the NN proper to support larger
> namespaces, and improve overall performance by decoupling the block manager
> from the namespace and its lock. Ideally, a distinct BM will be transparent
> to clients and DNs.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
