[
https://issues.apache.org/jira/browse/HDFS-5477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13869798#comment-13869798
]
Colin Patrick McCabe commented on HDFS-5477:
--------------------------------------------
Hi Daryn,
This seems like a great direction for HDFS to go in. Just a few comments.
You list scalability as a primary concern. However, even if we separate the BM
from the namespace management, a cluster with a large number of blocks will
still have a giant BM heap (if I understand correctly). So perhaps what we
need is the ability to have multiple block manager daemons?
It seems like there will be a lot of messages that will necessarily flow
between the namespace daemon and the block management daemon(s). What IPC
mechanism are you considering? TCP socket? UNIX domain socket? Shared
memory? Shared memory would clearly be the highest performance, and perhaps we
should consider that.
Is there an upstream svn branch for this yet?
> Block manager as a service
> --------------------------
>
> Key: HDFS-5477
> URL: https://issues.apache.org/jira/browse/HDFS-5477
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Affects Versions: 2.0.0-alpha, 3.0.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: Proposal.pdf, Proposal.pdf, Standalone BM.pdf,
> Standalone BM.pdf
>
>
> The block manager needs to evolve towards having the ability to run as a
> standalone service to improve NN vertical and horizontal scalability. The
> goal is reducing the memory footprint of the NN proper to support larger
> namespaces, and improve overall performance by decoupling the block manager
> from the namespace and its lock. Ideally, a distinct BM will be transparent
> to clients and DNs.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
