[jira] [Updated] (HDFS-5477) Block manager as a service
[ https://issues.apache.org/jira/browse/HDFS-5477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amir Langer updated HDFS-5477: -- Attachment: (was: patches.tar.gz) > Block manager as a service > -- > > Key: HDFS-5477 > URL: https://issues.apache.org/jira/browse/HDFS-5477 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: Block-Manager-as-a-Service.pdf, Proposal.pdf, > Proposal.pdf, Standalone BM.pdf, Standalone BM.pdf > > > The block manager needs to evolve towards having the ability to run as a > standalone service to improve NN vertical and horizontal scalability. The > goal is reducing the memory footprint of the NN proper to support larger > namespaces, and improve overall performance by decoupling the block manager > from the namespace and its lock. Ideally, a distinct BM will be transparent > to clients and DNs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-5477) Block manager as a service
[ https://issues.apache.org/jira/browse/HDFS-5477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward Bortnikov updated HDFS-5477: --- Attachment: Block-Manager-as-a-Service.pdf End-to-end specification and design of the Block-Manager-as-a-service feature, addressing the issues raised in the past discussions. > Block manager as a service > -- > > Key: HDFS-5477 > URL: https://issues.apache.org/jira/browse/HDFS-5477 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: Block-Manager-as-a-Service.pdf, Proposal.pdf, > Proposal.pdf, Standalone BM.pdf, Standalone BM.pdf > > > The block manager needs to evolve towards having the ability to run as a > standalone service to improve NN vertical and horizontal scalability. The > goal is reducing the memory footprint of the NN proper to support larger > namespaces, and improve overall performance by decoupling the block manager > from the namespace and its lock. Ideally, a distinct BM will be transparent > to clients and DNs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-5477) Block manager as a service
[ https://issues.apache.org/jira/browse/HDFS-5477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amir Langer updated HDFS-5477: -- Attachment: (was: Remote BM.pdf) > Block manager as a service > -- > > Key: HDFS-5477 > URL: https://issues.apache.org/jira/browse/HDFS-5477 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: Block-Manager-as-a-Service.pdf, Proposal.pdf, > Proposal.pdf, Standalone BM.pdf, Standalone BM.pdf > > > The block manager needs to evolve towards having the ability to run as a > standalone service to improve NN vertical and horizontal scalability. The > goal is reducing the memory footprint of the NN proper to support larger > namespaces, and improve overall performance by decoupling the block manager > from the namespace and its lock. Ideally, a distinct BM will be transparent > to clients and DNs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-5477) Block manager as a service
[ https://issues.apache.org/jira/browse/HDFS-5477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amir Langer updated HDFS-5477: -- Attachment: (was: Block Manager as a Service - Implementation decisions.pdf) > Block manager as a service > -- > > Key: HDFS-5477 > URL: https://issues.apache.org/jira/browse/HDFS-5477 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: Proposal.pdf, Proposal.pdf, Remote BM.pdf, Standalone > BM.pdf, Standalone BM.pdf, patches.tar.gz > > > The block manager needs to evolve towards having the ability to run as a > standalone service to improve NN vertical and horizontal scalability. The > goal is reducing the memory footprint of the NN proper to support larger > namespaces, and improve overall performance by decoupling the block manager > from the namespace and its lock. Ideally, a distinct BM will be transparent > to clients and DNs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-5477) Block manager as a service
[ https://issues.apache.org/jira/browse/HDFS-5477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amir Langer updated HDFS-5477: -- Attachment: Block Manager as a Service - Implementation decisions.pdf Attached some more documentation about our design and implementation decisions in the attached patches. (Block Manager as a Service - Implementation decisions.pdf) This document should complement the RemoteBM.pdf design doc. > Block manager as a service > -- > > Key: HDFS-5477 > URL: https://issues.apache.org/jira/browse/HDFS-5477 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: Block Manager as a Service - Implementation > decisions.pdf, Proposal.pdf, Proposal.pdf, Remote BM.pdf, Standalone BM.pdf, > Standalone BM.pdf, patches.tar.gz > > > The block manager needs to evolve towards having the ability to run as a > standalone service to improve NN vertical and horizontal scalability. The > goal is reducing the memory footprint of the NN proper to support larger > namespaces, and improve overall performance by decoupling the block manager > from the namespace and its lock. Ideally, a distinct BM will be transparent > to clients and DNs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-5477) Block manager as a service
[ https://issues.apache.org/jira/browse/HDFS-5477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amir Langer updated HDFS-5477: -- Attachment: Remote BM.pdf Attached design doc. RemoteBM.pdf to its JIRA > Block manager as a service > -- > > Key: HDFS-5477 > URL: https://issues.apache.org/jira/browse/HDFS-5477 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: Proposal.pdf, Proposal.pdf, Remote BM.pdf, Standalone > BM.pdf, Standalone BM.pdf, patches.tar.gz > > > The block manager needs to evolve towards having the ability to run as a > standalone service to improve NN vertical and horizontal scalability. The > goal is reducing the memory footprint of the NN proper to support larger > namespaces, and improve overall performance by decoupling the block manager > from the namespace and its lock. Ideally, a distinct BM will be transparent > to clients and DNs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-5477) Block manager as a service
[ https://issues.apache.org/jira/browse/HDFS-5477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amir Langer updated HDFS-5477: -- Attachment: patches.tar.gz All 38 patches which were used to create the 2 patches attached to the first 2 sub-tasks + a bit more work (merge into latest hadoop trunk and some improvements). These might be easier to view (if you're after viewing small-ish patches). In any case - This is not a small change whichever way you want to look at it - hence the number of patches. Also - Some of them are big because a change like separation of block instances between NameNode and BlockManager cannot be a small thing AND keep regression tests passing. The first patch was committed on top of commit 60b4b02230e439447121a8e0789cbce0be80c72c Chris Nauroth at 12/23/13, 8:33 PM HADOOP-10175. Har files system authority should preserve userinfo. Contributed by Chuan Liu. git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1553169 13f79535-47bb-0310-9956-ffa450edef68 There is a commit which merges it with a much more up to date trunk (sadly not a rebase). > Block manager as a service > -- > > Key: HDFS-5477 > URL: https://issues.apache.org/jira/browse/HDFS-5477 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: Proposal.pdf, Proposal.pdf, Standalone BM.pdf, > Standalone BM.pdf, patches.tar.gz > > > The block manager needs to evolve towards having the ability to run as a > standalone service to improve NN vertical and horizontal scalability. The > goal is reducing the memory footprint of the NN proper to support larger > namespaces, and improve overall performance by decoupling the block manager > from the namespace and its lock. Ideally, a distinct BM will be transparent > to clients and DNs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-5477) Block manager as a service
[ https://issues.apache.org/jira/browse/HDFS-5477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated HDFS-5477: - Attachment: Standalone BM.pdf Re-attach standalone pdf to fix graphics. > Block manager as a service > -- > > Key: HDFS-5477 > URL: https://issues.apache.org/jira/browse/HDFS-5477 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: Proposal.pdf, Proposal.pdf, Standalone BM.pdf, > Standalone BM.pdf > > > The block manager needs to evolve towards having the ability to run as a > standalone service to improve NN vertical and horizontal scalability. The > goal is reducing the memory footprint of the NN proper to support larger > namespaces, and improve overall performance by decoupling the block manager > from the namespace and its lock. Ideally, a distinct BM will be transparent > to clients and DNs. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-5477) Block manager as a service
[ https://issues.apache.org/jira/browse/HDFS-5477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated HDFS-5477: - Attachment: Proposal.pdf Fix formatting problems in PDF. > Block manager as a service > -- > > Key: HDFS-5477 > URL: https://issues.apache.org/jira/browse/HDFS-5477 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: Proposal.pdf, Proposal.pdf, Standalone BM.pdf > > > The block manager needs to evolve towards having the ability to run as a > standalone service to improve NN vertical and horizontal scalability. The > goal is reducing the memory footprint of the NN proper to support larger > namespaces, and improve overall performance by decoupling the block manager > from the namespace and its lock. Ideally, a distinct BM will be transparent > to clients and DNs. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-5477) Block manager as a service
[ https://issues.apache.org/jira/browse/HDFS-5477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated HDFS-5477: -- Attachment: Standalone BM.pdf Proposal.pdf Docs for the initial proposal and approach being prototyped. > Block manager as a service > -- > > Key: HDFS-5477 > URL: https://issues.apache.org/jira/browse/HDFS-5477 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: Proposal.pdf, Standalone BM.pdf > > > The block manager needs to evolve towards having the ability to run as a > standalone service to improve NN vertical and horizontal scalability. The > goal is reducing the memory footprint of the NN proper to support larger > namespaces, and improve overall performance by decoupling the block manager > from the namespace and its lock. Ideally, a distinct BM will be transparent > to clients and DNs. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
