Thanks for clarifying that Arpit. I'm a +0.9 since I haven't reviewed enough to +1, but everything thus far looks great.
Andrew On Fri, Dec 6, 2013 at 5:35 PM, Chen He <airb...@gmail.com> wrote: > +1 nice feature for HDFS > > > On Fri, Dec 6, 2013 at 7:32 PM, Arpit Agarwal <aagar...@hortonworks.com > >wrote: > > > Hi Andrew, > > > > Our plan as stated back in August was to do this work principally in two > > phases. > > > https://issues.apache.org/jira/browse/HDFS-2832?focusedCommentId=13739041 > > > > For the second phase which includes API support, we also need quota > > management. For changes of this scope, to do all the work at once while > > keeping the feature branch in sync with ongoing development in trunk is > > unmanageable. Hence we'd like to stick with the initial plan and develop > in > > phases. > > > > Even for datanode caching the initial merge did not include the quota > > management changes which are happening subsequently. > > > > Going forward, we will stabilize the current changes in trunk in the 2.4 > > time frame. Next we will add quota management and API support which can > > align with the 2.5 time frame, with the second merge potentially in > > March/April. > > > > Arpit > > > > > > On Fri, Dec 6, 2013 at 3:15 PM, Andrew Wang <andrew.w...@cloudera.com > > >wrote: > > > > > Hi everyone, > > > > > > I'm still getting up to speed on the changes here (my fault for not > > > following development more closely, other priorities etc etc), but the > > > branch thus far is already quite impressive. It's quite an undertaking > to > > > turn the DN into a collection of Storages, along with the corresponding > > > datastructure, tracking, and other changes in the NN and DN. > > > > > > Correct me if I'm wrong though, but this still leaves a substantial > part > > of > > > the design doc to be implemented. Looking at the list of remaining > > > subtasks, it seems like we still can't specify a storage type for a > file > > > (HDFS-5229) or write a file to a given storage type (HDFS-5391), along > > with > > > the corresponding client protocol changes. This leads me to two > > questions: > > > > > > - If this is merged, what can I do with the new code? Without client > > > changes or the ability to create a file on a different storage type, I > > > don't know how (for example) I could hand this to our QA team to test. > > I'm > > > wondering why we want to merge now rather than when the branch is more > > > feature complete. > > > - What's the plan for the implementation of the remaining features? How > > > many phases? What's the timeline for these phases? Particularly, > related > > to > > > the use cases presented in section 2 of the design doc. > > > > > > I'm also going to post some design doc questions to the JIRA, there > are a > > > few technical q's I'd like to get clarification on. > > > > > > Thanks, > > > Andrew > > > > > > > > > On Wed, Dec 4, 2013 at 7:21 AM, Sirianni, Eric < > eric.siria...@netapp.com > > > >wrote: > > > > > > > +1 > > > > > > > > My team has been developing and testing against the HDFS-2832 branch > > for > > > > the past month. It has proven to be quite stable. > > > > > > > > Eric > > > > > > > > -----Original Message----- > > > > From: Arpit Agarwal [mailto:aagar...@hortonworks.com] > > > > Sent: Monday, December 02, 2013 7:07 PM > > > > To: hdfs-...@hadoop.apache.org; common-dev@hadoop.apache.org > > > > Subject: [VOTE] Merge HDFS-2832 Heterogeneous Storage Phase 1 to > trunk > > > > > > > > Hello all, > > > > > > > > I would like to call a vote to merge phase 1 of the Heterogeneous > > Storage > > > > feature into trunk. > > > > > > > > *Scope of the changes:* > > > > The changes allow exposing the DataNode as a collection of storages > and > > > set > > > > the foundation for subsequent work to present Heterogeneous Storages > to > > > > applications. This allows DataNodes to send block and storage reports > > > > per-storage. In addition this change introduces the ability to add a > > > > 'storage type' tag to the storage directories. This enables > supporting > > > > different types of storages in addition to disk storage. > > > > > > > > Development of the feature is tracked in the jira > > > > https://issues.apache.org/jira/browse/HDFS-2832. > > > > > > > > *Details of development and testing:* > > > > Development has been done in a separate branch - > > > > https://svn.apache.org/repos/asf/hadoop/common/branches/HDFS-2832. > The > > > > updated design is posted at - > > > > > > > > > > > > > > https://issues.apache.org/jira/secure/attachment/12615761/20131125-HeterogeneousStorage.pdf > > > > . > > > > The changes involve ~6K changed lines of code, with a third of those > > > > changes being to tests. > > > > > > > > Please see the test plan > > > > > > > > > > > > > > https://issues.apache.org/jira/secure/attachment/12616642/20131202-HeterogeneousStorage-TestPlan.pdffor > > > > the details. Once the feature is > > > > merged into trunk, we will continue to test and fix any bugs that may > > be > > > > found on trunk as well as add further tests as outlined in the test > > plan. > > > > > > > > The bulk of the design and implementation was done by Suresh > Srinivas, > > > > Sanjay Radia, Nicholas Sze, Junping Du and me. Also, thanks to Eric > > > > Sirianni, Chris Nauroth, Steve Loughran, Bikas Saha, Andrew Wang and > > Todd > > > > Lipcon for providing feedback on the Jiras and in discussions. > > > > > > > > This vote runs for a week and closes on 12/9/2013 at 11:59 pm PT. > > > > > > > > Thanks, > > > > Arpit > > > > > > > > -- > > > > CONFIDENTIALITY NOTICE > > > > NOTICE: This message is intended for the use of the individual or > > entity > > > to > > > > which it is addressed and may contain information that is > confidential, > > > > privileged and exempt from disclosure under applicable law. If the > > reader > > > > of this message is not the intended recipient, you are hereby > notified > > > that > > > > any printing, copying, dissemination, distribution, disclosure or > > > > forwarding of this communication is strictly prohibited. If you have > > > > received this communication in error, please contact the sender > > > immediately > > > > and delete it from your system. Thank You. > > > > > > > > > > > -- > > CONFIDENTIALITY NOTICE > > NOTICE: This message is intended for the use of the individual or entity > to > > which it is addressed and may contain information that is confidential, > > privileged and exempt from disclosure under applicable law. If the reader > > of this message is not the intended recipient, you are hereby notified > that > > any printing, copying, dissemination, distribution, disclosure or > > forwarding of this communication is strictly prohibited. If you have > > received this communication in error, please contact the sender > immediately > > and delete it from your system. Thank You. > > >
