Just to give a quick update about the current status of HDFS-6584 (archival storage). So far after HDFS-7081 got committed this morning, the main functionalities have already been finished. As a summary two new DistributedFileSystem APIs are added: getStoragePolicies and setStoragePolicy. We have also been doing system tests for weeks and we will continue testing. There are still one or two pending issues but we're actively working on it. So I'm pretty confident that the archival storage work can get ready given the current plan to release 2.6 next week.
On Wed, Sep 24, 2014 at 3:35 PM, Jitendra Pandey <jiten...@hortonworks.com> wrote: > I also believe its worth a week's wait to include HDFS-6584 and HDFS-6581 > in 2.6. > > On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas <sur...@hortonworks.com> > wrote: > > > Given some of the features are in final stages of stabilization, > > Arun, we should hold off creating 2.6 branch or building an RC by a week? > > All the features in flux are important ones and worth delaying the > release > > by a week. > > > > On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang <andrew.w...@cloudera.com> > > wrote: > > > > > Hey Nicholas, > > > > > > My concern about Archival Storage isn't related to the code quality or > > the > > > size of the feature. I think that you and Jing did good work. My > concern > > is > > > that once we ship, we're locked into that set of archival storage APIs, > > and > > > these APIs are not yet finalized. Simply being able to turn off the > > feature > > > does not change the compatibility story. > > > > > > I'm willing to devote time to help review these JIRAs and kick the > tires > > on > > > the APIs, but my point above was that I'm not sure it'd all be done by > > the > > > end of the week. Testing might also reveal additional changes that need > > to > > > be made, which also might not happen by end-of-week. > > > > > > I guess the question before us is if we're comfortable putting > something > > in > > > branch-2.6 and then potentially adding API changes after. I'm okay with > > > that as long as we're all aware that this might happen. > > > > > > Arun, as RM is this cool with you? Again, I like this feature and I'm > > fine > > > with it's inclusion, just a heads up that we might need some extra time > > to > > > finalize things before an RC can be cut. > > > > > > Thanks, > > > Andrew > > > > > > On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze < > > > s29752-hadoop...@yahoo.com.invalid> wrote: > > > > > > > Hi, > > > > > > > > I am worry about KMS and transparent encryption since there are quite > > > many > > > > bugs discovered after it got merged to branch-2. It gives us an > > > impression > > > > that the feature is not yet well tested. Indeed, transparent > > encryption > > > is > > > > a complicated feature which changes the core part of HDFS. It is not > > > easy > > > > to get everything right. > > > > > > > > > > > > For HDFS-6584: Archival Storage, it is a relatively simple and low > risk > > > > feature. It introduces a new storage type ARCHIVE and the concept of > > > block > > > > storage policy to HDFS. When a cluster is configured with ARCHIVE > > > storage, > > > > the blocks will be stored using the appropriate storage types > specified > > > by > > > > storage policies assigned to the files/directories. Cluster admin > > could > > > > disable the feature by simply not configuring any storage type and > not > > > > setting any storage policy as before. As Suresh mentioned, > HDFS-6584 > > is > > > > in the final stages to be merged to branch-2. > > > > > > > > Regards, > > > > Tsz-Wo > > > > > > > > > > > > > > > > On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas < > > > > sur...@hortonworks.com> wrote: > > > > > > > > > > > > > > > > > > > > > > >I actually would like to see both archival storage and single > replica > > > > >memory writes to be in 2.6 release. Archival storage is in the final > > > > stages > > > > >of getting ready for branch-2 merge as Nicholas has already > indicated > > on > > > > >the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of > > > these > > > > >features are being in development for sometime. > > > > > > > > > >On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang < > > andrew.w...@cloudera.com> > > > > >wrote: > > > > > > > > > >> Hey Arun, > > > > >> > > > > >> Maybe we could do a quick run through of the Roadmap wiki and > > > > add/retarget > > > > >> things accordingly? > > > > >> > > > > >> I think the KMS and transparent encryption are ready to go. We've > > got > > > a > > > > >> very few further bug fixes pending, but that's it. > > > > >> > > > > >> Two HDFS things that I think probably won't make the end of the > week > > > are > > > > >> archival storage (HDFS-6584) and single replica memory writes > > > > (HDFS-6581), > > > > >> which I believe are under the HSM banner. HDFS-6484 was just > merged > > to > > > > >> trunk and I think needs a little more work before it goes into > > > branch-2. > > > > >> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit > > further > > > > off > > > > >> yet. > > > > >> > > > > >> Just my 2c as I did not work directly on these features. I just > > > > generally > > > > >> shy away from shipping bits quite this fresh. > > > > >> > > > > >> Thanks, > > > > >> Andrew > > > > >> > > > > >> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <a...@hortonworks.com > > > > > > wrote: > > > > >> > > > > >> > Looks like most of the content is in and hadoop-2.6 is shaping > up > > > > nicely. > > > > >> > > > > > >> > I'll create branch-2.6 by end of the week and we can go from > there > > > to > > > > >> > stabilize it - hopefully in the next few weeks. > > > > >> > > > > > >> > Thoughts? > > > > >> > > > > > >> > thanks, > > > > >> > Arun > > > > >> > > > > > >> > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy < > > a...@hortonworks.com > > > > > > > > >> > wrote: > > > > >> > > > > > >> > > Folks, > > > > >> > > > > > > >> > > With hadoop-2.5 nearly done, it's time to start thinking > ahead > > to > > > > >> > > hadoop-2.6. > > > > >> > > > > > > >> > > Currently, here is the Roadmap per the wiki: > > > > >> > > > > > > >> > > • HADOOP > > > > >> > > • Credential provider HADOOP-10607 > > > > >> > > • HDFS > > > > >> > > • Heterogeneous storage (Phase 2) - Support > APIs > > > for > > > > >> > using > > > > >> > > storage tiers by the applications HDFS-5682 > > > > >> > > • Memory as storage tier HDFS-5851 > > > > >> > > • YARN > > > > >> > > • Dynamic Resource Configuration YARN-291 > > > > >> > > • NodeManager Restart YARN-1336 > > > > >> > > • ResourceManager HA Phase 2 YARN-556 > > > > >> > > • Support for admin-specified labels in YARN > > > > YARN-796 > > > > >> > > • Support for automatic, shared cache for YARN > > > > >> > application > > > > >> > > artifacts YARN-1492 > > > > >> > > • Support NodeGroup layer topology on YARN > > YARN-18 > > > > >> > > • Support for Docker containers in YARN > > YARN-1964 > > > > >> > > • YARN service registry YARN-913 > > > > >> > > > > > > >> > > My suspicion is, as is normal, some will make the cut and > some > > > > won't. > > > > >> > > Please do add/subtract from the list as appropriate. Ideally, > it > > > > would > > > > >> be > > > > >> > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep > > up a > > > > >> > cadence. > > > > >> > > > > > > >> > > More importantly, as we discussed previously, we'd like > > > hadoop-2.6 > > > > to > > > > >> be > > > > >> > > the *last* Apache Hadoop 2.x release which support JDK6. I'll > > > start > > > > a > > > > >> > > discussion with other communities (HBase, Pig, Hive, Oozie > etc.) > > > and > > > > >> see > > > > >> > > how they feel about this. > > > > >> > > > > > > >> > > thanks, > > > > >> > > Arun > > > > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > > >> > -- > > > > >> > > > > > >> > -- > > > > >> > Arun C. Murthy > > > > >> > Hortonworks Inc. > > > > >> > http://hortonworks.com/ > > > > >> > > > > > >> > -- > > > > >> > CONFIDENTIALITY NOTICE > > > > >> > NOTICE: This message is intended for the use of the individual > or > > > > entity > > > > >> to > > > > >> > which it is addressed and may contain information that is > > > > confidential, > > > > >> > privileged and exempt from disclosure under applicable law. If > the > > > > reader > > > > >> > of this message is not the intended recipient, you are hereby > > > notified > > > > >> that > > > > >> > any printing, copying, dissemination, distribution, disclosure > or > > > > >> > forwarding of this communication is strictly prohibited. If you > > have > > > > >> > received this communication in error, please contact the sender > > > > >> immediately > > > > >> > and delete it from your system. Thank You. > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > > > >-- > > > > >http://hortonworks.com/download/ > > > > > > > > > >-- > > > > >CONFIDENTIALITY NOTICE > > > > >NOTICE: This message is intended for the use of the individual or > > entity > > > > to > > > > >which it is addressed and may contain information that is > > confidential, > > > > >privileged and exempt from disclosure under applicable law. If the > > > reader > > > > >of this message is not the intended recipient, you are hereby > notified > > > > that > > > > >any printing, copying, dissemination, distribution, disclosure or > > > > >forwarding of this communication is strictly prohibited. If you have > > > > >received this communication in error, please contact the sender > > > > immediately > > > > >and delete it from your system. Thank You. > > > > > > > > > > > > > > > > > > > > > > > > > -- > > http://hortonworks.com/download/ > > > > -- > > CONFIDENTIALITY NOTICE > > NOTICE: This message is intended for the use of the individual or entity > to > > which it is addressed and may contain information that is confidential, > > privileged and exempt from disclosure under applicable law. If the reader > > of this message is not the intended recipient, you are hereby notified > that > > any printing, copying, dissemination, distribution, disclosure or > > forwarding of this communication is strictly prohibited. If you have > > received this communication in error, please contact the sender > immediately > > and delete it from your system. Thank You. > > > > > > -- > <http://hortonworks.com/download/> > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
