Hey Nicholas, My concern about Archival Storage isn't related to the code quality or the size of the feature. I think that you and Jing did good work. My concern is that once we ship, we're locked into that set of archival storage APIs, and these APIs are not yet finalized. Simply being able to turn off the feature does not change the compatibility story.
I'm willing to devote time to help review these JIRAs and kick the tires on the APIs, but my point above was that I'm not sure it'd all be done by the end of the week. Testing might also reveal additional changes that need to be made, which also might not happen by end-of-week. I guess the question before us is if we're comfortable putting something in branch-2.6 and then potentially adding API changes after. I'm okay with that as long as we're all aware that this might happen. Arun, as RM is this cool with you? Again, I like this feature and I'm fine with it's inclusion, just a heads up that we might need some extra time to finalize things before an RC can be cut. Thanks, Andrew On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze < s29752-hadoop...@yahoo.com.invalid> wrote: > Hi, > > I am worry about KMS and transparent encryption since there are quite many > bugs discovered after it got merged to branch-2. It gives us an impression > that the feature is not yet well tested. Indeed, transparent encryption is > a complicated feature which changes the core part of HDFS. It is not easy > to get everything right. > > > For HDFS-6584: Archival Storage, it is a relatively simple and low risk > feature. It introduces a new storage type ARCHIVE and the concept of block > storage policy to HDFS. When a cluster is configured with ARCHIVE storage, > the blocks will be stored using the appropriate storage types specified by > storage policies assigned to the files/directories. Cluster admin could > disable the feature by simply not configuring any storage type and not > setting any storage policy as before. As Suresh mentioned, HDFS-6584 is > in the final stages to be merged to branch-2. > > Regards, > Tsz-Wo > > > > On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas < > sur...@hortonworks.com> wrote: > > > > > > > >I actually would like to see both archival storage and single replica > >memory writes to be in 2.6 release. Archival storage is in the final > stages > >of getting ready for branch-2 merge as Nicholas has already indicated on > >the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of these > >features are being in development for sometime. > > > >On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang <andrew.w...@cloudera.com> > >wrote: > > > >> Hey Arun, > >> > >> Maybe we could do a quick run through of the Roadmap wiki and > add/retarget > >> things accordingly? > >> > >> I think the KMS and transparent encryption are ready to go. We've got a > >> very few further bug fixes pending, but that's it. > >> > >> Two HDFS things that I think probably won't make the end of the week are > >> archival storage (HDFS-6584) and single replica memory writes > (HDFS-6581), > >> which I believe are under the HSM banner. HDFS-6484 was just merged to > >> trunk and I think needs a little more work before it goes into branch-2. > >> HDFS-6581 hasn't even been merged to trunk yet, so seems a bit further > off > >> yet. > >> > >> Just my 2c as I did not work directly on these features. I just > generally > >> shy away from shipping bits quite this fresh. > >> > >> Thanks, > >> Andrew > >> > >> On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy <a...@hortonworks.com> > wrote: > >> > >> > Looks like most of the content is in and hadoop-2.6 is shaping up > nicely. > >> > > >> > I'll create branch-2.6 by end of the week and we can go from there to > >> > stabilize it - hopefully in the next few weeks. > >> > > >> > Thoughts? > >> > > >> > thanks, > >> > Arun > >> > > >> > On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy <a...@hortonworks.com> > >> > wrote: > >> > > >> > > Folks, > >> > > > >> > > With hadoop-2.5 nearly done, it's time to start thinking ahead to > >> > > hadoop-2.6. > >> > > > >> > > Currently, here is the Roadmap per the wiki: > >> > > > >> > > • HADOOP > >> > > • Credential provider HADOOP-10607 > >> > > • HDFS > >> > > • Heterogeneous storage (Phase 2) - Support APIs for > >> > using > >> > > storage tiers by the applications HDFS-5682 > >> > > • Memory as storage tier HDFS-5851 > >> > > • YARN > >> > > • Dynamic Resource Configuration YARN-291 > >> > > • NodeManager Restart YARN-1336 > >> > > • ResourceManager HA Phase 2 YARN-556 > >> > > • Support for admin-specified labels in YARN > YARN-796 > >> > > • Support for automatic, shared cache for YARN > >> > application > >> > > artifacts YARN-1492 > >> > > • Support NodeGroup layer topology on YARN YARN-18 > >> > > • Support for Docker containers in YARN YARN-1964 > >> > > • YARN service registry YARN-913 > >> > > > >> > > My suspicion is, as is normal, some will make the cut and some > won't. > >> > > Please do add/subtract from the list as appropriate. Ideally, it > would > >> be > >> > > good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a > >> > cadence. > >> > > > >> > > More importantly, as we discussed previously, we'd like hadoop-2.6 > to > >> be > >> > > the *last* Apache Hadoop 2.x release which support JDK6. I'll start > a > >> > > discussion with other communities (HBase, Pig, Hive, Oozie etc.) and > >> see > >> > > how they feel about this. > >> > > > >> > > thanks, > >> > > Arun > >> > > > >> > > > >> > > >> > > >> > -- > >> > > >> > -- > >> > Arun C. Murthy > >> > Hortonworks Inc. > >> > http://hortonworks.com/ > >> > > >> > -- > >> > CONFIDENTIALITY NOTICE > >> > NOTICE: This message is intended for the use of the individual or > entity > >> to > >> > which it is addressed and may contain information that is > confidential, > >> > privileged and exempt from disclosure under applicable law. If the > reader > >> > of this message is not the intended recipient, you are hereby notified > >> that > >> > any printing, copying, dissemination, distribution, disclosure or > >> > forwarding of this communication is strictly prohibited. If you have > >> > received this communication in error, please contact the sender > >> immediately > >> > and delete it from your system. Thank You. > >> > > >> > > > > > > > >-- > >http://hortonworks.com/download/ > > > >-- > >CONFIDENTIALITY NOTICE > >NOTICE: This message is intended for the use of the individual or entity > to > >which it is addressed and may contain information that is confidential, > >privileged and exempt from disclosure under applicable law. If the reader > >of this message is not the intended recipient, you are hereby notified > that > >any printing, copying, dissemination, distribution, disclosure or > >forwarding of this communication is strictly prohibited. If you have > >received this communication in error, please contact the sender > immediately > >and delete it from your system. Thank You. > > > > >
