Sounds good to me. When it's determined to include EC in 2.9 release, it may be good to have a rough release date as Zhe asked, so accordingly the scope of EC can be discussed out. We still have quite a few of things as Phase I follow-on tasks to do before EC can be deployed in a production system. Phase II to develop non-striping EC for cold data would possibly be started after that. We might consider to include only Phase I and leave Phase II for next release according to the rough release date.
Regards, Kai -----Original Message----- From: Gangumalla, Uma [mailto:uma.ganguma...@intel.com] Sent: Tuesday, November 03, 2015 5:41 AM To: hdfs-dev@hadoop.apache.org Subject: Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk] +1 for EC to go into 2.9. Yes, 3.x would be long way to go when we plan +to have 2.8 and 2.9 releases. Regards, Uma On 11/2/15, 11:49 AM, "Vinod Vavilapalli" <vino...@hortonworks.com> wrote: >Forking the thread. Started looking at the 2.8 list, various features¹ >status and arrived here. > >While I understand the pervasive nature of EC and a need for a >significant bake-in, moving this to a 3.x release is not a good idea. >We will surely get a 2.8 out this year and, as needed, I can even spend >time getting started on a 2.9. OTOH, 3.x is long ways off, and given >all the incompatibilities there, it would be a while before users can >get their hands on EC if it were to be only on 3.x. At best, this may >force sites that want EC to backport the entire EC feature to older >releases, at worst this will be repeat the mess of 0.20 security release forks. > >If we think adding this to 2.8 (even if it switched off) is too much >risk per our original plan, let¹s move this to 2.9, there by leaving >enough time for stability, integration testing and bake-in, and a >realistic chance of having it end up on users¹ clusters soonish. > >+Vinod > >> On Oct 19, 2015, at 1:44 PM, Andrew Wang <andrew.w...@cloudera.com> >>wrote: >> >> I think our plan thus far has been to target this for 3.0. I'm okay >>with putting it in branch-2 if we've given a hard look at >>compatibility, but I'll note though that 2.8 is already looking like >>quite a large release, and our release bandwidth has been focused on >>the 2.6 and 2.7 maintenance releases. Adding another multi-hundred >>JIRAs to 2.8 might make it too unwieldy to get out the door. If we >>bump EC past that, 3.0 might very well be our next release vehicle. I >>do plan to revive the 3.0 schedule some time next year. With EC and >>JDK8 in a good spot, the only big feature remaining is classpath >>isolation. >> >> EC is also a pretty fundamental change to HDFS. Even if it's >>compatible, in terms of size and impact it might best belong in a new >>major release. >> >> Best, >> Andrew >> >> On Fri, Oct 16, 2015 at 7:04 PM, Vinayakumar B < >> vinayakumarb.apa...@gmail.com> wrote: >> >>> Is anyone else also thinks that feature is ready to goto branch-2 >>>as well? >>> >>> Its > 2 weeks EC landed on trunk. IMo Looks Its quite stable since >>>then and ready to go in branch-2. >>> >>> -Vinay >>> On Oct 6, 2015 12:51 AM, "Zhe Zhang" <zhezh...@cloudera.com> wrote: >>> >>>> Thanks Vinay for capturing the issue and Uma for offering the help. >>>> >>>> --- >>>> Zhe Zhang >>>> >>>> On Mon, Oct 5, 2015 at 12:19 PM, Gangumalla, Uma < >>> uma.ganguma...@intel.com >>>>> >>>> wrote: >>>> >>>>> Vinay, >>>>> >>>>> >>>>> I would merge them as part of HDFS-9182. >>>>> >>>>> Thanks, >>>>> Uma >>>>> >>>>> >>>>> >>>>> On 10/5/15, 12:48 AM, "Vinayakumar B" <vinayakum...@apache.org> >>>>>wrote: >>>>> >>>>>> Hi Andrew, >>>>>> I see CHANGES.txt entries not yet merged from >>> CHANGES-HDFS-EC-7285.txt. >>>>>> >>>>>> Was this intentional? >>>>>> >>>>>> Regards, >>>>>> Vinay >>>>>> >>>>>> On Wed, Sep 30, 2015 at 9:15 PM, Andrew Wang < >>> andrew.w...@cloudera.com> >>>>>> wrote: >>>>>> >>>>>>> Branch has been merged to trunk, thanks again to everyone who >>>>>>>worked >>>> on >>>>>>> the >>>>>>> feature! >>>>>>> >>>>>>> On Tue, Sep 29, 2015 at 10:44 PM, Zhe Zhang >>>>>>> <zhezh...@cloudera.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Thanks everyone who has participated in this discussion. >>>>>>>> >>>>>>>> With 7 +1's (5 binding and 2 non-binding), and no -1, this vote >>> has >>>>>>> passed. >>>>>>>> I will do a final 'git merge' with trunk and work with Andrew >>>>>>>> to >>>> merge >>>>>>> the >>>>>>>> branch to trunk. I'll update on this thread when the merge is >>> done. >>>>>>>> >>>>>>>> --- >>>>>>>> Zhe Zhang >>>>>>>> >>>>>>>> On Thu, Sep 24, 2015 at 11:08 PM, Liu, Yi A >>>>>>>> <yi.a....@intel.com> >>>>>>> wrote: >>>>>>>> >>>>>>>>> (Change it to binding.) >>>>>>>>> >>>>>>>>> +1 >>>>>>>>> I have been involved in the development and code review on the >>>>>>> feature >>>>>>>>> branch. It's a great feature and I think it's ready to merge >>>>>>>>> it >>>> into >>>>>>>> trunk. >>>>>>>>> >>>>>>>>> Thanks all for the contribution. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Yi Liu >>>>>>>>> >>>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: Liu, Yi A >>>>>>>>> Sent: Friday, September 25, 2015 1:51 PM >>>>>>>>> To: hdfs-dev@hadoop.apache.org >>>>>>>>> Subject: RE: [VOTE] Merge HDFS-7285 (erasure coding) branch to >>>> trunk >>>>>>>>> >>>>>>>>> +1 (non-binding) >>>>>>>>> I have been involved in the development and code review on the >>>>>>> feature >>>>>>>>> branch. It's a great feature and I think it's ready to merge >>>>>>>>> it >>>> into >>>>>>>> trunk. >>>>>>>>> >>>>>>>>> Thanks all for the contribution. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Yi Liu >>>>>>>>> >>>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: Vinayakumar B [mailto:vinayakum...@apache.org] >>>>>>>>> Sent: Friday, September 25, 2015 12:21 PM >>>>>>>>> To: hdfs-dev@hadoop.apache.org >>>>>>>>> Subject: Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to >>>> trunk >>>>>>>>> >>>>>>>>> +1, >>>>>>>>> >>>>>>>>> I've been involved starting from design and development of >>>>>>> ErasureCoding. >>>>>>>>> I think phase 1 of this development is ready to be merged to >>>> trunk. >>>>>>>>> It had come a long way to the current state with significant >>>> effort >>>>>>> of >>>>>>>>> many Contributors and Reviewers for both design and code. >>>>>>>>> >>>>>>>>> Thanks Everyone for the efforts. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Vinay >>>>>>>>> >>>>>>>>> On Wed, Sep 23, 2015 at 10:53 PM, Jing Zhao <ji...@apache.org> >>>>>>> wrote: >>>>>>>>> >>>>>>>>>> +1 >>>>>>>>>> >>>>>>>>>> I've been involved in both development and review on the >>> branch, >>>>>>> and >>>>>>> I >>>>>>>>>> believe it's now ready to get merged into trunk. Many thanks >>> to >>>>>>> all >>>>>>>>>> the contributors and reviewers! >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> -Jing >>>>>>>>>> >>>>>>>>>> On Tue, Sep 22, 2015 at 6:17 PM, Zheng, Kai < >>>> kai.zh...@intel.com> >>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Non-binding +1 >>>>>>>>>>> >>>>>>>>>>> According to our extensive performance tests, striping + >>> ISA-L >>>>>>> coder >>>>>>>>>> based >>>>>>>>>>> erasure coding not only can save storage, but also can >>>> increase >>>>>>> the >>>>>>>>>>> throughput of a client or a cluster. It will be a great >>>>>>> addition to >>>>>>>>>>> HDFS and its users. Based on the latest branch codes, we >>> also >>>>>>>>>>> observed it's >>>>>>>>>> very >>>>>>>>>>> reliable in the concurrent tests. We'll provide the perf >>> test >>>>>>> report >>>>>>>>>> after >>>>>>>>>>> it's sorted out and hope it helps. >>>>>>>>>>> Thanks! >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Kai >>>>>>>>>>> >>>>>>>>>>> -----Original Message----- >>>>>>>>>>> From: Gangumalla, Uma [mailto:uma.ganguma...@intel.com] >>>>>>>>>>> Sent: Wednesday, September 23, 2015 8:50 AM >>>>>>>>>>> To: hdfs-dev@hadoop.apache.org; >>> common-...@hadoop.apache.org >>>>>>>>>>> Subject: Re: [VOTE] Merge HDFS-7285 (erasure coding) branch >>> to >>>>>>> trunk >>>>>>>>>>> >>>>>>>>>>> +1 >>>>>>>>>>> >>>>>>>>>>> Great addition to HDFS. Thanks all contributors for the nice >>>>>>> work. >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Uma >>>>>>>>>>> >>>>>>>>>>> On 9/22/15, 3:40 PM, "Zhe Zhang" <zhezh...@cloudera.com> >>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> I'd like to propose a vote to merge the HDFS-7285 feature >>>>>>> branch >>>>>>>>>>>> back to trunk. Since November 2014 we have been designing >>> and >>>>>>>>>>>> developing this feature under the umbrella JIRAs HDFS-7285 >>>> and >>>>>>>>>>>> HADOOP-11264, and have committed approximately 210 patches. >>>>>>>>>>>> >>>>>>>>>>>> The HDFS-7285 feature branch was created to support the >>> first >>>>>>> phase >>>>>>>>>>>> of HDFS erasure coding (HDFS-EC). The objective of HDFS-EC >>> is >>>>>>> to >>>>>>>>>>>> significantly reduce storage space usage in HDFS clusters. >>>>>>> Instead >>>>>>>>>>>> of always creating 3 replicas of each block with 200% >>> storage >>>>>>> space >>>>>>>>>>>> overhead, HDFS-EC provides data durability through parity >>>> data >>>>>>>> blocks. >>>>>>>>>>>> With most EC configurations, the storage overhead is no >>> more >>>>>>> than >>>>>>>> 50%. >>>>>>>>>>>> Based on profiling results of production clusters, we >>> decided >>>>>>> to >>>>>>>>>>>> support EC with the striped block layout in the first >>> phase, >>>> so >>>>>>>>>>>> that small files can be better handled. This means dividing >>>>>>> each >>>>>>>>>>>> logical HDFS file block into smaller units (striping cells) >>>> and >>>>>>>>>>>> spreading them on a set of DataNodes in round-robin >>> fashion. >>>>>>> Parity >>>>>>>>>>>> cells are generated for each stripe of original data cells. >>>> We >>>>>>> have >>>>>>>>>>>> made changes to NameNode, client, and DataNode to >>> generalize >>>>>>> the >>>>>>>>>>>> block concept and handle the mapping between a logical file >>>>>>> block >>>>>>>>>>>> and its internal storage blocks. For further details please >>>> see >>>>>>> the >>>>>>>>>>>> design doc on HDFS-7285. >>>>>>>>>>>> HADOOP-11264 focuses on providing flexible and >>>> high-performance >>>>>>>>>>>> codec calculation support. >>>>>>>>>>>> >>>>>>>>>>>> The nightly Jenkins job of the branch has reported several >>>>>>>>>>>> successful runs, and doesn't show new flaky tests compared >>>> with >>>>>>>>>>>> trunk. We have posted several versions of the test plan >>>>>>> including >>>>>>>>>>>> both unit testing and cluster testing, and have executed >>> most >>>>>>> tests >>>>>>>>>>>> in the plan. The most basic functionalities have been >>>>>>> extensively >>>>>>>>>>>> tested and verified in several real clusters with different >>>>>>>>>>>> hardware configurations; results have been very stable. We >>>> have >>>>>>>>>>>> created follow-on tasks for more advanced error handling >>> and >>>>>>>>> optimization under the umbrella HDFS-8031. >>>>>>>>>>>> We also plan to implement or harden the integration of EC >>>> with >>>>>>>>>>>> existing features such as WebHDFS, snapshot, append, >>>> truncate, >>>>>>>>>>>> hflush, hsync, and so forth. >>>>>>>>>>>> >>>>>>>>>>>> Development of this feature has been a collaboration across >>>>>>> many >>>>>>>>>>>> companies and institutions. I'd like to thank J. Andreina, >>>>>>> Takanobu >>>>>>>>>>>> Asanuma, Vinayakumar B, Li Bo, Takuya Fukudome, Uma >>> Maheswara >>>>>>> Rao >>>>>>>>>>>> G, Rui Li, Yi Liu, Colin McCabe, Xinwei Qin, Rakesh R, Gao >>>> Rui, >>>>>>> Kai >>>>>>>>>>>> Sasaki, Walter Su, Tsz Wo Nicholas Sze, Andrew Wang, Yong >>>>>>> Zhang, >>>>>>>>>>>> Jing Zhao, Hui Zheng and Kai Zheng for their code >>>> contributions >>>>>>> and >>>>>>>>> reviews. >>>>>>>>>>>> Andrew and Kai Zheng also made fundamental contributions to >>>> the >>>>>>>>>>>> initial design. Rui Li, Gao Rui, Kai Sasaki, Kai Zheng and >>>> many >>>>>>>>>>>> other contributors have made great efforts in system >>> testing. >>>>>>> Many >>>>>>>>>>>> thanks go to Weihua Jiang for proposing the JIRA, and ATM, >>>> Todd >>>>>>>>>>>> Lipcon, Silvius Rus, Suresh, as well as many others for >>>>>>> providing >>>>>>>>> helpful feedbacks. >>>>>>>>>>>> >>>>>>>>>>>> Following the community convention, this vote will last >>> for 7 >>>>>>> days >>>>>>>>>>>> (ending September 29th). Votes from Hadoop committers are >>>>>>> binding >>>>>>>>>>>> but non-binding votes are very welcome as well. And here's >>> my >>>>>>>>>>>> non-binding >>>>>>>>>> +1. >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> --- >>>>>>>>>>>> Zhe Zhang >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>>> >>>> >>> >
