Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]
Sorry for getting back to the thread late. Since Ming/Daryn have not raised further concerns, I'm +0 on including EC in 2.9. The upside is that we can perhaps position EC in 2.9 as a "beta" feature to gain some production experiences, and graduate it from beta in 3.0. The downside is as Elliot stated, this will delay the upgrade to 2.9 for some customers. On Tue, Dec 8, 2015 at 2:54 PM, Zhe Zhang wrote: > Hi Vinod, > > Thanks for the update. After seeing the comment from Elliot I checked with > Ming Ma and Daryn Sharp offline, to get feedback based on their large scale > deployments (Twitter and Yahoo). > > Based on my reading, Ming doesn't have a strong preference between 2.9 and > 3.0, and Daryn is still having some questions about the stability of EC > code. > > Maybe we should give Ming/Daryn some more time to reply to this thread? > > I think we should also address Elliot's comments above regarding major vs. > minor releases, how that impacts the 2.9 upgrade timing etc. > > Thanks, > --- > Zhe Zhang > > On Tue, Dec 8, 2015 at 2:04 PM, Vinod Kumar Vavilapalli < > vino...@apache.org> wrote: > >> Forgot to update this thread. I branched off 2.8 last week. So, we can >> now go ahead and do a merge of HDFS-7285 into branch-2 (version 2.9) like >> we discussed before. >> >> Thanks >> +Vinod >> >> >> > On Nov 3, 2015, at 4:40 PM, Vinod Kumar Vavilapalli < >> vino...@hortonworks.com> wrote: >> > >> > That makes sense. >> > >> > Thanks for the discussion everyone, let’s stick to this tentative plan >> of EC for 2.9. >> > >> > I just updated the Roadmap wiki to reflect the same. >> > >> > +Vinod >> > >> > >> >> On Nov 2, 2015, at 4:26 PM, Zheng, Kai wrote: >> >> >> >> Yeah, so for the issues we recently resolved on trunk and are >> addressing as follow-on tasks in Phase I, we would label them with "erasure >> coding" and maybe also set the target version as "2.9" for the convenience? >> >> >> >> -Original Message- >> >> From: Jing Zhao [mailto:ji...@apache.org] >> >> Sent: Tuesday, November 03, 2015 8:04 AM >> >> To: hdfs-dev@hadoop.apache.org >> >> Subject: Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge >> HDFS-7285 (erasure coding) branch to trunk] >> >> >> >> +1 for the plan about Phase I & II. >> >> >> >> BTW, maybe out of the scope of this thread, just want to mention we >> should either move the jira under HDFS-8031 or update the jira component as >> "erasure-coding" when making further improvement or fixing bugs in EC. In >> this way it will be easier for later backporting EC to 2.9. >> >> >> >> On Mon, Nov 2, 2015 at 3:48 PM, Vinayakumar B < >> vinayakumarb.apa...@gmail.com >> >>> wrote: >> >> >> >>> +1 for the idea. >> >>> On Nov 3, 2015 07:22, "Zheng, Kai" wrote: >> >>> >> >>>> Sounds good to me. When it's determined to include EC in 2.9 >> >>>> release, it may be good to have a rough release date as Zhe asked, >> >>>> so accordingly the scope of EC can be discussed out. We still have >> >>>> quite a few of things as Phase I follow-on tasks to do before EC can >> >>>> be deployed in a production system. Phase II to develop non-striping >> >>>> EC for cold data would possibly >> >>> be >> >>>> started after that. We might consider to include only Phase I and >> >>>> leave Phase II for next release according to the rough release date. >> >>>> >> >>>> Regards, >> >>>> Kai >> >>>> >> >>>> -Original Message- >> >>>> From: Gangumalla, Uma [mailto:uma.ganguma...@intel.com] >> >>>> Sent: Tuesday, November 03, 2015 5:41 AM >> >>>> To: hdfs-dev@hadoop.apache.org >> >>>> Subject: Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge >> >>>> HDFS-7285 (erasure coding) branch to trunk] >> >>>> >> >>>> +1 for EC to go into 2.9. Yes, 3.x would be long way to go when we >> >>>> +plan to >> >>>> have 2.8 and 2.9 releases. >> >>>> >> >>>> Regards, >> >>>> Uma >> >>>> >> >>>> On 11/2/15, 11:49 AM, "
Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]
Hi Vinod, Thanks for the update. After seeing the comment from Elliot I checked with Ming Ma and Daryn Sharp offline, to get feedback based on their large scale deployments (Twitter and Yahoo). Based on my reading, Ming doesn't have a strong preference between 2.9 and 3.0, and Daryn is still having some questions about the stability of EC code. Maybe we should give Ming/Daryn some more time to reply to this thread? I think we should also address Elliot's comments above regarding major vs. minor releases, how that impacts the 2.9 upgrade timing etc. Thanks, --- Zhe Zhang On Tue, Dec 8, 2015 at 2:04 PM, Vinod Kumar Vavilapalli wrote: > Forgot to update this thread. I branched off 2.8 last week. So, we can now > go ahead and do a merge of HDFS-7285 into branch-2 (version 2.9) like we > discussed before. > > Thanks > +Vinod > > > > On Nov 3, 2015, at 4:40 PM, Vinod Kumar Vavilapalli < > vino...@hortonworks.com> wrote: > > > > That makes sense. > > > > Thanks for the discussion everyone, let’s stick to this tentative plan > of EC for 2.9. > > > > I just updated the Roadmap wiki to reflect the same. > > > > +Vinod > > > > > >> On Nov 2, 2015, at 4:26 PM, Zheng, Kai wrote: > >> > >> Yeah, so for the issues we recently resolved on trunk and are > addressing as follow-on tasks in Phase I, we would label them with "erasure > coding" and maybe also set the target version as "2.9" for the convenience? > >> > >> -Original Message----- > >> From: Jing Zhao [mailto:ji...@apache.org] > >> Sent: Tuesday, November 03, 2015 8:04 AM > >> To: hdfs-dev@hadoop.apache.org > >> Subject: Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 > (erasure coding) branch to trunk] > >> > >> +1 for the plan about Phase I & II. > >> > >> BTW, maybe out of the scope of this thread, just want to mention we > should either move the jira under HDFS-8031 or update the jira component as > "erasure-coding" when making further improvement or fixing bugs in EC. In > this way it will be easier for later backporting EC to 2.9. > >> > >> On Mon, Nov 2, 2015 at 3:48 PM, Vinayakumar B < > vinayakumarb.apa...@gmail.com > >>> wrote: > >> > >>> +1 for the idea. > >>> On Nov 3, 2015 07:22, "Zheng, Kai" wrote: > >>> > >>>> Sounds good to me. When it's determined to include EC in 2.9 > >>>> release, it may be good to have a rough release date as Zhe asked, > >>>> so accordingly the scope of EC can be discussed out. We still have > >>>> quite a few of things as Phase I follow-on tasks to do before EC can > >>>> be deployed in a production system. Phase II to develop non-striping > >>>> EC for cold data would possibly > >>> be > >>>> started after that. We might consider to include only Phase I and > >>>> leave Phase II for next release according to the rough release date. > >>>> > >>>> Regards, > >>>> Kai > >>>> > >>>> -Original Message- > >>>> From: Gangumalla, Uma [mailto:uma.ganguma...@intel.com] > >>>> Sent: Tuesday, November 03, 2015 5:41 AM > >>>> To: hdfs-dev@hadoop.apache.org > >>>> Subject: Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge > >>>> HDFS-7285 (erasure coding) branch to trunk] > >>>> > >>>> +1 for EC to go into 2.9. Yes, 3.x would be long way to go when we > >>>> +plan to > >>>> have 2.8 and 2.9 releases. > >>>> > >>>> Regards, > >>>> Uma > >>>> > >>>> On 11/2/15, 11:49 AM, "Vinod Vavilapalli" > >>> wrote: > >>>> > >>>>> Forking the thread. Started looking at the 2.8 list, various > >>>>> features¹ status and arrived here. > >>>>> > >>>>> While I understand the pervasive nature of EC and a need for a > >>>>> significant bake-in, moving this to a 3.x release is not a good idea. > >>>>> We will surely get a 2.8 out this year and, as needed, I can even > >>>>> spend time getting started on a 2.9. OTOH, 3.x is long ways off, > >>>>> and given all the incompatibilities there, it would be a while > >>>>> before users can get their hands on EC if it were to be only on > >>>>> 3.x. At best, this may force sites that wa
Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]
Forgot to update this thread. I branched off 2.8 last week. So, we can now go ahead and do a merge of HDFS-7285 into branch-2 (version 2.9) like we discussed before. Thanks +Vinod > On Nov 3, 2015, at 4:40 PM, Vinod Kumar Vavilapalli > wrote: > > That makes sense. > > Thanks for the discussion everyone, let’s stick to this tentative plan of EC > for 2.9. > > I just updated the Roadmap wiki to reflect the same. > > +Vinod > > >> On Nov 2, 2015, at 4:26 PM, Zheng, Kai wrote: >> >> Yeah, so for the issues we recently resolved on trunk and are addressing as >> follow-on tasks in Phase I, we would label them with "erasure coding" and >> maybe also set the target version as "2.9" for the convenience? >> >> -Original Message- >> From: Jing Zhao [mailto:ji...@apache.org] >> Sent: Tuesday, November 03, 2015 8:04 AM >> To: hdfs-dev@hadoop.apache.org >> Subject: Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 >> (erasure coding) branch to trunk] >> >> +1 for the plan about Phase I & II. >> >> BTW, maybe out of the scope of this thread, just want to mention we should >> either move the jira under HDFS-8031 or update the jira component as >> "erasure-coding" when making further improvement or fixing bugs in EC. In >> this way it will be easier for later backporting EC to 2.9. >> >> On Mon, Nov 2, 2015 at 3:48 PM, Vinayakumar B >> wrote: >> >>> +1 for the idea. >>> On Nov 3, 2015 07:22, "Zheng, Kai" wrote: >>> >>>> Sounds good to me. When it's determined to include EC in 2.9 >>>> release, it may be good to have a rough release date as Zhe asked, >>>> so accordingly the scope of EC can be discussed out. We still have >>>> quite a few of things as Phase I follow-on tasks to do before EC can >>>> be deployed in a production system. Phase II to develop non-striping >>>> EC for cold data would possibly >>> be >>>> started after that. We might consider to include only Phase I and >>>> leave Phase II for next release according to the rough release date. >>>> >>>> Regards, >>>> Kai >>>> >>>> -Original Message- >>>> From: Gangumalla, Uma [mailto:uma.ganguma...@intel.com] >>>> Sent: Tuesday, November 03, 2015 5:41 AM >>>> To: hdfs-dev@hadoop.apache.org >>>> Subject: Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge >>>> HDFS-7285 (erasure coding) branch to trunk] >>>> >>>> +1 for EC to go into 2.9. Yes, 3.x would be long way to go when we >>>> +plan to >>>> have 2.8 and 2.9 releases. >>>> >>>> Regards, >>>> Uma >>>> >>>> On 11/2/15, 11:49 AM, "Vinod Vavilapalli" >>> wrote: >>>> >>>>> Forking the thread. Started looking at the 2.8 list, various >>>>> features¹ status and arrived here. >>>>> >>>>> While I understand the pervasive nature of EC and a need for a >>>>> significant bake-in, moving this to a 3.x release is not a good idea. >>>>> We will surely get a 2.8 out this year and, as needed, I can even >>>>> spend time getting started on a 2.9. OTOH, 3.x is long ways off, >>>>> and given all the incompatibilities there, it would be a while >>>>> before users can get their hands on EC if it were to be only on >>>>> 3.x. At best, this may force sites that want EC to backport the >>>>> entire EC feature to older releases, at worst this will be repeat >>>>> the mess of 0.20 security release >>>> forks. >>>>> >>>>> If we think adding this to 2.8 (even if it switched off) is too >>>>> much risk per our original plan, let¹s move this to 2.9, there by >>>>> leaving enough time for stability, integration testing and bake-in, >>>>> and a realistic chance of having it end up on users¹ clusters soonish. >>>>> >>>>> +Vinod >>>>> >>>>>> On Oct 19, 2015, at 1:44 PM, Andrew Wang >>>>>> >>>>>> wrote: >>>>>> >>>>>> I think our plan thus far has been to target this for 3.0. I'm >>>>>> okay with putting it in branch-2 if we've given a hard look at >>>>>> compatibility, but I'll note thoug
Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]
I don't really see the difference between 2.9 with a ton of scary changes (EC is a lot more NN stuff than a usual release) and a 3.0. What's the downside of getting a major version. It relaxes the compat a little bit. It would allow some bake time before it's stable. Put another way. I'm upgrading our prod clusters to 2.7 now and 2.8 when it comes out. That's because those releases are normal dot releases. I would hold off a much longer time for a 2.9 with EC. That to me really signals that it's a different type of release and that should be messaged with a major version rev. On Wed, Nov 4, 2015 at 4:06 PM, Andrew Purtell wrote: > Yes we can mostly likely help you. Please come over to dev@bigtop. > > > > On Nov 4, 2015, at 12:40 PM, Andrew Wang > wrote: > > > > We used to get help from Bigtop when it comes to integration testing. Do > we > > think that's possible for 2.8? > > > > On Wed, Nov 4, 2015 at 10:08 AM, Steve Loughran > > wrote: > > > >> > On 2 Nov 2015, at 23:11, Vinod Vavilapalli > >>> wrote: > >>> > >>> Yes, I’ve already started looking at 2.8.0, that is exactly how I ended > >> up with this discussion on the state of EC. > >>> > >>> +Vinod > >>> > >>> > On Nov 2, 2015, at 3:02 PM, Haohui Mai >>> ricet...@gmail.com>> wrote: > >>> > >>> Is it a good time to start the discussion on the issues of releasing > 2.8? > >> > >> Before rushing to release 2.8, people should be trying in downstream > apps > >> today,. As well as identifying hdfs-client related issues, I've just > >> discovered that the MiniYARNCluster has added lots of stack traces > >> (YARN-4330), and I'm sure there are other regressions. > >> > >> It's generally not that hard to take a downstream project and try to > build > >> with a hadoop version of 2.8.0-SNAPSHOT; compilation and classpath > problems > >> will show up immediately; unit test regressions can be at least > identified > >> by switching between 2.7.1 and 2.8.0-SNAPSHOT. >
Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]
Yes we can mostly likely help you. Please come over to dev@bigtop. > On Nov 4, 2015, at 12:40 PM, Andrew Wang wrote: > > We used to get help from Bigtop when it comes to integration testing. Do we > think that's possible for 2.8? > > On Wed, Nov 4, 2015 at 10:08 AM, Steve Loughran > wrote: > >> On 2 Nov 2015, at 23:11, Vinod Vavilapalli >>> wrote: >>> >>> Yes, I’ve already started looking at 2.8.0, that is exactly how I ended >> up with this discussion on the state of EC. >>> >>> +Vinod >>> >>> On Nov 2, 2015, at 3:02 PM, Haohui Mai >> ricet...@gmail.com>> wrote: >>> >>> Is it a good time to start the discussion on the issues of releasing 2.8? >> >> Before rushing to release 2.8, people should be trying in downstream apps >> today,. As well as identifying hdfs-client related issues, I've just >> discovered that the MiniYARNCluster has added lots of stack traces >> (YARN-4330), and I'm sure there are other regressions. >> >> It's generally not that hard to take a downstream project and try to build >> with a hadoop version of 2.8.0-SNAPSHOT; compilation and classpath problems >> will show up immediately; unit test regressions can be at least identified >> by switching between 2.7.1 and 2.8.0-SNAPSHOT.
Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]
We used to get help from Bigtop when it comes to integration testing. Do we think that's possible for 2.8? On Wed, Nov 4, 2015 at 10:08 AM, Steve Loughran wrote: > > > On 2 Nov 2015, at 23:11, Vinod Vavilapalli > wrote: > > > > Yes, I’ve already started looking at 2.8.0, that is exactly how I ended > up with this discussion on the state of EC. > > > > +Vinod > > > > > > On Nov 2, 2015, at 3:02 PM, Haohui Mai ricet...@gmail.com>> wrote: > > > > Is it a good time to start the discussion on the issues of releasing 2.8? > > > > Before rushing to release 2.8, people should be trying in downstream apps > today,. As well as identifying hdfs-client related issues, I've just > discovered that the MiniYARNCluster has added lots of stack traces > (YARN-4330), and I'm sure there are other regressions. > > It's generally not that hard to take a downstream project and try to build > with a hadoop version of 2.8.0-SNAPSHOT; compilation and classpath problems > will show up immediately; unit test regressions can be at least identified > by switching between 2.7.1 and 2.8.0-SNAPSHOT.
Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]
> On 2 Nov 2015, at 23:11, Vinod Vavilapalli wrote: > > Yes, I’ve already started looking at 2.8.0, that is exactly how I ended up > with this discussion on the state of EC. > > +Vinod > > > On Nov 2, 2015, at 3:02 PM, Haohui Mai > mailto:ricet...@gmail.com>> wrote: > > Is it a good time to start the discussion on the issues of releasing 2.8? > Before rushing to release 2.8, people should be trying in downstream apps today,. As well as identifying hdfs-client related issues, I've just discovered that the MiniYARNCluster has added lots of stack traces (YARN-4330), and I'm sure there are other regressions. It's generally not that hard to take a downstream project and try to build with a hadoop version of 2.8.0-SNAPSHOT; compilation and classpath problems will show up immediately; unit test regressions can be at least identified by switching between 2.7.1 and 2.8.0-SNAPSHOT.
Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]
To add to the above discussion on umbrella JIRA: the COMMON side changes of EC have been tracked under HADOOP-11264 (before merge) and HADOOP-11842 (after merge). --- Zhe Zhang On Tue, Nov 3, 2015 at 4:40 PM, Vinod Vavilapalli wrote: > That makes sense. > > Thanks for the discussion everyone, let’s stick to this tentative plan of > EC for 2.9. > > I just updated the Roadmap wiki to reflect the same. > > +Vinod > > > > On Nov 2, 2015, at 4:26 PM, Zheng, Kai wrote: > > > > Yeah, so for the issues we recently resolved on trunk and are addressing > as follow-on tasks in Phase I, we would label them with "erasure coding" > and maybe also set the target version as "2.9" for the convenience? > > > > -Original Message- > > From: Jing Zhao [mailto:ji...@apache.org] > > Sent: Tuesday, November 03, 2015 8:04 AM > > To: hdfs-dev@hadoop.apache.org > > Subject: Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 > (erasure coding) branch to trunk] > > > > +1 for the plan about Phase I & II. > > > > BTW, maybe out of the scope of this thread, just want to mention we > should either move the jira under HDFS-8031 or update the jira component as > "erasure-coding" when making further improvement or fixing bugs in EC. In > this way it will be easier for later backporting EC to 2.9. > > > > On Mon, Nov 2, 2015 at 3:48 PM, Vinayakumar B < > vinayakumarb.apa...@gmail.com > >> wrote: > > > >> +1 for the idea. > >> On Nov 3, 2015 07:22, "Zheng, Kai" wrote: > >> > >>> Sounds good to me. When it's determined to include EC in 2.9 > >>> release, it may be good to have a rough release date as Zhe asked, > >>> so accordingly the scope of EC can be discussed out. We still have > >>> quite a few of things as Phase I follow-on tasks to do before EC can > >>> be deployed in a production system. Phase II to develop non-striping > >>> EC for cold data would possibly > >> be > >>> started after that. We might consider to include only Phase I and > >>> leave Phase II for next release according to the rough release date. > >>> > >>> Regards, > >>> Kai > >>> > >>> -Original Message- > >>> From: Gangumalla, Uma [mailto:uma.ganguma...@intel.com] > >>> Sent: Tuesday, November 03, 2015 5:41 AM > >>> To: hdfs-dev@hadoop.apache.org > >>> Subject: Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge > >>> HDFS-7285 (erasure coding) branch to trunk] > >>> > >>> +1 for EC to go into 2.9. Yes, 3.x would be long way to go when we > >>> +plan to > >>> have 2.8 and 2.9 releases. > >>> > >>> Regards, > >>> Uma > >>> > >>> On 11/2/15, 11:49 AM, "Vinod Vavilapalli" > >> wrote: > >>> > >>>> Forking the thread. Started looking at the 2.8 list, various > >>>> features¹ status and arrived here. > >>>> > >>>> While I understand the pervasive nature of EC and a need for a > >>>> significant bake-in, moving this to a 3.x release is not a good idea. > >>>> We will surely get a 2.8 out this year and, as needed, I can even > >>>> spend time getting started on a 2.9. OTOH, 3.x is long ways off, > >>>> and given all the incompatibilities there, it would be a while > >>>> before users can get their hands on EC if it were to be only on > >>>> 3.x. At best, this may force sites that want EC to backport the > >>>> entire EC feature to older releases, at worst this will be repeat > >>>> the mess of 0.20 security release > >>> forks. > >>>> > >>>> If we think adding this to 2.8 (even if it switched off) is too > >>>> much risk per our original plan, let¹s move this to 2.9, there by > >>>> leaving enough time for stability, integration testing and bake-in, > >>>> and a realistic chance of having it end up on users¹ clusters soonish. > >>>> > >>>> +Vinod > >>>> > >>>>> On Oct 19, 2015, at 1:44 PM, Andrew Wang > >>>>> > >>>>> wrote: > >>>>> > >>>>> I think our plan thus far has been to target this for 3.0. I'm > >>>>> okay with putting it in branch-2 if we've given a hard look at > >>>>> compatibility, but I'll n
Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]
That makes sense. Thanks for the discussion everyone, let’s stick to this tentative plan of EC for 2.9. I just updated the Roadmap wiki to reflect the same. +Vinod > On Nov 2, 2015, at 4:26 PM, Zheng, Kai wrote: > > Yeah, so for the issues we recently resolved on trunk and are addressing as > follow-on tasks in Phase I, we would label them with "erasure coding" and > maybe also set the target version as "2.9" for the convenience? > > -Original Message- > From: Jing Zhao [mailto:ji...@apache.org] > Sent: Tuesday, November 03, 2015 8:04 AM > To: hdfs-dev@hadoop.apache.org > Subject: Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 > (erasure coding) branch to trunk] > > +1 for the plan about Phase I & II. > > BTW, maybe out of the scope of this thread, just want to mention we should > either move the jira under HDFS-8031 or update the jira component as > "erasure-coding" when making further improvement or fixing bugs in EC. In > this way it will be easier for later backporting EC to 2.9. > > On Mon, Nov 2, 2015 at 3:48 PM, Vinayakumar B > wrote: > >> +1 for the idea. >> On Nov 3, 2015 07:22, "Zheng, Kai" wrote: >> >>> Sounds good to me. When it's determined to include EC in 2.9 >>> release, it may be good to have a rough release date as Zhe asked, >>> so accordingly the scope of EC can be discussed out. We still have >>> quite a few of things as Phase I follow-on tasks to do before EC can >>> be deployed in a production system. Phase II to develop non-striping >>> EC for cold data would possibly >> be >>> started after that. We might consider to include only Phase I and >>> leave Phase II for next release according to the rough release date. >>> >>> Regards, >>> Kai >>> >>> -Original Message- >>> From: Gangumalla, Uma [mailto:uma.ganguma...@intel.com] >>> Sent: Tuesday, November 03, 2015 5:41 AM >>> To: hdfs-dev@hadoop.apache.org >>> Subject: Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge >>> HDFS-7285 (erasure coding) branch to trunk] >>> >>> +1 for EC to go into 2.9. Yes, 3.x would be long way to go when we >>> +plan to >>> have 2.8 and 2.9 releases. >>> >>> Regards, >>> Uma >>> >>> On 11/2/15, 11:49 AM, "Vinod Vavilapalli" >> wrote: >>> >>>> Forking the thread. Started looking at the 2.8 list, various >>>> features¹ status and arrived here. >>>> >>>> While I understand the pervasive nature of EC and a need for a >>>> significant bake-in, moving this to a 3.x release is not a good idea. >>>> We will surely get a 2.8 out this year and, as needed, I can even >>>> spend time getting started on a 2.9. OTOH, 3.x is long ways off, >>>> and given all the incompatibilities there, it would be a while >>>> before users can get their hands on EC if it were to be only on >>>> 3.x. At best, this may force sites that want EC to backport the >>>> entire EC feature to older releases, at worst this will be repeat >>>> the mess of 0.20 security release >>> forks. >>>> >>>> If we think adding this to 2.8 (even if it switched off) is too >>>> much risk per our original plan, let¹s move this to 2.9, there by >>>> leaving enough time for stability, integration testing and bake-in, >>>> and a realistic chance of having it end up on users¹ clusters soonish. >>>> >>>> +Vinod >>>> >>>>> On Oct 19, 2015, at 1:44 PM, Andrew Wang >>>>> >>>>> wrote: >>>>> >>>>> I think our plan thus far has been to target this for 3.0. I'm >>>>> okay with putting it in branch-2 if we've given a hard look at >>>>> compatibility, but I'll note though that 2.8 is already looking >>>>> like quite a large release, and our release bandwidth has been >>>>> focused on the 2.6 and 2.7 maintenance releases. Adding another >>>>> multi-hundred JIRAs to 2.8 might make it too unwieldy to get out >>>>> the door. If we bump EC past that, 3.0 might very well be our >>>>> next release vehicle. I do plan to revive the 3.0 schedule some >>>>> time next year. With EC and >>>>> JDK8 in a good spot, the only big feature remaining is classpath >>>>> isolation. >>>>> >
RE: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]
Thanks Andrew for pointing this. It sounds good. Yes we have umbrella JIRAs for the follow-on tasks, HDFS-8031 for the HDFS side, and HADOOP-11842 for the HADOOP side. -Original Message- From: Andrew Wang [mailto:andrew.w...@cloudera.com] Sent: Tuesday, November 03, 2015 8:49 AM To: hdfs-dev@hadoop.apache.org Subject: Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk] If we use an umbrella JIRA to categorize all the ongoing EC work, that will let us more easily change the target version later. For instance, if we decide to bump Phase II out of 2.9, then we just need to change the target version of the Phase II umbrella rather than all the subtasks. On Mon, Nov 2, 2015 at 4:26 PM, Zheng, Kai wrote: > Yeah, so for the issues we recently resolved on trunk and are > addressing as follow-on tasks in Phase I, we would label them with "erasure > coding" > and maybe also set the target version as "2.9" for the convenience? > > -Original Message- > From: Jing Zhao [mailto:ji...@apache.org] > Sent: Tuesday, November 03, 2015 8:04 AM > To: hdfs-dev@hadoop.apache.org > Subject: Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge > HDFS-7285 (erasure coding) branch to trunk] > > +1 for the plan about Phase I & II. > > BTW, maybe out of the scope of this thread, just want to mention we > should either move the jira under HDFS-8031 or update the jira > component as "erasure-coding" when making further improvement or > fixing bugs in EC. In this way it will be easier for later backporting EC to > 2.9. > > On Mon, Nov 2, 2015 at 3:48 PM, Vinayakumar B < > vinayakumarb.apa...@gmail.com > > wrote: > > > +1 for the idea. > > On Nov 3, 2015 07:22, "Zheng, Kai" wrote: > > > > > Sounds good to me. When it's determined to include EC in 2.9 > > > release, it may be good to have a rough release date as Zhe asked, > > > so accordingly the scope of EC can be discussed out. We still have > > > quite a few of things as Phase I follow-on tasks to do before EC > > > can be deployed in a production system. Phase II to develop > > > non-striping EC for cold data would possibly > > be > > > started after that. We might consider to include only Phase I and > > > leave Phase II for next release according to the rough release date. > > > > > > Regards, > > > Kai > > > > > > -Original Message- > > > From: Gangumalla, Uma [mailto:uma.ganguma...@intel.com] > > > Sent: Tuesday, November 03, 2015 5:41 AM > > > To: hdfs-dev@hadoop.apache.org > > > Subject: Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge > > > HDFS-7285 (erasure coding) branch to trunk] > > > > > > +1 for EC to go into 2.9. Yes, 3.x would be long way to go when we > > > +plan to > > > have 2.8 and 2.9 releases. > > > > > > Regards, > > > Uma > > > > > > On 11/2/15, 11:49 AM, "Vinod Vavilapalli" > > > > > wrote: > > > > > > >Forking the thread. Started looking at the 2.8 list, various > > > >features¹ status and arrived here. > > > > > > > >While I understand the pervasive nature of EC and a need for a > > > >significant bake-in, moving this to a 3.x release is not a good idea. > > > >We will surely get a 2.8 out this year and, as needed, I can even > > > >spend time getting started on a 2.9. OTOH, 3.x is long ways off, > > > >and given all the incompatibilities there, it would be a while > > > >before users can get their hands on EC if it were to be only on > > > >3.x. At best, this may force sites that want EC to backport the > > > >entire EC feature to older releases, at worst this will be repeat > > > >the mess of 0.20 security release > > > forks. > > > > > > > >If we think adding this to 2.8 (even if it switched off) is too > > > >much risk per our original plan, let¹s move this to 2.9, there by > > > >leaving enough time for stability, integration testing and > > > >bake-in, and a realistic chance of having it end up on users¹ clusters > > > >soonish. > > > > > > > >+Vinod > > > > > > > >> On Oct 19, 2015, at 1:44 PM, Andrew Wang > > > >> > > > >>wrote: > > > >> > > > >> I think our plan thus far has been to target this for 3.0. I'm > > > >>okay with putting it in branch-2
Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]
If we use an umbrella JIRA to categorize all the ongoing EC work, that will let us more easily change the target version later. For instance, if we decide to bump Phase II out of 2.9, then we just need to change the target version of the Phase II umbrella rather than all the subtasks. On Mon, Nov 2, 2015 at 4:26 PM, Zheng, Kai wrote: > Yeah, so for the issues we recently resolved on trunk and are addressing > as follow-on tasks in Phase I, we would label them with "erasure coding" > and maybe also set the target version as "2.9" for the convenience? > > -Original Message- > From: Jing Zhao [mailto:ji...@apache.org] > Sent: Tuesday, November 03, 2015 8:04 AM > To: hdfs-dev@hadoop.apache.org > Subject: Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 > (erasure coding) branch to trunk] > > +1 for the plan about Phase I & II. > > BTW, maybe out of the scope of this thread, just want to mention we should > either move the jira under HDFS-8031 or update the jira component as > "erasure-coding" when making further improvement or fixing bugs in EC. In > this way it will be easier for later backporting EC to 2.9. > > On Mon, Nov 2, 2015 at 3:48 PM, Vinayakumar B < > vinayakumarb.apa...@gmail.com > > wrote: > > > +1 for the idea. > > On Nov 3, 2015 07:22, "Zheng, Kai" wrote: > > > > > Sounds good to me. When it's determined to include EC in 2.9 > > > release, it may be good to have a rough release date as Zhe asked, > > > so accordingly the scope of EC can be discussed out. We still have > > > quite a few of things as Phase I follow-on tasks to do before EC can > > > be deployed in a production system. Phase II to develop non-striping > > > EC for cold data would possibly > > be > > > started after that. We might consider to include only Phase I and > > > leave Phase II for next release according to the rough release date. > > > > > > Regards, > > > Kai > > > > > > -----Original Message- > > > From: Gangumalla, Uma [mailto:uma.ganguma...@intel.com] > > > Sent: Tuesday, November 03, 2015 5:41 AM > > > To: hdfs-dev@hadoop.apache.org > > > Subject: Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge > > > HDFS-7285 (erasure coding) branch to trunk] > > > > > > +1 for EC to go into 2.9. Yes, 3.x would be long way to go when we > > > +plan to > > > have 2.8 and 2.9 releases. > > > > > > Regards, > > > Uma > > > > > > On 11/2/15, 11:49 AM, "Vinod Vavilapalli" > > wrote: > > > > > > >Forking the thread. Started looking at the 2.8 list, various > > > >features¹ status and arrived here. > > > > > > > >While I understand the pervasive nature of EC and a need for a > > > >significant bake-in, moving this to a 3.x release is not a good idea. > > > >We will surely get a 2.8 out this year and, as needed, I can even > > > >spend time getting started on a 2.9. OTOH, 3.x is long ways off, > > > >and given all the incompatibilities there, it would be a while > > > >before users can get their hands on EC if it were to be only on > > > >3.x. At best, this may force sites that want EC to backport the > > > >entire EC feature to older releases, at worst this will be repeat > > > >the mess of 0.20 security release > > > forks. > > > > > > > >If we think adding this to 2.8 (even if it switched off) is too > > > >much risk per our original plan, let¹s move this to 2.9, there by > > > >leaving enough time for stability, integration testing and bake-in, > > > >and a realistic chance of having it end up on users¹ clusters soonish. > > > > > > > >+Vinod > > > > > > > >> On Oct 19, 2015, at 1:44 PM, Andrew Wang > > > >> > > > >>wrote: > > > >> > > > >> I think our plan thus far has been to target this for 3.0. I'm > > > >>okay with putting it in branch-2 if we've given a hard look at > > > >>compatibility, but I'll note though that 2.8 is already looking > > > >>like quite a large release, and our release bandwidth has been > > > >>focused on the 2.6 and 2.7 maintenance releases. Adding another > > > >>multi-hundred JIRAs to 2.8 might make it too unwieldy to get out > > > >>the door. If we bump EC past that, 3.0 might very well be our > > > >>next release vehicle. I
RE: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]
Yeah, so for the issues we recently resolved on trunk and are addressing as follow-on tasks in Phase I, we would label them with "erasure coding" and maybe also set the target version as "2.9" for the convenience? -Original Message- From: Jing Zhao [mailto:ji...@apache.org] Sent: Tuesday, November 03, 2015 8:04 AM To: hdfs-dev@hadoop.apache.org Subject: Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk] +1 for the plan about Phase I & II. BTW, maybe out of the scope of this thread, just want to mention we should either move the jira under HDFS-8031 or update the jira component as "erasure-coding" when making further improvement or fixing bugs in EC. In this way it will be easier for later backporting EC to 2.9. On Mon, Nov 2, 2015 at 3:48 PM, Vinayakumar B wrote: > +1 for the idea. > On Nov 3, 2015 07:22, "Zheng, Kai" wrote: > > > Sounds good to me. When it's determined to include EC in 2.9 > > release, it may be good to have a rough release date as Zhe asked, > > so accordingly the scope of EC can be discussed out. We still have > > quite a few of things as Phase I follow-on tasks to do before EC can > > be deployed in a production system. Phase II to develop non-striping > > EC for cold data would possibly > be > > started after that. We might consider to include only Phase I and > > leave Phase II for next release according to the rough release date. > > > > Regards, > > Kai > > > > -Original Message- > > From: Gangumalla, Uma [mailto:uma.ganguma...@intel.com] > > Sent: Tuesday, November 03, 2015 5:41 AM > > To: hdfs-dev@hadoop.apache.org > > Subject: Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge > > HDFS-7285 (erasure coding) branch to trunk] > > > > +1 for EC to go into 2.9. Yes, 3.x would be long way to go when we > > +plan to > > have 2.8 and 2.9 releases. > > > > Regards, > > Uma > > > > On 11/2/15, 11:49 AM, "Vinod Vavilapalli" > wrote: > > > > >Forking the thread. Started looking at the 2.8 list, various > > >features¹ status and arrived here. > > > > > >While I understand the pervasive nature of EC and a need for a > > >significant bake-in, moving this to a 3.x release is not a good idea. > > >We will surely get a 2.8 out this year and, as needed, I can even > > >spend time getting started on a 2.9. OTOH, 3.x is long ways off, > > >and given all the incompatibilities there, it would be a while > > >before users can get their hands on EC if it were to be only on > > >3.x. At best, this may force sites that want EC to backport the > > >entire EC feature to older releases, at worst this will be repeat > > >the mess of 0.20 security release > > forks. > > > > > >If we think adding this to 2.8 (even if it switched off) is too > > >much risk per our original plan, let¹s move this to 2.9, there by > > >leaving enough time for stability, integration testing and bake-in, > > >and a realistic chance of having it end up on users¹ clusters soonish. > > > > > >+Vinod > > > > > >> On Oct 19, 2015, at 1:44 PM, Andrew Wang > > >> > > >>wrote: > > >> > > >> I think our plan thus far has been to target this for 3.0. I'm > > >>okay with putting it in branch-2 if we've given a hard look at > > >>compatibility, but I'll note though that 2.8 is already looking > > >>like quite a large release, and our release bandwidth has been > > >>focused on the 2.6 and 2.7 maintenance releases. Adding another > > >>multi-hundred JIRAs to 2.8 might make it too unwieldy to get out > > >>the door. If we bump EC past that, 3.0 might very well be our > > >>next release vehicle. I do plan to revive the 3.0 schedule some > > >>time next year. With EC and > > >>JDK8 in a good spot, the only big feature remaining is classpath > > >>isolation. > > >> > > >> EC is also a pretty fundamental change to HDFS. Even if it's > > >>compatible, in terms of size and impact it might best belong in a > > >>new major release. > > >> > > >> Best, > > >> Andrew > > >> > > >> On Fri, Oct 16, 2015 at 7:04 PM, Vinayakumar B < > > >> vinayakumarb.apa...@gmail.com> wrote: > > >> > > >>> Is anyone else also thinks that feature is ready to goto > > >>>branc
Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]
+1 for the plan about Phase I & II. BTW, maybe out of the scope of this thread, just want to mention we should either move the jira under HDFS-8031 or update the jira component as "erasure-coding" when making further improvement or fixing bugs in EC. In this way it will be easier for later backporting EC to 2.9. On Mon, Nov 2, 2015 at 3:48 PM, Vinayakumar B wrote: > +1 for the idea. > On Nov 3, 2015 07:22, "Zheng, Kai" wrote: > > > Sounds good to me. When it's determined to include EC in 2.9 release, it > > may be good to have a rough release date as Zhe asked, so accordingly the > > scope of EC can be discussed out. We still have quite a few of things as > > Phase I follow-on tasks to do before EC can be deployed in a production > > system. Phase II to develop non-striping EC for cold data would possibly > be > > started after that. We might consider to include only Phase I and leave > > Phase II for next release according to the rough release date. > > > > Regards, > > Kai > > > > -Original Message- > > From: Gangumalla, Uma [mailto:uma.ganguma...@intel.com] > > Sent: Tuesday, November 03, 2015 5:41 AM > > To: hdfs-dev@hadoop.apache.org > > Subject: Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 > > (erasure coding) branch to trunk] > > > > +1 for EC to go into 2.9. Yes, 3.x would be long way to go when we plan > > +to > > have 2.8 and 2.9 releases. > > > > Regards, > > Uma > > > > On 11/2/15, 11:49 AM, "Vinod Vavilapalli" > wrote: > > > > >Forking the thread. Started looking at the 2.8 list, various features¹ > > >status and arrived here. > > > > > >While I understand the pervasive nature of EC and a need for a > > >significant bake-in, moving this to a 3.x release is not a good idea. > > >We will surely get a 2.8 out this year and, as needed, I can even spend > > >time getting started on a 2.9. OTOH, 3.x is long ways off, and given > > >all the incompatibilities there, it would be a while before users can > > >get their hands on EC if it were to be only on 3.x. At best, this may > > >force sites that want EC to backport the entire EC feature to older > > >releases, at worst this will be repeat the mess of 0.20 security release > > forks. > > > > > >If we think adding this to 2.8 (even if it switched off) is too much > > >risk per our original plan, let¹s move this to 2.9, there by leaving > > >enough time for stability, integration testing and bake-in, and a > > >realistic chance of having it end up on users¹ clusters soonish. > > > > > >+Vinod > > > > > >> On Oct 19, 2015, at 1:44 PM, Andrew Wang > > >>wrote: > > >> > > >> I think our plan thus far has been to target this for 3.0. I'm okay > > >>with putting it in branch-2 if we've given a hard look at > > >>compatibility, but I'll note though that 2.8 is already looking like > > >>quite a large release, and our release bandwidth has been focused on > > >>the 2.6 and 2.7 maintenance releases. Adding another multi-hundred > > >>JIRAs to 2.8 might make it too unwieldy to get out the door. If we > > >>bump EC past that, 3.0 might very well be our next release vehicle. I > > >>do plan to revive the 3.0 schedule some time next year. With EC and > > >>JDK8 in a good spot, the only big feature remaining is classpath > > >>isolation. > > >> > > >> EC is also a pretty fundamental change to HDFS. Even if it's > > >>compatible, in terms of size and impact it might best belong in a new > > >>major release. > > >> > > >> Best, > > >> Andrew > > >> > > >> On Fri, Oct 16, 2015 at 7:04 PM, Vinayakumar B < > > >> vinayakumarb.apa...@gmail.com> wrote: > > >> > > >>> Is anyone else also thinks that feature is ready to goto branch-2 > > >>>as well? > > >>> > > >>> Its > 2 weeks EC landed on trunk. IMo Looks Its quite stable since > > >>>then and ready to go in branch-2. > > >>> > > >>> -Vinay > > >>> On Oct 6, 2015 12:51 AM, "Zhe Zhang" wrote: > > >>> > > >>>> Thanks Vinay for capturing the issue and Uma for offering the help. > > >>>> > > >>>> --- > > >>>> Zhe Zhang > > >>>
RE: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]
+1 for the idea. On Nov 3, 2015 07:22, "Zheng, Kai" wrote: > Sounds good to me. When it's determined to include EC in 2.9 release, it > may be good to have a rough release date as Zhe asked, so accordingly the > scope of EC can be discussed out. We still have quite a few of things as > Phase I follow-on tasks to do before EC can be deployed in a production > system. Phase II to develop non-striping EC for cold data would possibly be > started after that. We might consider to include only Phase I and leave > Phase II for next release according to the rough release date. > > Regards, > Kai > > -Original Message- > From: Gangumalla, Uma [mailto:uma.ganguma...@intel.com] > Sent: Tuesday, November 03, 2015 5:41 AM > To: hdfs-dev@hadoop.apache.org > Subject: Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 > (erasure coding) branch to trunk] > > +1 for EC to go into 2.9. Yes, 3.x would be long way to go when we plan > +to > have 2.8 and 2.9 releases. > > Regards, > Uma > > On 11/2/15, 11:49 AM, "Vinod Vavilapalli" wrote: > > >Forking the thread. Started looking at the 2.8 list, various features¹ > >status and arrived here. > > > >While I understand the pervasive nature of EC and a need for a > >significant bake-in, moving this to a 3.x release is not a good idea. > >We will surely get a 2.8 out this year and, as needed, I can even spend > >time getting started on a 2.9. OTOH, 3.x is long ways off, and given > >all the incompatibilities there, it would be a while before users can > >get their hands on EC if it were to be only on 3.x. At best, this may > >force sites that want EC to backport the entire EC feature to older > >releases, at worst this will be repeat the mess of 0.20 security release > forks. > > > >If we think adding this to 2.8 (even if it switched off) is too much > >risk per our original plan, let¹s move this to 2.9, there by leaving > >enough time for stability, integration testing and bake-in, and a > >realistic chance of having it end up on users¹ clusters soonish. > > > >+Vinod > > > >> On Oct 19, 2015, at 1:44 PM, Andrew Wang > >>wrote: > >> > >> I think our plan thus far has been to target this for 3.0. I'm okay > >>with putting it in branch-2 if we've given a hard look at > >>compatibility, but I'll note though that 2.8 is already looking like > >>quite a large release, and our release bandwidth has been focused on > >>the 2.6 and 2.7 maintenance releases. Adding another multi-hundred > >>JIRAs to 2.8 might make it too unwieldy to get out the door. If we > >>bump EC past that, 3.0 might very well be our next release vehicle. I > >>do plan to revive the 3.0 schedule some time next year. With EC and > >>JDK8 in a good spot, the only big feature remaining is classpath > >>isolation. > >> > >> EC is also a pretty fundamental change to HDFS. Even if it's > >>compatible, in terms of size and impact it might best belong in a new > >>major release. > >> > >> Best, > >> Andrew > >> > >> On Fri, Oct 16, 2015 at 7:04 PM, Vinayakumar B < > >> vinayakumarb.apa...@gmail.com> wrote: > >> > >>> Is anyone else also thinks that feature is ready to goto branch-2 > >>>as well? > >>> > >>> Its > 2 weeks EC landed on trunk. IMo Looks Its quite stable since > >>>then and ready to go in branch-2. > >>> > >>> -Vinay > >>> On Oct 6, 2015 12:51 AM, "Zhe Zhang" wrote: > >>> > >>>> Thanks Vinay for capturing the issue and Uma for offering the help. > >>>> > >>>> --- > >>>> Zhe Zhang > >>>> > >>>> On Mon, Oct 5, 2015 at 12:19 PM, Gangumalla, Uma < > >>> uma.ganguma...@intel.com > >>>>> > >>>> wrote: > >>>> > >>>>> Vinay, > >>>>> > >>>>> > >>>>> I would merge them as part of HDFS-9182. > >>>>> > >>>>> Thanks, > >>>>> Uma > >>>>> > >>>>> > >>>>> > >>>>> On 10/5/15, 12:48 AM, "Vinayakumar B" > >>>>>wrote: > >>>>> > >>>>>> Hi Andrew, > >>>>>> I see CHANGES.txt entries not yet merged from > >>> CHANGES-HDFS-EC-7285.txt. > >>>>>> > >>>&
RE: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]
Sounds good to me. When it's determined to include EC in 2.9 release, it may be good to have a rough release date as Zhe asked, so accordingly the scope of EC can be discussed out. We still have quite a few of things as Phase I follow-on tasks to do before EC can be deployed in a production system. Phase II to develop non-striping EC for cold data would possibly be started after that. We might consider to include only Phase I and leave Phase II for next release according to the rough release date. Regards, Kai -Original Message- From: Gangumalla, Uma [mailto:uma.ganguma...@intel.com] Sent: Tuesday, November 03, 2015 5:41 AM To: hdfs-dev@hadoop.apache.org Subject: Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk] +1 for EC to go into 2.9. Yes, 3.x would be long way to go when we plan +to have 2.8 and 2.9 releases. Regards, Uma On 11/2/15, 11:49 AM, "Vinod Vavilapalli" wrote: >Forking the thread. Started looking at the 2.8 list, various features¹ >status and arrived here. > >While I understand the pervasive nature of EC and a need for a >significant bake-in, moving this to a 3.x release is not a good idea. >We will surely get a 2.8 out this year and, as needed, I can even spend >time getting started on a 2.9. OTOH, 3.x is long ways off, and given >all the incompatibilities there, it would be a while before users can >get their hands on EC if it were to be only on 3.x. At best, this may >force sites that want EC to backport the entire EC feature to older >releases, at worst this will be repeat the mess of 0.20 security release forks. > >If we think adding this to 2.8 (even if it switched off) is too much >risk per our original plan, let¹s move this to 2.9, there by leaving >enough time for stability, integration testing and bake-in, and a >realistic chance of having it end up on users¹ clusters soonish. > >+Vinod > >> On Oct 19, 2015, at 1:44 PM, Andrew Wang >>wrote: >> >> I think our plan thus far has been to target this for 3.0. I'm okay >>with putting it in branch-2 if we've given a hard look at >>compatibility, but I'll note though that 2.8 is already looking like >>quite a large release, and our release bandwidth has been focused on >>the 2.6 and 2.7 maintenance releases. Adding another multi-hundred >>JIRAs to 2.8 might make it too unwieldy to get out the door. If we >>bump EC past that, 3.0 might very well be our next release vehicle. I >>do plan to revive the 3.0 schedule some time next year. With EC and >>JDK8 in a good spot, the only big feature remaining is classpath >>isolation. >> >> EC is also a pretty fundamental change to HDFS. Even if it's >>compatible, in terms of size and impact it might best belong in a new >>major release. >> >> Best, >> Andrew >> >> On Fri, Oct 16, 2015 at 7:04 PM, Vinayakumar B < >> vinayakumarb.apa...@gmail.com> wrote: >> >>> Is anyone else also thinks that feature is ready to goto branch-2 >>>as well? >>> >>> Its > 2 weeks EC landed on trunk. IMo Looks Its quite stable since >>>then and ready to go in branch-2. >>> >>> -Vinay >>> On Oct 6, 2015 12:51 AM, "Zhe Zhang" wrote: >>> >>>> Thanks Vinay for capturing the issue and Uma for offering the help. >>>> >>>> --- >>>> Zhe Zhang >>>> >>>> On Mon, Oct 5, 2015 at 12:19 PM, Gangumalla, Uma < >>> uma.ganguma...@intel.com >>>>> >>>> wrote: >>>> >>>>> Vinay, >>>>> >>>>> >>>>> I would merge them as part of HDFS-9182. >>>>> >>>>> Thanks, >>>>> Uma >>>>> >>>>> >>>>> >>>>> On 10/5/15, 12:48 AM, "Vinayakumar B" >>>>>wrote: >>>>> >>>>>> Hi Andrew, >>>>>> I see CHANGES.txt entries not yet merged from >>> CHANGES-HDFS-EC-7285.txt. >>>>>> >>>>>> Was this intentional? >>>>>> >>>>>> Regards, >>>>>> Vinay >>>>>> >>>>>> On Wed, Sep 30, 2015 at 9:15 PM, Andrew Wang < >>> andrew.w...@cloudera.com> >>>>>> wrote: >>>>>> >>>>>>> Branch has been merged to trunk, thanks again to everyone who >>>>>>>worked >>>> on >>>>>>> the >>>
Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]
Yes, I’ve already started looking at 2.8.0, that is exactly how I ended up with this discussion on the state of EC. +Vinod On Nov 2, 2015, at 3:02 PM, Haohui Mai mailto:ricet...@gmail.com>> wrote: Is it a good time to start the discussion on the issues of releasing 2.8?
Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]
+1 on putting EC on 2.9. Is it a good time to start the discussion on the issues of releasing 2.8? ~Haohui On Mon, Nov 2, 2015 at 1:40 PM, Gangumalla, Uma wrote: > +1 for EC to go into 2.9. Yes, 3.x would be long way to go when we plan to > have 2.8 and 2.9 releases. > > Regards, > Uma > > On 11/2/15, 11:49 AM, "Vinod Vavilapalli" wrote: > >>Forking the thread. Started looking at the 2.8 list, various features¹ >>status and arrived here. >> >>While I understand the pervasive nature of EC and a need for a >>significant bake-in, moving this to a 3.x release is not a good idea. We >>will surely get a 2.8 out this year and, as needed, I can even spend time >>getting started on a 2.9. OTOH, 3.x is long ways off, and given all the >>incompatibilities there, it would be a while before users can get their >>hands on EC if it were to be only on 3.x. At best, this may force sites >>that want EC to backport the entire EC feature to older releases, at >>worst this will be repeat the mess of 0.20 security release forks. >> >>If we think adding this to 2.8 (even if it switched off) is too much risk >>per our original plan, let¹s move this to 2.9, there by leaving enough >>time for stability, integration testing and bake-in, and a realistic >>chance of having it end up on users¹ clusters soonish. >> >>+Vinod >> >>> On Oct 19, 2015, at 1:44 PM, Andrew Wang >>>wrote: >>> >>> I think our plan thus far has been to target this for 3.0. I'm okay with >>> putting it in branch-2 if we've given a hard look at compatibility, but >>> I'll note though that 2.8 is already looking like quite a large release, >>> and our release bandwidth has been focused on the 2.6 and 2.7 >>>maintenance >>> releases. Adding another multi-hundred JIRAs to 2.8 might make it too >>> unwieldy to get out the door. If we bump EC past that, 3.0 might very >>>well >>> be our next release vehicle. I do plan to revive the 3.0 schedule some >>>time >>> next year. With EC and JDK8 in a good spot, the only big feature >>>remaining >>> is classpath isolation. >>> >>> EC is also a pretty fundamental change to HDFS. Even if it's >>>compatible, in >>> terms of size and impact it might best belong in a new major release. >>> >>> Best, >>> Andrew >>> >>> On Fri, Oct 16, 2015 at 7:04 PM, Vinayakumar B < >>> vinayakumarb.apa...@gmail.com> wrote: >>> Is anyone else also thinks that feature is ready to goto branch-2 as well? Its > 2 weeks EC landed on trunk. IMo Looks Its quite stable since then and ready to go in branch-2. -Vinay On Oct 6, 2015 12:51 AM, "Zhe Zhang" wrote: > Thanks Vinay for capturing the issue and Uma for offering the help. > > --- > Zhe Zhang > > On Mon, Oct 5, 2015 at 12:19 PM, Gangumalla, Uma < uma.ganguma...@intel.com >> > wrote: > >> Vinay, >> >> >> I would merge them as part of HDFS-9182. >> >> Thanks, >> Uma >> >> >> >> On 10/5/15, 12:48 AM, "Vinayakumar B" >>wrote: >> >>> Hi Andrew, >>> I see CHANGES.txt entries not yet merged from CHANGES-HDFS-EC-7285.txt. >>> >>> Was this intentional? >>> >>> Regards, >>> Vinay >>> >>> On Wed, Sep 30, 2015 at 9:15 PM, Andrew Wang < andrew.w...@cloudera.com> >>> wrote: >>> Branch has been merged to trunk, thanks again to everyone who worked > on the feature! On Tue, Sep 29, 2015 at 10:44 PM, Zhe Zhang wrote: > Thanks everyone who has participated in this discussion. > > With 7 +1's (5 binding and 2 non-binding), and no -1, this vote has passed. > I will do a final 'git merge' with trunk and work with Andrew to > merge the > branch to trunk. I'll update on this thread when the merge is done. > > --- > Zhe Zhang > > On Thu, Sep 24, 2015 at 11:08 PM, Liu, Yi A wrote: > >> (Change it to binding.) >> >> +1 >> I have been involved in the development and code review on the feature >> branch. It's a great feature and I think it's ready to merge it > into > trunk. >> >> Thanks all for the contribution. >> >> Regards, >> Yi Liu >> >> >> -Original Message- >> From: Liu, Yi A >> Sent: Friday, September 25, 2015 1:51 PM >> To: hdfs-dev@hadoop.apache.org >> Subject: RE: [VOTE] Merge HDFS-7285 (erasure coding) branch to > trunk >> >> +1 (non-binding) >> I have been involved in the development and code review on the feature >> branch. It's a great feature and I think it's ready to merge it > into > trunk. >> >> Thanks all for the contribution. >> >> Regards, >> Yi Liu
Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]
+1 for EC to go into 2.9. Yes, 3.x would be long way to go when we plan to have 2.8 and 2.9 releases. Regards, Uma On 11/2/15, 11:49 AM, "Vinod Vavilapalli" wrote: >Forking the thread. Started looking at the 2.8 list, various features¹ >status and arrived here. > >While I understand the pervasive nature of EC and a need for a >significant bake-in, moving this to a 3.x release is not a good idea. We >will surely get a 2.8 out this year and, as needed, I can even spend time >getting started on a 2.9. OTOH, 3.x is long ways off, and given all the >incompatibilities there, it would be a while before users can get their >hands on EC if it were to be only on 3.x. At best, this may force sites >that want EC to backport the entire EC feature to older releases, at >worst this will be repeat the mess of 0.20 security release forks. > >If we think adding this to 2.8 (even if it switched off) is too much risk >per our original plan, let¹s move this to 2.9, there by leaving enough >time for stability, integration testing and bake-in, and a realistic >chance of having it end up on users¹ clusters soonish. > >+Vinod > >> On Oct 19, 2015, at 1:44 PM, Andrew Wang >>wrote: >> >> I think our plan thus far has been to target this for 3.0. I'm okay with >> putting it in branch-2 if we've given a hard look at compatibility, but >> I'll note though that 2.8 is already looking like quite a large release, >> and our release bandwidth has been focused on the 2.6 and 2.7 >>maintenance >> releases. Adding another multi-hundred JIRAs to 2.8 might make it too >> unwieldy to get out the door. If we bump EC past that, 3.0 might very >>well >> be our next release vehicle. I do plan to revive the 3.0 schedule some >>time >> next year. With EC and JDK8 in a good spot, the only big feature >>remaining >> is classpath isolation. >> >> EC is also a pretty fundamental change to HDFS. Even if it's >>compatible, in >> terms of size and impact it might best belong in a new major release. >> >> Best, >> Andrew >> >> On Fri, Oct 16, 2015 at 7:04 PM, Vinayakumar B < >> vinayakumarb.apa...@gmail.com> wrote: >> >>> Is anyone else also thinks that feature is ready to goto branch-2 as >>>well? >>> >>> Its > 2 weeks EC landed on trunk. IMo Looks Its quite stable since >>>then and >>> ready to go in branch-2. >>> >>> -Vinay >>> On Oct 6, 2015 12:51 AM, "Zhe Zhang" wrote: >>> Thanks Vinay for capturing the issue and Uma for offering the help. --- Zhe Zhang On Mon, Oct 5, 2015 at 12:19 PM, Gangumalla, Uma < >>> uma.ganguma...@intel.com > wrote: > Vinay, > > > I would merge them as part of HDFS-9182. > > Thanks, > Uma > > > > On 10/5/15, 12:48 AM, "Vinayakumar B" >wrote: > >> Hi Andrew, >> I see CHANGES.txt entries not yet merged from >>> CHANGES-HDFS-EC-7285.txt. >> >> Was this intentional? >> >> Regards, >> Vinay >> >> On Wed, Sep 30, 2015 at 9:15 PM, Andrew Wang < >>> andrew.w...@cloudera.com> >> wrote: >> >>> Branch has been merged to trunk, thanks again to everyone who >>>worked on >>> the >>> feature! >>> >>> On Tue, Sep 29, 2015 at 10:44 PM, Zhe Zhang >>> wrote: >>> Thanks everyone who has participated in this discussion. With 7 +1's (5 binding and 2 non-binding), and no -1, this vote >>> has >>> passed. I will do a final 'git merge' with trunk and work with Andrew to merge >>> the branch to trunk. I'll update on this thread when the merge is >>> done. --- Zhe Zhang On Thu, Sep 24, 2015 at 11:08 PM, Liu, Yi A >>> wrote: > (Change it to binding.) > > +1 > I have been involved in the development and code review on the >>> feature > branch. It's a great feature and I think it's ready to merge it into trunk. > > Thanks all for the contribution. > > Regards, > Yi Liu > > > -Original Message- > From: Liu, Yi A > Sent: Friday, September 25, 2015 1:51 PM > To: hdfs-dev@hadoop.apache.org > Subject: RE: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk > > +1 (non-binding) > I have been involved in the development and code review on the >>> feature > branch. It's a great feature and I think it's ready to merge it into trunk. > > Thanks all for the contribution. > > Regards, > Yi Liu > > > -Original Message- > From: Vinayakumar B [mailto:vinayakum...@apache.org] > Sent: Friday, September 25, 2015 12:21 PM > To: hdfs-dev@hadoop.apache.org > Subject: Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to
Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]
Thanks Vinod for the proposal and Andrew/Jing for the comments. 2.9 sounds a good plan. As Andrew pointed out, 2.8 is already quite big. And even when disabled, EC logic has been baked in NN pretty deeply. Do we have a tentative date or estimate for 2.9? --- Zhe Zhang On Mon, Nov 2, 2015 at 1:22 PM, Jing Zhao wrote: > Thanks for the discussion, Vinod and Andrew. Backporting EC to 2.9 sounds > good to me. > > On Mon, Nov 2, 2015 at 12:06 PM, Andrew Wang > wrote: > > > Thanks for forking the thread Vinod, > > > > SGTM, though I really do recommend waiting for 2.9 given the current size > > of 2.8. I'm not a fan of an "off by default" half-measure, since it > doesn't > > change our compatibility requirements, and there's some major NN surgery > > that can't really be disabled. > > > > If we do find a major user who's backported this to their own branch-2 > > fork, I agree that's motivation to get it in an upstream release > quicker. I > > haven't heard anything along these lines though. > > > > On Mon, Nov 2, 2015 at 11:49 AM, Vinod Vavilapalli < > > vino...@hortonworks.com> > > wrote: > > > > > Forking the thread. Started looking at the 2.8 list, various features’ > > > status and arrived here. > > > > > > While I understand the pervasive nature of EC and a need for a > > significant > > > bake-in, moving this to a 3.x release is not a good idea. We will > surely > > > get a 2.8 out this year and, as needed, I can even spend time getting > > > started on a 2.9. OTOH, 3.x is long ways off, and given all the > > > incompatibilities there, it would be a while before users can get their > > > hands on EC if it were to be only on 3.x. At best, this may force sites > > > that want EC to backport the entire EC feature to older releases, at > > worst > > > this will be repeat the mess of 0.20 security release forks. > > > > > > If we think adding this to 2.8 (even if it switched off) is too much > risk > > > per our original plan, let’s move this to 2.9, there by leaving enough > > time > > > for stability, integration testing and bake-in, and a realistic chance > of > > > having it end up on users’ clusters soonish. > > > > > > +Vinod > > > > > > > On Oct 19, 2015, at 1:44 PM, Andrew Wang > > > wrote: > > > > > > > > I think our plan thus far has been to target this for 3.0. I'm okay > > with > > > > putting it in branch-2 if we've given a hard look at compatibility, > but > > > > I'll note though that 2.8 is already looking like quite a large > > release, > > > > and our release bandwidth has been focused on the 2.6 and 2.7 > > maintenance > > > > releases. Adding another multi-hundred JIRAs to 2.8 might make it too > > > > unwieldy to get out the door. If we bump EC past that, 3.0 might very > > > well > > > > be our next release vehicle. I do plan to revive the 3.0 schedule > some > > > time > > > > next year. With EC and JDK8 in a good spot, the only big feature > > > remaining > > > > is classpath isolation. > > > > > > > > EC is also a pretty fundamental change to HDFS. Even if it's > > compatible, > > > in > > > > terms of size and impact it might best belong in a new major release. > > > > > > > > Best, > > > > Andrew > > > > > > > > On Fri, Oct 16, 2015 at 7:04 PM, Vinayakumar B < > > > > vinayakumarb.apa...@gmail.com> wrote: > > > > > > > >> Is anyone else also thinks that feature is ready to goto branch-2 > as > > > well? > > > >> > > > >> Its > 2 weeks EC landed on trunk. IMo Looks Its quite stable since > > then > > > and > > > >> ready to go in branch-2. > > > >> > > > >> -Vinay > > > >> On Oct 6, 2015 12:51 AM, "Zhe Zhang" wrote: > > > >> > > > >>> Thanks Vinay for capturing the issue and Uma for offering the help. > > > >>> > > > >>> --- > > > >>> Zhe Zhang > > > >>> > > > >>> On Mon, Oct 5, 2015 at 12:19 PM, Gangumalla, Uma < > > > >> uma.ganguma...@intel.com > > > > > > >>> wrote: > > > >>> > > > Vinay, > > > > > > > > > I would merge them as part of HDFS-9182. > > > > > > Thanks, > > > Uma > > > > > > > > > > > > On 10/5/15, 12:48 AM, "Vinayakumar B" > > > wrote: > > > > > > > Hi Andrew, > > > > I see CHANGES.txt entries not yet merged from > > > >> CHANGES-HDFS-EC-7285.txt. > > > > > > > > Was this intentional? > > > > > > > > Regards, > > > > Vinay > > > > > > > > On Wed, Sep 30, 2015 at 9:15 PM, Andrew Wang < > > > >> andrew.w...@cloudera.com> > > > > wrote: > > > > > > > >> Branch has been merged to trunk, thanks again to everyone who > > worked > > > >>> on > > > >> the > > > >> feature! > > > >> > > > >> On Tue, Sep 29, 2015 at 10:44 PM, Zhe Zhang < > > zhezh...@cloudera.com> > > > >> wrote: > > > >> > > > >>> Thanks everyone who has participated in this discussion. > > > >>> > > > >>> With 7 +1's (5 binding and 2 non-binding), and no -1, this vote > > > >> has > > > >> passed. > > > >>> I will do a final 'git merge' with trunk
Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]
Thanks for the discussion, Vinod and Andrew. Backporting EC to 2.9 sounds good to me. On Mon, Nov 2, 2015 at 12:06 PM, Andrew Wang wrote: > Thanks for forking the thread Vinod, > > SGTM, though I really do recommend waiting for 2.9 given the current size > of 2.8. I'm not a fan of an "off by default" half-measure, since it doesn't > change our compatibility requirements, and there's some major NN surgery > that can't really be disabled. > > If we do find a major user who's backported this to their own branch-2 > fork, I agree that's motivation to get it in an upstream release quicker. I > haven't heard anything along these lines though. > > On Mon, Nov 2, 2015 at 11:49 AM, Vinod Vavilapalli < > vino...@hortonworks.com> > wrote: > > > Forking the thread. Started looking at the 2.8 list, various features’ > > status and arrived here. > > > > While I understand the pervasive nature of EC and a need for a > significant > > bake-in, moving this to a 3.x release is not a good idea. We will surely > > get a 2.8 out this year and, as needed, I can even spend time getting > > started on a 2.9. OTOH, 3.x is long ways off, and given all the > > incompatibilities there, it would be a while before users can get their > > hands on EC if it were to be only on 3.x. At best, this may force sites > > that want EC to backport the entire EC feature to older releases, at > worst > > this will be repeat the mess of 0.20 security release forks. > > > > If we think adding this to 2.8 (even if it switched off) is too much risk > > per our original plan, let’s move this to 2.9, there by leaving enough > time > > for stability, integration testing and bake-in, and a realistic chance of > > having it end up on users’ clusters soonish. > > > > +Vinod > > > > > On Oct 19, 2015, at 1:44 PM, Andrew Wang > > wrote: > > > > > > I think our plan thus far has been to target this for 3.0. I'm okay > with > > > putting it in branch-2 if we've given a hard look at compatibility, but > > > I'll note though that 2.8 is already looking like quite a large > release, > > > and our release bandwidth has been focused on the 2.6 and 2.7 > maintenance > > > releases. Adding another multi-hundred JIRAs to 2.8 might make it too > > > unwieldy to get out the door. If we bump EC past that, 3.0 might very > > well > > > be our next release vehicle. I do plan to revive the 3.0 schedule some > > time > > > next year. With EC and JDK8 in a good spot, the only big feature > > remaining > > > is classpath isolation. > > > > > > EC is also a pretty fundamental change to HDFS. Even if it's > compatible, > > in > > > terms of size and impact it might best belong in a new major release. > > > > > > Best, > > > Andrew > > > > > > On Fri, Oct 16, 2015 at 7:04 PM, Vinayakumar B < > > > vinayakumarb.apa...@gmail.com> wrote: > > > > > >> Is anyone else also thinks that feature is ready to goto branch-2 as > > well? > > >> > > >> Its > 2 weeks EC landed on trunk. IMo Looks Its quite stable since > then > > and > > >> ready to go in branch-2. > > >> > > >> -Vinay > > >> On Oct 6, 2015 12:51 AM, "Zhe Zhang" wrote: > > >> > > >>> Thanks Vinay for capturing the issue and Uma for offering the help. > > >>> > > >>> --- > > >>> Zhe Zhang > > >>> > > >>> On Mon, Oct 5, 2015 at 12:19 PM, Gangumalla, Uma < > > >> uma.ganguma...@intel.com > > > > >>> wrote: > > >>> > > Vinay, > > > > > > I would merge them as part of HDFS-9182. > > > > Thanks, > > Uma > > > > > > > > On 10/5/15, 12:48 AM, "Vinayakumar B" > > wrote: > > > > > Hi Andrew, > > > I see CHANGES.txt entries not yet merged from > > >> CHANGES-HDFS-EC-7285.txt. > > > > > > Was this intentional? > > > > > > Regards, > > > Vinay > > > > > > On Wed, Sep 30, 2015 at 9:15 PM, Andrew Wang < > > >> andrew.w...@cloudera.com> > > > wrote: > > > > > >> Branch has been merged to trunk, thanks again to everyone who > worked > > >>> on > > >> the > > >> feature! > > >> > > >> On Tue, Sep 29, 2015 at 10:44 PM, Zhe Zhang < > zhezh...@cloudera.com> > > >> wrote: > > >> > > >>> Thanks everyone who has participated in this discussion. > > >>> > > >>> With 7 +1's (5 binding and 2 non-binding), and no -1, this vote > > >> has > > >> passed. > > >>> I will do a final 'git merge' with trunk and work with Andrew to > > >>> merge > > >> the > > >>> branch to trunk. I'll update on this thread when the merge is > > >> done. > > >>> > > >>> --- > > >>> Zhe Zhang > > >>> > > >>> On Thu, Sep 24, 2015 at 11:08 PM, Liu, Yi A > > >> wrote: > > >>> > > (Change it to binding.) > > > > +1 > > I have been involved in the development and code review on the > > >> feature > > branch. It's a great feature and I think it's ready to merge it > > >>> into > > >>> trunk. > > > > Thanks all for the con
Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]
Thanks for forking the thread Vinod, SGTM, though I really do recommend waiting for 2.9 given the current size of 2.8. I'm not a fan of an "off by default" half-measure, since it doesn't change our compatibility requirements, and there's some major NN surgery that can't really be disabled. If we do find a major user who's backported this to their own branch-2 fork, I agree that's motivation to get it in an upstream release quicker. I haven't heard anything along these lines though. On Mon, Nov 2, 2015 at 11:49 AM, Vinod Vavilapalli wrote: > Forking the thread. Started looking at the 2.8 list, various features’ > status and arrived here. > > While I understand the pervasive nature of EC and a need for a significant > bake-in, moving this to a 3.x release is not a good idea. We will surely > get a 2.8 out this year and, as needed, I can even spend time getting > started on a 2.9. OTOH, 3.x is long ways off, and given all the > incompatibilities there, it would be a while before users can get their > hands on EC if it were to be only on 3.x. At best, this may force sites > that want EC to backport the entire EC feature to older releases, at worst > this will be repeat the mess of 0.20 security release forks. > > If we think adding this to 2.8 (even if it switched off) is too much risk > per our original plan, let’s move this to 2.9, there by leaving enough time > for stability, integration testing and bake-in, and a realistic chance of > having it end up on users’ clusters soonish. > > +Vinod > > > On Oct 19, 2015, at 1:44 PM, Andrew Wang > wrote: > > > > I think our plan thus far has been to target this for 3.0. I'm okay with > > putting it in branch-2 if we've given a hard look at compatibility, but > > I'll note though that 2.8 is already looking like quite a large release, > > and our release bandwidth has been focused on the 2.6 and 2.7 maintenance > > releases. Adding another multi-hundred JIRAs to 2.8 might make it too > > unwieldy to get out the door. If we bump EC past that, 3.0 might very > well > > be our next release vehicle. I do plan to revive the 3.0 schedule some > time > > next year. With EC and JDK8 in a good spot, the only big feature > remaining > > is classpath isolation. > > > > EC is also a pretty fundamental change to HDFS. Even if it's compatible, > in > > terms of size and impact it might best belong in a new major release. > > > > Best, > > Andrew > > > > On Fri, Oct 16, 2015 at 7:04 PM, Vinayakumar B < > > vinayakumarb.apa...@gmail.com> wrote: > > > >> Is anyone else also thinks that feature is ready to goto branch-2 as > well? > >> > >> Its > 2 weeks EC landed on trunk. IMo Looks Its quite stable since then > and > >> ready to go in branch-2. > >> > >> -Vinay > >> On Oct 6, 2015 12:51 AM, "Zhe Zhang" wrote: > >> > >>> Thanks Vinay for capturing the issue and Uma for offering the help. > >>> > >>> --- > >>> Zhe Zhang > >>> > >>> On Mon, Oct 5, 2015 at 12:19 PM, Gangumalla, Uma < > >> uma.ganguma...@intel.com > > >>> wrote: > >>> > Vinay, > > > I would merge them as part of HDFS-9182. > > Thanks, > Uma > > > > On 10/5/15, 12:48 AM, "Vinayakumar B" > wrote: > > > Hi Andrew, > > I see CHANGES.txt entries not yet merged from > >> CHANGES-HDFS-EC-7285.txt. > > > > Was this intentional? > > > > Regards, > > Vinay > > > > On Wed, Sep 30, 2015 at 9:15 PM, Andrew Wang < > >> andrew.w...@cloudera.com> > > wrote: > > > >> Branch has been merged to trunk, thanks again to everyone who worked > >>> on > >> the > >> feature! > >> > >> On Tue, Sep 29, 2015 at 10:44 PM, Zhe Zhang > >> wrote: > >> > >>> Thanks everyone who has participated in this discussion. > >>> > >>> With 7 +1's (5 binding and 2 non-binding), and no -1, this vote > >> has > >> passed. > >>> I will do a final 'git merge' with trunk and work with Andrew to > >>> merge > >> the > >>> branch to trunk. I'll update on this thread when the merge is > >> done. > >>> > >>> --- > >>> Zhe Zhang > >>> > >>> On Thu, Sep 24, 2015 at 11:08 PM, Liu, Yi A > >> wrote: > >>> > (Change it to binding.) > > +1 > I have been involved in the development and code review on the > >> feature > branch. It's a great feature and I think it's ready to merge it > >>> into > >>> trunk. > > Thanks all for the contribution. > > Regards, > Yi Liu > > > -Original Message- > From: Liu, Yi A > Sent: Friday, September 25, 2015 1:51 PM > To: hdfs-dev@hadoop.apache.org > Subject: RE: [VOTE] Merge HDFS-7285 (erasure coding) branch to > >>> trunk > > +1 (non-binding) > I have been involved in the development and code review on the > >> feature > branch. It's a great
Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]
Forking the thread. Started looking at the 2.8 list, various features’ status and arrived here. While I understand the pervasive nature of EC and a need for a significant bake-in, moving this to a 3.x release is not a good idea. We will surely get a 2.8 out this year and, as needed, I can even spend time getting started on a 2.9. OTOH, 3.x is long ways off, and given all the incompatibilities there, it would be a while before users can get their hands on EC if it were to be only on 3.x. At best, this may force sites that want EC to backport the entire EC feature to older releases, at worst this will be repeat the mess of 0.20 security release forks. If we think adding this to 2.8 (even if it switched off) is too much risk per our original plan, let’s move this to 2.9, there by leaving enough time for stability, integration testing and bake-in, and a realistic chance of having it end up on users’ clusters soonish. +Vinod > On Oct 19, 2015, at 1:44 PM, Andrew Wang wrote: > > I think our plan thus far has been to target this for 3.0. I'm okay with > putting it in branch-2 if we've given a hard look at compatibility, but > I'll note though that 2.8 is already looking like quite a large release, > and our release bandwidth has been focused on the 2.6 and 2.7 maintenance > releases. Adding another multi-hundred JIRAs to 2.8 might make it too > unwieldy to get out the door. If we bump EC past that, 3.0 might very well > be our next release vehicle. I do plan to revive the 3.0 schedule some time > next year. With EC and JDK8 in a good spot, the only big feature remaining > is classpath isolation. > > EC is also a pretty fundamental change to HDFS. Even if it's compatible, in > terms of size and impact it might best belong in a new major release. > > Best, > Andrew > > On Fri, Oct 16, 2015 at 7:04 PM, Vinayakumar B < > vinayakumarb.apa...@gmail.com> wrote: > >> Is anyone else also thinks that feature is ready to goto branch-2 as well? >> >> Its > 2 weeks EC landed on trunk. IMo Looks Its quite stable since then and >> ready to go in branch-2. >> >> -Vinay >> On Oct 6, 2015 12:51 AM, "Zhe Zhang" wrote: >> >>> Thanks Vinay for capturing the issue and Uma for offering the help. >>> >>> --- >>> Zhe Zhang >>> >>> On Mon, Oct 5, 2015 at 12:19 PM, Gangumalla, Uma < >> uma.ganguma...@intel.com >>> wrote: >>> Vinay, I would merge them as part of HDFS-9182. Thanks, Uma On 10/5/15, 12:48 AM, "Vinayakumar B" wrote: > Hi Andrew, > I see CHANGES.txt entries not yet merged from >> CHANGES-HDFS-EC-7285.txt. > > Was this intentional? > > Regards, > Vinay > > On Wed, Sep 30, 2015 at 9:15 PM, Andrew Wang < >> andrew.w...@cloudera.com> > wrote: > >> Branch has been merged to trunk, thanks again to everyone who worked >>> on >> the >> feature! >> >> On Tue, Sep 29, 2015 at 10:44 PM, Zhe Zhang >> wrote: >> >>> Thanks everyone who has participated in this discussion. >>> >>> With 7 +1's (5 binding and 2 non-binding), and no -1, this vote >> has >> passed. >>> I will do a final 'git merge' with trunk and work with Andrew to >>> merge >> the >>> branch to trunk. I'll update on this thread when the merge is >> done. >>> >>> --- >>> Zhe Zhang >>> >>> On Thu, Sep 24, 2015 at 11:08 PM, Liu, Yi A >> wrote: >>> (Change it to binding.) +1 I have been involved in the development and code review on the >> feature branch. It's a great feature and I think it's ready to merge it >>> into >>> trunk. Thanks all for the contribution. Regards, Yi Liu -Original Message- From: Liu, Yi A Sent: Friday, September 25, 2015 1:51 PM To: hdfs-dev@hadoop.apache.org Subject: RE: [VOTE] Merge HDFS-7285 (erasure coding) branch to >>> trunk +1 (non-binding) I have been involved in the development and code review on the >> feature branch. It's a great feature and I think it's ready to merge it >>> into >>> trunk. Thanks all for the contribution. Regards, Yi Liu -Original Message- From: Vinayakumar B [mailto:vinayakum...@apache.org] Sent: Friday, September 25, 2015 12:21 PM To: hdfs-dev@hadoop.apache.org Subject: Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to >>> trunk +1, I've been involved starting from design and development of >> ErasureCoding. I think phase 1 of this development is ready to be merged to >>> trunk. It had come a long way to the current state with significant >>> effort >> of many Contributors and Revi
