Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]

2016-01-07 Thread Zhe Zhang
on tasks in Phase I, we would label them with "erasure >> coding" and maybe also set the target version as "2.9" for the convenience? >> >> >> >> -Original Message- >> >> From: Jing Zhao [mailto:ji...@apache.org] >> >> Sent: Tu

Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]

2015-12-08 Thread Zhe Zhang
> coding" and maybe also set the target version as "2.9" for the convenience? > >> > >> -Original Message----- > >> From: Jing Zhao [mailto:ji...@apache.org] > >> Sent: Tuesday, November 03, 2015 8:04 AM > >> To: hdfs-dev@hadoop.apach

Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]

2015-12-08 Thread Vinod Kumar Vavilapalli
.@apache.org] >> Sent: Tuesday, November 03, 2015 8:04 AM >> To: hdfs-dev@hadoop.apache.org >> Subject: Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 >> (erasure coding) branch to trunk] >> >> +1 for the plan about Phase I & II. >>

Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]

2015-11-04 Thread Elliott Clark
I don't really see the difference between 2.9 with a ton of scary changes (EC is a lot more NN stuff than a usual release) and a 3.0. What's the downside of getting a major version. It relaxes the compat a little bit. It would allow some bake time before it's stable. Put another way. I'm upgradin

Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]

2015-11-04 Thread Andrew Purtell
Yes we can mostly likely help you. Please come over to dev@bigtop. > On Nov 4, 2015, at 12:40 PM, Andrew Wang wrote: > > We used to get help from Bigtop when it comes to integration testing. Do we > think that's possible for 2.8? > > On Wed, Nov 4, 2015 at 10:08 AM, Steve Loughran > wrote: >

Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]

2015-11-04 Thread Andrew Wang
We used to get help from Bigtop when it comes to integration testing. Do we think that's possible for 2.8? On Wed, Nov 4, 2015 at 10:08 AM, Steve Loughran wrote: > > > On 2 Nov 2015, at 23:11, Vinod Vavilapalli > wrote: > > > > Yes, I’ve already started looking at 2.8.0, that is exactly how I e

Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]

2015-11-04 Thread Steve Loughran
> On 2 Nov 2015, at 23:11, Vinod Vavilapalli wrote: > > Yes, I’ve already started looking at 2.8.0, that is exactly how I ended up > with this discussion on the state of EC. > > +Vinod > > > On Nov 2, 2015, at 3:02 PM, Haohui Mai > mailto:ricet...@gmail.com>> wrote: > > Is it a good time t

Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]

2015-11-04 Thread Zhe Zhang
t; be deployed in a production system. Phase II to develop non-striping > >>> EC for cold data would possibly > >> be > >>> started after that. We might consider to include only Phase I and > >>> leave Phase II for next release according to the rough release date. > >>&g

Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]

2015-11-03 Thread Vinod Vavilapalli
, 2015 8:04 AM > To: hdfs-dev@hadoop.apache.org > Subject: Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 > (erasure coding) branch to trunk] > > +1 for the plan about Phase I & II. > > BTW, maybe out of the scope of this thread, just want to mention we sho

RE: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]

2015-11-03 Thread Zheng, Kai
: hdfs-dev@hadoop.apache.org Subject: Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk] If we use an umbrella JIRA to categorize all the ongoing EC work, that will let us more easily change the target version later. For instance, if we decide to bump

Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]

2015-11-02 Thread Andrew Wang
; -Original Message- > From: Jing Zhao [mailto:ji...@apache.org] > Sent: Tuesday, November 03, 2015 8:04 AM > To: hdfs-dev@hadoop.apache.org > Subject: Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 > (erasure coding) branch to trunk] > > +1 for the pla

RE: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]

2015-11-02 Thread Zheng, Kai
..@apache.org] Sent: Tuesday, November 03, 2015 8:04 AM To: hdfs-dev@hadoop.apache.org Subject: Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk] +1 for the plan about Phase I & II. BTW, maybe out of the scope of this thread, just want to mention we

Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]

2015-11-02 Thread Jing Zhao
...@intel.com] > > Sent: Tuesday, November 03, 2015 5:41 AM > > To: hdfs-dev@hadoop.apache.org > > Subject: Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 > > (erasure coding) branch to trunk] > > > > +1 for EC to go into 2.9. Yes, 3.x woul

RE: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]

2015-11-02 Thread Vinayakumar B
ugh release date. > > Regards, > Kai > > -Original Message- > From: Gangumalla, Uma [mailto:uma.ganguma...@intel.com] > Sent: Tuesday, November 03, 2015 5:41 AM > To: hdfs-dev@hadoop.apache.org > Subject: Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-72

RE: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]

2015-11-02 Thread Zheng, Kai
nguma...@intel.com] Sent: Tuesday, November 03, 2015 5:41 AM To: hdfs-dev@hadoop.apache.org Subject: Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk] +1 for EC to go into 2.9. Yes, 3.x would be long way to go when we plan +to have 2.8 and 2.9 rel

Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]

2015-11-02 Thread Vinod Vavilapalli
Yes, I’ve already started looking at 2.8.0, that is exactly how I ended up with this discussion on the state of EC. +Vinod On Nov 2, 2015, at 3:02 PM, Haohui Mai mailto:ricet...@gmail.com>> wrote: Is it a good time to start the discussion on the issues of releasing 2.8?

Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]

2015-11-02 Thread Haohui Mai
+1 on putting EC on 2.9. Is it a good time to start the discussion on the issues of releasing 2.8? ~Haohui On Mon, Nov 2, 2015 at 1:40 PM, Gangumalla, Uma wrote: > +1 for EC to go into 2.9. Yes, 3.x would be long way to go when we plan to > have 2.8 and 2.9 releases. > > Regards, > Uma > > On 1

Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]

2015-11-02 Thread Gangumalla, Uma
+1 for EC to go into 2.9. Yes, 3.x would be long way to go when we plan to have 2.8 and 2.9 releases. Regards, Uma On 11/2/15, 11:49 AM, "Vinod Vavilapalli" wrote: >Forking the thread. Started looking at the 2.8 list, various features¹ >status and arrived here. > >While I understand the pervasi

Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]

2015-11-02 Thread Zhe Zhang
Thanks Vinod for the proposal and Andrew/Jing for the comments. 2.9 sounds a good plan. As Andrew pointed out, 2.8 is already quite big. And even when disabled, EC logic has been baked in NN pretty deeply. Do we have a tentative date or estimate for 2.9? --- Zhe Zhang On Mon, Nov 2, 2015 at 1:22

Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]

2015-11-02 Thread Jing Zhao
Thanks for the discussion, Vinod and Andrew. Backporting EC to 2.9 sounds good to me. On Mon, Nov 2, 2015 at 12:06 PM, Andrew Wang wrote: > Thanks for forking the thread Vinod, > > SGTM, though I really do recommend waiting for 2.9 given the current size > of 2.8. I'm not a fan of an "off by def

Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]

2015-11-02 Thread Andrew Wang
Thanks for forking the thread Vinod, SGTM, though I really do recommend waiting for 2.9 given the current size of 2.8. I'm not a fan of an "off by default" half-measure, since it doesn't change our compatibility requirements, and there's some major NN surgery that can't really be disabled. If we

Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]

2015-11-02 Thread Vinod Vavilapalli
Forking the thread. Started looking at the 2.8 list, various features’ status and arrived here. While I understand the pervasive nature of EC and a need for a significant bake-in, moving this to a 3.x release is not a good idea. We will surely get a 2.8 out this year and, as needed, I can even