Hi folks, Very optimistically, we're still on track for a 3.0 alpha this month. Here's a JIRA query for 3.0 and 2.8:
https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20MAPREDUCE%2C%20YARN)%20AND%20%22Target%20Version%2Fs%22%20in%20(3.0.0%2C%202.8.0)%20AND%20statusCategory%20not%20in%20(Complete)%20ORDER%20BY%20priority I think two of these are true alpha blockers: HADOOP-12892 and HADOOP-12893. I'm trying to help push both of those forward. For the rest, I think it's probably okay to delay until the next alpha, since we're planning a few alphas leading up to beta. That said, if you are the owner of a Blocker targeted at 3.0.0, I'd encourage reviving those patches. The earlier the better for incompatible changes. In all likelihood, this first release will slip into early May, but I'll be disappointed if we don't have an RC out before ApacheCon. Best, Andrew On Mon, Feb 22, 2016 at 3:19 PM, Colin P. McCabe <cmcc...@apache.org> wrote: > I think starting a 3.0 alpha soon would be a great idea. As some > other people commented, this would come with no compatibility > guarantees, so that we can iron out any issues. > > Colin > > On Mon, Feb 22, 2016 at 1:26 PM, Zhe Zhang <zhezh...@cloudera.com> wrote: > > Thanks Andrew for driving the effort! > > > > +1 (non-binding) on starting the 3.0 release process now with 3.0 as an > > alpha. > > > > I wanted to echo Andrew's point that backporting EC to branch-2 is a lot > of > > work. Considering that no concrete backporting plan has been proposed, it > > seems quite uncertain whether / when it can be released in 2.9. I think > we > > should rather concentrate our EC dev efforts to harden key features under > > the follow-on umbrella HDFS-8031 and make it solid for a 3.0 release. > > > > Sincerely, > > Zhe > > > > On Mon, Feb 22, 2016 at 9:25 AM Colin P. McCabe <cmcc...@apache.org> > wrote: > > > >> +1 for a release of 3.0. There are a lot of significant, > >> compatibility-breaking, but necessary changes in this release... we've > >> touched on some of them in this thread. > >> > >> +1 for a parallel release of 2.8 as well. I think we are pretty close > >> to this, barring a dozen or so blockers. > >> > >> best, > >> Colin > >> > >> On Mon, Feb 22, 2016 at 2:56 AM, Steve Loughran <ste...@hortonworks.com > > > >> wrote: > >> > > >> >> On 20 Feb 2016, at 15:34, Junping Du <j...@hortonworks.com> wrote: > >> >> > >> >> Shall we consolidate effort for 2.8.0 and 3.0.0? It doesn't sounds > >> reasonable to have two alpha releases to go in parallel. Is EC feature > the > >> main motivation of releasing hadoop 3 here? If so, I don't understand > why > >> this feature cannot land on 2.8.x or 2.9.x as an alpha feature. > >> > > >> > > >> > > >> >> If we release 3.0 in a month like plan proposed below, it means we > will > >> have 4 active releases going in parallel - two alpha releases (2.8 and > 3.0) > >> and two stable releases (2.6.x and 2.7.x). It brings a lot of > challenges in > >> issues tracking and patch committing, not even mention the tremendous > >> effort of release verification and voting. > >> >> I would like to propose to wait 2.8 release become stable (may be 2nd > >> release in 2.8 branch cause first release is alpha due to discussion in > >> another email thread), then we can move to 3.0 as the only alpha > release. > >> In the meantime, we can bring more significant features (like ATS v2, > etc.) > >> to trunk and consolidate stable releases in 2.6.x and 2.7.x. I believe > that > >> make life easier. :) > >> >> Thoughts? > >> >> > >> > > >> > 2.8.0 is relatively close to shipping. I say relatively as I'm doing > >> some work with ATS 1.5 downstream and I'd like to make sure all that > works. > >> There's also a large collection of S3 and swift patches needing > attention > >> from any reviewers with time and credentials. > >> > > >> > 3.x is going to take multiple iterations to stabilise, and with more > >> changes, more significant a rollout. I'd also like to do a complete > update > >> of all the dependencies before a final release, so we can have less > >> pressure to upgrade for a while, and get Sean's classloader patch in so > >> it's slightly less visible. > >> > > >> > That means 3.0 is going to be an alpha release, not final. > >> > > >> > one thing that could be shared is any build.xml automation of the > >> release process, to at least take away most of the manual steps in the > >> process, to have something more repeatable. > >> > > >> > -steve > >> > > >> > > >> >> Thanks, > >> >> > >> >> Junping > >> >> ________________________________________ > >> >> From: Yongjun Zhang <yzh...@cloudera.com> > >> >> Sent: Friday, February 19, 2016 8:05 PM > >> >> To: hdfs-...@hadoop.apache.org > >> >> Cc: common-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; > >> yarn-...@hadoop.apache.org > >> >> Subject: Re: Looking to a Hadoop 3 release > >> >> > >> >> Thanks Andrew for initiating the effort! > >> >> > >> >> +1 on pushing 3.x with extended alpha cycle, and continuing the more > >> stable > >> >> 2.x releases. > >> >> > >> >> --Yongjun > >> >> > >> >> On Thu, Feb 18, 2016 at 5:58 PM, Andrew Wang < > andrew.w...@cloudera.com> > >> >> wrote: > >> >> > >> >>> Hi Kai, > >> >>> > >> >>> Sure, I'm open to it. It's a new major release, so we're allowed to > >> make > >> >>> these kinds of big changes. The idea behind the extended alpha > cycle is > >> >>> that downstreams can give us feedback. This way if we do anything > too > >> >>> radical, we can address it in the next alpha and have downstreams > >> re-test. > >> >>> > >> >>> Best, > >> >>> Andrew > >> >>> > >> >>> On Thu, Feb 18, 2016 at 5:23 PM, Zheng, Kai <kai.zh...@intel.com> > >> wrote: > >> >>> > >> >>>> Thanks Andrew for driving this. Wonder if it's a good chance for > >> >>>> HADOOP-12579 (Deprecate and remove WriteableRPCEngine) to be in. > Note > >> >>> it's > >> >>>> not an incompatible change, but feel better to be done in the major > >> >>> release. > >> >>>> > >> >>>> Regards, > >> >>>> Kai > >> >>>> > >> >>>> -----Original Message----- > >> >>>> From: Andrew Wang [mailto:andrew.w...@cloudera.com] > >> >>>> Sent: Friday, February 19, 2016 7:04 AM > >> >>>> To: hdfs-...@hadoop.apache.org; Kihwal Lee <kih...@yahoo-inc.com> > >> >>>> Cc: mapreduce-dev@hadoop.apache.org; common-...@hadoop.apache.org; > >> >>>> yarn-...@hadoop.apache.org > >> >>>> Subject: Re: Looking to a Hadoop 3 release > >> >>>> > >> >>>> Hi Kihwal, > >> >>>> > >> >>>> I think there's still value in continuing the 2.x releases. 3.x > comes > >> >>> with > >> >>>> the incompatible bump to a JDK8 runtime, and also the fact that 3.x > >> won't > >> >>>> be beta or GA for some number of months. In the meanwhile, it'd be > >> good > >> >>> to > >> >>>> keep putting out regular, stable 2.x releases. > >> >>>> > >> >>>> Best, > >> >>>> Andrew > >> >>>> > >> >>>> > >> >>>> On Thu, Feb 18, 2016 at 2:50 PM, Kihwal Lee > >> <kih...@yahoo-inc.com.invalid > >> >>>> > >> >>>> wrote: > >> >>>> > >> >>>>> Moving Hadoop 3 forward sounds fine. If EC is one of the main > >> >>>>> motivations, are we getting rid of branch-2.8? > >> >>>>> > >> >>>>> Kihwal > >> >>>>> > >> >>>>> From: Andrew Wang <andrew.w...@cloudera.com> > >> >>>>> To: "common-...@hadoop.apache.org" <common-...@hadoop.apache.org> > >> >>>>> Cc: "yarn-...@hadoop.apache.org" <yarn-...@hadoop.apache.org>; " > >> >>>>> mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org > >; > >> >>>>> hdfs-dev <hdfs-...@hadoop.apache.org> > >> >>>>> Sent: Thursday, February 18, 2016 4:35 PM > >> >>>>> Subject: Re: Looking to a Hadoop 3 release > >> >>>>> > >> >>>>> Hi all, > >> >>>>> > >> >>>>> Reviving this thread. I've seen renewed interest in a trunk > release > >> >>>>> since HDFS erasure coding has not yet made it to branch-2. Along > with > >> >>>>> JDK8, the shell script rewrite, and many other improvements, I > think > >> >>>>> it's time to revisit Hadoop 3.0 release plans. > >> >>>>> > >> >>>>> My overall plan is still the same as in my original email: a > series > >> of > >> >>>>> regular alpha releases leading up to beta and GA. Alpha releases > make > >> >>>>> it easier for downstreams to integrate with our code, and making > them > >> >>>>> regular means features can be included when they are ready. > >> >>>>> > >> >>>>> I know there are some incompatible changes waiting in the wings > (i.e. > >> >>>>> HDFS-6984 making FileStatus a PB rather than Writable, some of > >> >>>>> HADOOP-9991 bumping dependency versions) that would be good to get > >> in. > >> >>>>> If you have changes like this, please set the target version to > 3.0.0 > >> >>>>> and mark them "Incompatible". We can use this JIRA query to track: > >> >>>>> > >> >>>>> > >> >>>>> > >> https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%2 > >> >>>>> > >> 0HDFS%2C%20YARN%2C%20MAPREDUCE)%20and%20%22Target%20Version%2Fs%22%20% > >> >>>>> > >> 3D%20%223.0.0%22%20and%20resolution%3D%22unresolved%22%20and%20%22Hado > >> >>>>> > op%20Flags%22%3D%22Incompatible%20change%22%20order%20by%20priority > >> >>>>> > >> >>>>> There's some release-related stuff that needs to be sorted out > >> >>>>> (namely, the new CHANGES.txt and release note generation from > Yetus), > >> >>>>> but I'd tentatively like to roll the first alpha a month out, so > >> third > >> >>>>> week of March. > >> >>>>> > >> >>>>> Best, > >> >>>>> Andrew > >> >>>>> > >> >>>>> On Mon, Mar 9, 2015 at 7:23 PM, Raymie Stata < > rst...@altiscale.com> > >> >>>> wrote: > >> >>>>> > >> >>>>>> Avoiding the use of JDK8 language features (and, presumably, > APIs) > >> >>>>>> means you've abandoned #1, i.e., you haven't (really) bumped the > JDK > >> >>>>>> source version to JDK8. > >> >>>>>> > >> >>>>>> Also, note that releasing from trunk is a way of achieving #3, > it's > >> >>>>>> not a way of abandoning it. > >> >>>>>> > >> >>>>>> > >> >>>>>> > >> >>>>>> On Mon, Mar 9, 2015 at 7:10 PM, Andrew Wang > >> >>>>>> <andrew.w...@cloudera.com> > >> >>>>>> wrote: > >> >>>>>>> Hi Raymie, > >> >>>>>>> > >> >>>>>>> Konst proposed just releasing off of trunk rather than cutting a > >> >>>>>> branch-2, > >> >>>>>>> and there was general agreement there. So, consider #3 > abandoned. > >> >>>>>>> 1&2 > >> >>>>> can > >> >>>>>>> be achieved at the same time, we just need to avoid using JDK8 > >> >>>>>>> language features in trunk so things can be backported. > >> >>>>>>> > >> >>>>>>> Best, > >> >>>>>>> Andrew > >> >>>>>>> > >> >>>>>>> On Mon, Mar 9, 2015 at 7:01 PM, Raymie Stata > >> >>>>>>> <rst...@altiscale.com> > >> >>>>>> wrote: > >> >>>>>>> > >> >>>>>>>> In this (and the related threads), I see the following three > >> >>>>>> requirements: > >> >>>>>>>> > >> >>>>>>>> 1. "Bump the source JDK version to JDK8" (ie, drop JDK7 > support). > >> >>>>>>>> > >> >>>>>>>> 2. "We'll still be releasing 2.x releases for a while, with > >> >>>>>>>> similar feature sets as 3.x." > >> >>>>>>>> > >> >>>>>>>> 3. Avoid the "risk of split-brain behavior" by "minimize > >> >>>>>>>> backporting headaches. Pulling trunk > branch-2 > branch-2.x is > >> >>>> already tedious. > >> >>>>>>>> Adding a branch-3, branch-3.x would be obnoxious." > >> >>>>>>>> > >> >>>>>>>> These three cannot be achieved at the same time. Which do we > >> >>>> abandon? > >> >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> On Mon, Mar 9, 2015 at 12:45 PM, sanjay Radia > >> >>>>>>>> <sanjayo...@gmail.com> > >> >>>>>>>> wrote: > >> >>>>>>>>> > >> >>>>>>>>>> On Mar 5, 2015, at 3:21 PM, Siddharth Seth <ss...@apache.org > > > >> >>>>> wrote: > >> >>>>>>>>>> > >> >>>>>>>>>> 2) Simplification of configs - potentially separating client > >> >>>>>>>>>> side > >> >>>>>>>> configs > >> >>>>>>>>>> and those used by daemons. This is another source of > perpetual > >> >>>>>> confusion > >> >>>>>>>>>> for users. > >> >>>>>>>>> + 1 on this. > >> >>>>>>>>> > >> >>>>>>>>> sanjay > >> >>>>>>>> > >> >>>>>> > >> >>>>> > >> >>>>> > >> >>>>> > >> >>>> > >> >>> > >> >> > >> > > >> >
