RE: Apache Hadoop 2.8.3 Release Plan

[Zheng, Kai]

Thanks Andrew for the comments.

Yes, if we're "strictly" following the "maintenance release" practice, that'd 
be great and it's never my intent to overload it and cause mess.

>> If we're struggling with being able to deliver new features in a safe and 
>> timely fashion, let's try to address that...

This is interesting. Do you aware any means to do that? Thanks!

Regards,
Kai

-Original Message-
From: Andrew Wang [mailto:andrew.w...@cloudera.com] 
Sent: Tuesday, November 21, 2017 2:22 PM
To: Zheng, Kai <kai.zh...@intel.com>
Cc: Junping Du <j...@hortonworks.com>; common-...@hadoop.apache.org; 
hdfs-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; 
yarn-...@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.3 Release Plan

I'm against including new features in maintenance releases, since they're meant 
to be bug-fix only.

If we're struggling with being able to deliver new features in a safe and 
timely fashion, let's try to address that, not overload the meaning of 
"maintenance release".

Best,
Andrew

On Mon, Nov 20, 2017 at 5:20 PM, Zheng, Kai <kai.zh...@intel.com> wrote:

> Hi Junping,
>
> Thank you for making 2.8.2 happen and now planning the 2.8.3 release.
>
> I have an ask, is it convenient to include the back port work for OSS 
> connector module? We have some Hadoop users that wish to have it by 
> default for convenience, though in the past they used it by back 
> porting themselves. I have raised this and got thoughts from Chris and 
> Steve. Looks like this is more wanted for 2.9 but I wanted to ask 
> again here for broad feedback and thoughts by this chance. The back 
> port patch is available for
> 2.8 and the one for branch-2 was already in. IMO, 2.8.x is promising 
> as we can see some shift from 2.7.x, hence it's worth more important 
> features and efforts. How would you think? Thanks!
>
> https://issues.apache.org/jira/browse/HADOOP-14964
>
> Regards,
> Kai
>
> -Original Message-
> From: Junping Du [mailto:j...@hortonworks.com]
> Sent: Tuesday, November 14, 2017 9:02 AM
> To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
> mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
> Subject: Apache Hadoop 2.8.3 Release Plan
>
> Hi,
> We have several important fixes get landed on branch-2.8 and I 
> would like to cut off branch-2.8.3 now to start 2.8.3 release work.
> So far, I don't see any pending blockers on 2.8.3, so my current 
> plan is to cut off 1st RC of 2.8.3 in next several days:
>  -  For all coming commits to land on branch-2.8, please mark 
> the fix version as 2.8.4.
>  -  If there is a really important fix for 2.8.3 and getting 
> closed, please notify me ahead before landing it on branch-2.8.3.
> Please let me know if you have any thoughts or comments on the plan.
>
> Thanks,
>
> Junping
> 
> From: dujunp...@gmail.com <dujunp...@gmail.com> on behalf of 俊平堵 < 
> junping...@apache.org>
> Sent: Friday, October 27, 2017 3:33 PM
> To: gene...@hadoop.apache.org
> Subject: [ANNOUNCE] Apache Hadoop 2.8.2 Release.
>
> Hi all,
>
> It gives me great pleasure to announce that the Apache Hadoop 
> community has voted to release Apache Hadoop 2.8.2, which is now 
> available for download from Apache mirrors[1]. For download 
> instructions please refer to the Apache Hadoop Release page [2].
>
> Apache Hadoop 2.8.2 is the first GA release of Apache Hadoop 2.8 line 
> and our newest stable release for entire Apache Hadoop project. For 
> major changes incuded in Hadoop 2.8 line, please refer Hadoop 2.8.2 main 
> page[3].
>
> This release has 315 resolved issues since previous 2.8.1 release with 
> following
> breakdown:
>- 91 in Hadoop Common
>- 99 in HDFS
>- 105 in YARN
>- 20 in MapReduce
> Please read the log of CHANGES[4] and RELEASENOTES[5] for more details.
>
> The release news is posted on the Hadoop website too, you can go to 
> the downloads section directly [6].
>
> Thank you all for contributing to the Apache Hadoop release!
>
>
> Cheers,
>
> Junping
>
>
> [1] http://www.apache.org/dyn/closer.cgi/hadoop/common
>
> [2] http://hadoop.apache.org/releases.html
>
> [3] http://hadoop.apache.org/docs/r2.8.2/index.html
>
> [4]
> http://hadoop.apache.org/docs/r2.8.2/hadoop-project-dist/
> hadoop-common/release/2.8.2/CHANGES.2.8.2.html
>
> [5]
> http://hadoop.apache.org/docs/r2.8.2/hadoop-project-dist/
> hadoop-common/release/2.8.2/RELEASENOTES.2.8.2.html
>
> [6] http://hadoop.apache.org/releases.html#Download
>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>

RE: Apache Hadoop 2.8.3 Release Plan

[Zheng, Kai]

Hi Junping,

Thank you for making 2.8.2 happen and now planning the 2.8.3 release. 

I have an ask, is it convenient to include the back port work for OSS connector 
module? We have some Hadoop users that wish to have it by default for 
convenience, though in the past they used it by back porting themselves. I have 
raised this and got thoughts from Chris and Steve. Looks like this is more 
wanted for 2.9 but I wanted to ask again here for broad feedback and thoughts 
by this chance. The back port patch is available for 2.8 and the one for 
branch-2 was already in. IMO, 2.8.x is promising as we can see some shift from 
2.7.x, hence it's worth more important features and efforts. How would you 
think? Thanks!

https://issues.apache.org/jira/browse/HADOOP-14964

Regards,
Kai

-Original Message-
From: Junping Du [mailto:j...@hortonworks.com] 
Sent: Tuesday, November 14, 2017 9:02 AM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Apache Hadoop 2.8.3 Release Plan

Hi,
We have several important fixes get landed on branch-2.8 and I would like 
to cut off branch-2.8.3 now to start 2.8.3 release work. 
So far, I don't see any pending blockers on 2.8.3, so my current plan is to 
cut off 1st RC of 2.8.3 in next several days: 
 -  For all coming commits to land on branch-2.8, please mark the fix 
version as 2.8.4.
 -  If there is a really important fix for 2.8.3 and getting closed, 
please notify me ahead before landing it on branch-2.8.3.
Please let me know if you have any thoughts or comments on the plan.

Thanks,

Junping

From: dujunp...@gmail.com  on behalf of 俊平堵 

Sent: Friday, October 27, 2017 3:33 PM
To: gene...@hadoop.apache.org
Subject: [ANNOUNCE] Apache Hadoop 2.8.2 Release.

Hi all,

It gives me great pleasure to announce that the Apache Hadoop community has 
voted to release Apache Hadoop 2.8.2, which is now available for download from 
Apache mirrors[1]. For download instructions please refer to the Apache Hadoop 
Release page [2].

Apache Hadoop 2.8.2 is the first GA release of Apache Hadoop 2.8 line and our 
newest stable release for entire Apache Hadoop project. For major changes 
incuded in Hadoop 2.8 line, please refer Hadoop 2.8.2 main page[3].

This release has 315 resolved issues since previous 2.8.1 release with following
breakdown:
   - 91 in Hadoop Common
   - 99 in HDFS
   - 105 in YARN
   - 20 in MapReduce
Please read the log of CHANGES[4] and RELEASENOTES[5] for more details.

The release news is posted on the Hadoop website too, you can go to the 
downloads section directly [6].

Thank you all for contributing to the Apache Hadoop release!


Cheers,

Junping


[1] http://www.apache.org/dyn/closer.cgi/hadoop/common

[2] http://hadoop.apache.org/releases.html

[3] http://hadoop.apache.org/docs/r2.8.2/index.html

[4]
http://hadoop.apache.org/docs/r2.8.2/hadoop-project-dist/hadoop-common/release/2.8.2/CHANGES.2.8.2.html

[5]
http://hadoop.apache.org/docs/r2.8.2/hadoop-project-dist/hadoop-common/release/2.8.2/RELEASENOTES.2.8.2.html

[6] http://hadoop.apache.org/releases.html#Download


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

RE: [VOTE] Merge yarn-native-services branch into trunk

[Zheng, Kai]

Cool to have this feature! Thanks Jian and all.

Regards,
Kai

-Original Message-
From: Vinod Kumar Vavilapalli [mailto:vino...@apache.org] 
Sent: Tuesday, November 07, 2017 7:20 AM
To: Jian He 
Cc: yarn-...@hadoop.apache.org; common-...@hadoop.apache.org; Hdfs-dev 
; mapreduce-dev@hadoop.apache.org
Subject: Re: [VOTE] Merge yarn-native-services branch into trunk

Congratulations to all the contributors involved, this is a great step forward!

+Vinod

> On Nov 6, 2017, at 2:40 PM, Jian He  wrote:
> 
> Okay, I just merged the branch to trunk (108 commits in total !) 
> Again, thanks for all who contributed to this feature!
> 
> Jian
> 
> On Nov 6, 2017, at 1:26 PM, Jian He 
> > wrote:
> 
> Here’s +1 from myself.
> The vote passes with 7 (+1) bindings and 2 (+1) non-bindings.
> 
> Thanks for all who voted. I’ll merge to trunk by the end of today.
> 
> Jian
> 
> On Nov 6, 2017, at 8:38 AM, Billie Rinaldi 
> > wrote:
> 
> +1 (binding)
> 
> On Mon, Oct 30, 2017 at 1:06 PM, Jian He 
> > wrote:
> Hi All,
> 
> I would like to restart the vote for merging yarn-native-services to trunk.
> Since last vote, we have been working on several issues in documentation, 
> DNS, CLI modifications etc. We believe now the feature is in a much better 
> shape.
> 
> Some back ground:
> At a high level, the following are the key feautres implemented.
> - YARN-5079[1]. A native YARN framework (ApplicationMaster) to orchestrate 
> existing services to YARN either docker or non-docker based.
> - YARN-4793[2]. A Rest API service embeded in RM (optional)  for user 
> to deploy a service via a simple JSON spec
> - YARN-4757[3]. Extending today's service registry with a simple DNS 
> service to enable users to discover services deployed on YARN via 
> standard DNS lookup
> - YARN-6419[4]. UI support for native-services on the new YARN UI All 
> these new services are optional and are sitting outside of the existing 
> system, and have no impact on existing system if disabled.
> 
> Special thanks to a team of folks who worked hard towards this: Billie 
> Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith Sharma K 
> S, Sunil G, Akhil PB, Eric Yang. This effort could not be possible without 
> their ideas and hard work.
> Also thanks Allen for some review and verifications.
> 
> Thanks,
> Jian
> 
> [1] https://issues.apache.org/jira/browse/YARN-5079
> [2] https://issues.apache.org/jira/browse/YARN-4793
> [3] https://issues.apache.org/jira/browse/YARN-4757
> [4] https://issues.apache.org/jira/browse/YARN-6419
> 
> 
> 


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

RE: [DISCUSS] A final minor release off branch-2?

[Zheng, Kai]

Thanks Vinod.

>> Of the top of my head, one of the biggest areas is application 
>> compatibility. When folks move from 2.x to 3.x, are their apps binary 
>> compatible? Source compatible? Or need changes?
I thought these are good concerns from overall perspective. On the other hand, 
I've discussed with quite a few 3.0 potential users, it looks like most of them 
are interested in the erasure coding feature and a major scenario for that is 
to back up their large volume of data to save storage cost. They might run 
analytics workload using Hive, Spark, Impala and Kylin on the new cluster based 
on the version, but it's not a must at the first time. They understand there 
might be some gaps so they'd migrate their workloads incrementally. For the 
major analytics workload, we've performed lots of benchmark and integration 
tests as well as other sides I believe, we did find some issues but they should 
be fixed in downstream projects. I thought the release of GA will accelerate 
the progress and expose the issues if any. We couldn't wait for it being 
matured. There isn't perfectness.

>> The main goal of the bridging release is to ease transition on stuff that is 
>> guaranteed to be broken.
This sounds a good consideration. I'm thinking if I'm a Hadoop user, for 
example, I'm using 2.7.4 or 2.8.2 or whatever 2.x version, would I first 
upgrade to this bridging release then use the bridge support to upgrade to 3.x 
version? I'm not sure. On the other hand, I might tend to look for some guides 
or supports in 3.x docs about how to upgrade from 2.7 to 3.x. 

Frankly speaking, working on some bridging release not targeting any feature 
isn't so attractive to me as a contributor. Overall, the final minor release 
off branch-2 is good, we should also give 3.x more time to evolve and mature, 
therefore it looks to me we would have to work on two release lines meanwhile 
for some time. I'd like option C), and suggest we focus on the recent releases.

Just some thoughts.

Regards,
Kai

-Original Message-
From: Vinod Kumar Vavilapalli [mailto:vino...@apache.org] 
Sent: Tuesday, November 07, 2017 9:43 AM
To: Andrew Wang 
Cc: Arun Suresh ; common-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org; Hdfs-dev ; 
mapreduce-dev@hadoop.apache.org
Subject: Re: [DISCUSS] A final minor release off branch-2?

The main goal of the bridging release is to ease transition on stuff that is 
guaranteed to be broken.

Of the top of my head, one of the biggest areas is application compatibility. 
When folks move from 2.x to 3.x, are their apps binary compatible? Source 
compatible? Or need changes?

In 1.x -> 2.x upgrade, we did a bunch of work to atleast make old apps be 
source compatible. This means relooking at the API compatibility in 3.x and 
their impact of migrating applications. We will have to revist and un-deprecate 
old APIs, un-delete old APIs and write documentation on how apps can be 
migrated.

Most of this work will be in 3.x line. The bridging release on the other hand 
will have deprecation for APIs that cannot be undeleted. This may be already 
have been done in many places. But we need to make sure and fill gaps if any.

Other areas that I can recall from the old days
 - Config migration: Many configs are deprecated or deleted. We need 
documentation to help folks to move. We also need deprecations in the bridging 
release for configs that cannot be undeleted.
 - You mentioned rolling-upgrades: It will be good to exactly outline the type 
of testing. For e.g., the rolling-upgrades orchestration order has direct 
implication on the testing done.
 - Story for downgrades?
 - Copying data between 2.x clusters and 3.x clusters: Does this work already? 
Is it broken anywhere that we cannot fix? Do we need bridging features for this 
work?

+Vinod

> On Nov 6, 2017, at 12:49 PM, Andrew Wang  wrote:
> 
> What are the known gaps that need bridging between 2.x and 3.x?
> 
> From an HDFS perspective, we've tested wire compat, rolling upgrade, 
> and rollback.
> 
> From a YARN perspective, we've tested wire compat and rolling upgrade. 
> Arun just mentioned an NM rollback issue that I'm not familiar with.
> 
> Anything else? External to this discussion, these should be documented 
> as known issues for 3.0.
> 
> Best.
> Andrew
> 
> On Sun, Nov 5, 2017 at 1:46 PM, Arun Suresh  wrote:
> 
>> Thanks for starting this discussion VInod.
>> 
>> I agree (C) is a bad idea.
>> I would prefer (A) given that ATM, branch-2 is still very close to
>> branch-2.9 - and it is a good time to make a collective decision to 
>> lock down commits to branch-2.
>> 
>> I think we should also clearly define what the 'bridging' release 
>> should be.
>> I assume it means the following:
>> * Any 2.x user wanting to move to 3.x must first upgrade to the 
>> bridging release first and then upgrade to the 3.x release.
>> * With 

RE: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0

[Zheng, Kai]

Thanks Sammi.

My non-binding +1 to make the release candidate.

Regards,
Kai

-Original Message-
From: Chen, Sammi 
Sent: Friday, September 02, 2016 4:59 PM
To: Zheng, Kai <kai.zh...@intel.com>; Andrew Wang <andrew.w...@cloudera.com>; 
Arun Suresh <asur...@apache.org>
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; Chen, Sammi 
<sammi.c...@intel.com>
Subject: RE: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0

+1 (non-binding).

Thanks for driving this Andrew!

* Download and built from source.
* Setup a 10 node cluster (1 name node + 9 data nodes)
* Verified normal HDFS file put/get operation with 3x replication
* With 2 data nodes failure, verified HDFS file put/get operation with 3x 
replication, file integrity is OK
* Enable Erasure Code policy "RS-DEFAULT-6-3-64k", verified HDFS file put/get 
operation
* Enable Erasure Code policy "RS-DEFAULT-6-3-64k", with 3 data nodes failure, 
verified HDFS file put/get operation, file integrity is OK

Cheers
-Sammi

-Original Message-
From: Zheng, Kai
Sent: Friday, September 02, 2016 3:25 PM
To: Chen, Sammi
Subject: FW: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0

Hi Sammi,

Could you help provide our feedback? I know you did lots of tests. Thanks!

Regards,
Kai

-Original Message-
From: Arun Suresh [mailto:asur...@apache.org]
Sent: Friday, September 02, 2016 11:33 AM
To: Andrew Wang <andrew.w...@cloudera.com>
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0

+1 (binding).

Thanks for driving this Andrew..

* Download and built from source.
* Setup a 5 mode cluster.
* Verified that MR works with opportunistic containers
* Verified that the AMRMClient supports 'allocationRequestId'

Cheers
-Arun

On Thu, Sep 1, 2016 at 4:31 PM, Aaron Fabbri <fab...@cloudera.com> wrote:

> +1, non-binding.
>
> I built everything on OS X and ran the s3a contract tests successfully:
>
> mvn test -Dtest=org.apache.hadoop.fs.contract.s3a.\*
>
> ...
>
> Results :
>
>
> Tests run: 78, Failures: 0, Errors: 0, Skipped: 1
>
>
> [INFO]
> --
> --
>
> [INFO] BUILD SUCCESS
>
> [INFO]
> --
> --
>
> On Thu, Sep 1, 2016 at 3:39 PM, Andrew Wang <andrew.w...@cloudera.com>
> wrote:
>
> > Good point Allen, I forgot about `hadoop version`. Since it's 
> > populated
> by
> > a version-info.properties file, people can always cat that file.
> >
> > On Thu, Sep 1, 2016 at 3:21 PM, Allen Wittenauer <
> a...@effectivemachines.com
> > >
> > wrote:
> >
> > >
> > > > On Sep 1, 2016, at 3:18 PM, Allen Wittenauer <
> a...@effectivemachines.com
> > >
> > > wrote:
> > > >
> > > >
> > > >> On Sep 1, 2016, at 2:57 PM, Andrew Wang 
> > > >> <andrew.w...@cloudera.com>
> > > wrote:
> > > >>
> > > >> Steve requested a git hash for this release. This led us into a
> brief
> > > >> discussion of our use of git tags, wherein we realized that 
> > > >> although release tags are immutable (start with "rel/"), RC tags are 
> > > >> not.
> This
> > is
> > > >> based on the HowToRelease instructions.
> > > >
> > > >   We should probably embed the git hash in one of the files 
> > > > that
> > > gets gpg signed.  That's an easy change to create-release.
> > >
> > >
> > > (Well, one more easily accessible than 'hadoop version')
> >
>

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

RE: [DISCUSS] The order of classpath isolation work and updating/shading dependencies on trunk

[Zheng, Kai]

For the leveldb thing, wouldn't we have an alternative option in Java for the 
platforms where leveldb isn't supported yet due to whatever reasons. IMO, 
native library would be best to be used for optimization and production for 
performance. For development and pure Java platform, by default pure Java 
approach should still be provided and used. That is to say, if no Hadoop native 
is used, all the functionalities should still work and not break. 

HDFS erasure coding goes in the way. For that, we spent much effort in 
developing an ISA-L compatible erasure coder in pure Java that's used by 
default, though for performance the ISA-L native one is recommended in 
production deployment.

Regards,
Kai

-Original Message-
From: Allen Wittenauer [mailto:a...@effectivemachines.com] 
Sent: Saturday, July 23, 2016 8:16 AM
To: Sangjin Lee 
Cc: Sean Busbey ; common-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org
Subject: Re: [DISCUSS] The order of classpath isolation work and 
updating/shading dependencies on trunk


But if I don't use ApplicationClassLoader, my java app is basically screwed 
then, right?

Also:  right now, the non-Linux and/or non-x86 platforms have to supply their 
own leveldbjni jar (or at least the C level library?) in order to make YARN 
even functional.  How is that going to work with the class path manipulation?


> On Jul 22, 2016, at 9:57 AM, Sangjin Lee  wrote:
> 
> The work on HADOOP-13070 and the ApplicationClassLoader are generic and go 
> beyond YARN. It can be used in any JVM that uses hadoop. The current use 
> cases are MR containers, hadoop's RunJar (as in "hadoop jar"), and the YARN 
> node manager auxiliary services. I'm not sure if that's what you were asking, 
> but I hope it helps.
> 
> Regards,
> Sangjin
> 
> On Fri, Jul 22, 2016 at 9:16 AM, Sean Busbey  wrote:
> My work on HADOOP-11804 *only* helps processes that sit outside of 
> YARN. :)
> 
> On Fri, Jul 22, 2016 at 10:48 AM, Allen Wittenauer 
>  wrote:
> >
> > Does any of this work actually help processes that sit outside of YARN?
> >
> >> On Jul 21, 2016, at 12:29 PM, Sean Busbey  wrote:
> >>
> >> thanks for bringing this up! big +1 on upgrading dependencies for 3.0.
> >>
> >> I have an updated patch for HADOOP-11804 ready to post this week. 
> >> I've been updating HBase's master branch to try to make use of it, 
> >> but could use some other reviews.
> >>
> >> On Thu, Jul 21, 2016 at 4:30 AM, Tsuyoshi Ozawa  wrote:
> >>> Hi developers,
> >>>
> >>> I'd like to discuss how to make an advance towards dependency 
> >>> management in Apache Hadoop trunk code since there has been lots 
> >>> work about updating dependencies in parallel. Summarizing recent 
> >>> works and activities as follows:
> >>>
> >>> 0) Currently, we have merged minimum update dependencies for 
> >>> making Hadoop JDK-8 compatible(compilable and runnable on JDK-8).
> >>> 1) After that, some people suggest that we should update the other 
> >>> dependencies on trunk(e.g. protobuf, netty, jackthon etc.).
> >>> 2) In parallel, Sangjin and Sean are working on classpath isolation:
> >>> HADOOP-13070, HADOOP-11804 and HADOOP-11656.
> >>>
> >>> Main problems we try to solve in the activities above is as follows:
> >>>
> >>> * 1) tries to solve dependency hell between user-level jar and 
> >>> system(Hadoop)-level jar.
> >>> * 2) tries to solve updating old libraries.
> >>>
> >>> IIUC, 1) and 2) looks not related, but it's related in fact. 2) 
> >>> tries to separate class loader between client-side dependencies 
> >>> and server-side dependencies in Hadoop, so we can the change 
> >>> policy of updating libraries after doing 2). We can also decide 
> >>> which libraries can be shaded after 2).
> >>>
> >>> Hence, IMHO, a straight way we should go to is doing 2 at first.
> >>> After that, we can update both client-side and server-side 
> >>> dependencies based on new policy(maybe we should discuss what kind 
> >>> of incompatibility is acceptable, and the others are not).
> >>>
> >>> Thoughts?
> >>>
> >>> Thanks,
> >>> - Tsuyoshi
> >>>
> >>> --
> >>> --- To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> >>> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> >>>
> >>
> >>
> >>
> >> --
> >> busbey
> >>
> >> ---
> >> -- To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> >> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >>
> >
> >
> > 
> > - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> >
> 
> 
> 
> --

RE: Setting JIRA fix versions for 3.0.0 releases

[Zheng, Kai]

My humble feeling is almost the same regarding the urgent need of a 3.0 alpha 
release.

Considering EC, shell-script rewriting and etc. are significant changes and 
there are interested users that want to evaluate EC storage method, an alpha 
3.0 release will definitely help a lot allowing users to try the new features 
and then expose critical bugs or gaps. This may take quite some time, and 
should be very important to build confidence preparing for a solid 3.0 release. 
I understand Vinod's concern and the need of lining up the release efforts, so 
if it's to work on multiple 2.x releases it should be avoided. Mentioning 3.0 
alpha, it's different and the best would be to allow parallel going to speed up 
EC and the like, meanwhile any 2.x release won't be in a hurry pushed by 3.0 
release. 

Thanks for any consideration.

Regards,
Kai

-Original Message-
From: Andrew Wang [mailto:andrew.w...@cloudera.com] 
Sent: Friday, July 22, 2016 3:33 AM
To: Vinod Kumar Vavilapalli 
Cc: common-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; 
yarn-...@hadoop.apache.org
Subject: Re: Setting JIRA fix versions for 3.0.0 releases

I really, really want a 3.0.0-alpha1 ASAP, since it's basically impossible for 
downstreams to test incompat changes and new features without a release 
artifact. I've been doing test builds, and branch-3.0.0-alpha1 is ready for an 
RC besides possibly this fix version issue.

I'm not too worried about splitting community bandwidth, for the following
reasons:

* 3.0.0-alpha1 is very explicitly an alpha, which means no quality or 
compatibility guarantees. It needs less vetting than a 2.x release.
* Given that 3.0.0 is still in alpha, there aren't many true showstopper bugs. 
Most blockers I see are also apply to both 2.x as well as 3.0.0.
* Community bandwidth isn't zero-sum. This particularly applies to people 
working on features that are only present in trunk, like EC, shell script 
rewrite, etc.

Longer-term, I assume the 2.x line is not ending with 2.8. So we'd still have 
the issue of things committed for 2.9.0 that will be appearing for the first 
time in 3.0.0-alpha1. Assuming a script exists to fix up 2.9 JIRAs, it's only 
incrementally more work to also fix up 2.8 and other unreleased versions too.

Best,
Andrew

On Thu, Jul 21, 2016 at 11:53 AM, Vinod Kumar Vavilapalli < vino...@apache.org> 
wrote:

> The L & N fixes just went out, I’m working to push out 2.7.3 - running 
> into a Nexus issue. Once that goes out, I’ll immediately do a 2.8.0.
>
> Like I requested before in one of the 3.x threads, can we just line up
> 3.0.0-alpha1 right behind 2.8.0?
>
> That simplifies most of this confusion, we can avoid splitting the 
> bandwidth from the community on fixing blockers / vetting these 
> concurrent releases. Waiting a little more for 3.0.0 alpha to avoid 
> most of this is worth it, IMO.
>
> Thanks
> +Vinod
>
> > On Jul 21, 2016, at 11:34 AM, Andrew Wang 
> wrote:
> >
> > Hi all,
> >
> > Since we're planning to spin releases off of both branch-2 and 
> > trunk, the changelog for 3.0.0-alpha1 based on JIRA information 
> > isn't accurate. This is because historically, we've only set 2.x fix 
> > versions, and 2.8.0 and
> > 2.9.0 and etc have not been released. So there's a whole bunch of 
> > changes which will show up for the first time in 3.0.0-alpha1.
> >
> > I think I can write a script to (carefully) add 3.0.0-alpha1 to 
> > these JIRAs, but I figured I'd give a heads up here in case anyone 
> > felt differently. I can also update the HowToCommit page to match.
> >
> > Thanks,
> > Andrew
>
>

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

RE: Looking to a Hadoop 3 release

[Zheng, Kai]

Thanks Andrew for driving this. Wonder if it's a good chance for HADOOP-12579 
(Deprecate and remove WriteableRPCEngine) to be in. Note it's not an 
incompatible change, but feel better to be done in the major release.

Regards,
Kai

-Original Message-
From: Andrew Wang [mailto:andrew.w...@cloudera.com] 
Sent: Friday, February 19, 2016 7:04 AM
To: hdfs-...@hadoop.apache.org; Kihwal Lee 
Cc: mapreduce-dev@hadoop.apache.org; common-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org
Subject: Re: Looking to a Hadoop 3 release

Hi Kihwal,

I think there's still value in continuing the 2.x releases. 3.x comes with the 
incompatible bump to a JDK8 runtime, and also the fact that 3.x won't be beta 
or GA for some number of months. In the meanwhile, it'd be good to keep putting 
out regular, stable 2.x releases.

Best,
Andrew


On Thu, Feb 18, 2016 at 2:50 PM, Kihwal Lee 
wrote:

> Moving Hadoop 3 forward sounds fine. If EC is one of the main 
> motivations, are we getting rid of branch-2.8?
>
> Kihwal
>
>   From: Andrew Wang 
>  To: "common-...@hadoop.apache.org" 
> Cc: "yarn-...@hadoop.apache.org" ; "
> mapreduce-dev@hadoop.apache.org" ;
> hdfs-dev 
>  Sent: Thursday, February 18, 2016 4:35 PM
>  Subject: Re: Looking to a Hadoop 3 release
>
> Hi all,
>
> Reviving this thread. I've seen renewed interest in a trunk release 
> since HDFS erasure coding has not yet made it to branch-2. Along with 
> JDK8, the shell script rewrite, and many other improvements, I think 
> it's time to revisit Hadoop 3.0 release plans.
>
> My overall plan is still the same as in my original email: a series of 
> regular alpha releases leading up to beta and GA. Alpha releases make 
> it easier for downstreams to integrate with our code, and making them 
> regular means features can be included when they are ready.
>
> I know there are some incompatible changes waiting in the wings (i.e. 
> HDFS-6984 making FileStatus a PB rather than Writable, some of
> HADOOP-9991 bumping dependency versions) that would be good to get in. 
> If you have changes like this, please set the target version to 3.0.0 
> and mark them "Incompatible". We can use this JIRA query to track:
>
>
> https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%2
> 0HDFS%2C%20YARN%2C%20MAPREDUCE)%20and%20%22Target%20Version%2Fs%22%20%
> 3D%20%223.0.0%22%20and%20resolution%3D%22unresolved%22%20and%20%22Hado
> op%20Flags%22%3D%22Incompatible%20change%22%20order%20by%20priority
>
> There's some release-related stuff that needs to be sorted out 
> (namely, the new CHANGES.txt and release note generation from Yetus), 
> but I'd tentatively like to roll the first alpha a month out, so third 
> week of March.
>
> Best,
> Andrew
>
> On Mon, Mar 9, 2015 at 7:23 PM, Raymie Stata  wrote:
>
> > Avoiding the use of JDK8 language features (and, presumably, APIs) 
> > means you've abandoned #1, i.e., you haven't (really) bumped the JDK 
> > source version to JDK8.
> >
> > Also, note that releasing from trunk is a way of achieving #3, it's 
> > not a way of abandoning it.
> >
> >
> >
> > On Mon, Mar 9, 2015 at 7:10 PM, Andrew Wang 
> > 
> > wrote:
> > > Hi Raymie,
> > >
> > > Konst proposed just releasing off of trunk rather than cutting a
> > branch-2,
> > > and there was general agreement there. So, consider #3 abandoned. 
> > > 1&2
> can
> > > be achieved at the same time, we just need to avoid using JDK8 
> > > language features in trunk so things can be backported.
> > >
> > > Best,
> > > Andrew
> > >
> > > On Mon, Mar 9, 2015 at 7:01 PM, Raymie Stata 
> > > 
> > wrote:
> > >
> > >> In this (and the related threads), I see the following three
> > requirements:
> > >>
> > >> 1. "Bump the source JDK version to JDK8" (ie, drop JDK7 support).
> > >>
> > >> 2. "We'll still be releasing 2.x releases for a while, with 
> > >> similar feature sets as 3.x."
> > >>
> > >> 3. Avoid the "risk of split-brain behavior" by "minimize 
> > >> backporting headaches. Pulling trunk > branch-2 > branch-2.x is already 
> > >> tedious.
> > >> Adding a branch-3, branch-3.x would be obnoxious."
> > >>
> > >> These three cannot be achieved at the same time.  Which do we abandon?
> > >>
> > >>
> > >> On Mon, Mar 9, 2015 at 12:45 PM, sanjay Radia 
> > >> 
> > >> wrote:
> > >> >
> > >> >> On Mar 5, 2015, at 3:21 PM, Siddharth Seth 
> wrote:
> > >> >>
> > >> >> 2) Simplification of configs - potentially separating client 
> > >> >> side
> > >> configs
> > >> >> and those used by daemons. This is another source of perpetual
> > confusion
> > >> >> for users.
> > >> > + 1 on this.
> > >> >
> > >> > sanjay
> > >>
> >
>
>
>

RE: IMPORTANT: testing patches for branches

[Zheng, Kai]

Hi Allen,

This sounds great. 

 Naming a patch foo-HDFS-7285.00.patch should get tested on the HDFS-7285 
 branch.
Does it happen locally in developer's machine when running test-patch.sh, or 
also mean something in Hadoop Jenkins building when a JIRA becoming patch 
available? Thanks.

Regards,
Kai

-Original Message-
From: Allen Wittenauer [mailto:a...@altiscale.com] 
Sent: Thursday, April 23, 2015 3:35 AM
To: common-...@hadoop.apache.org
Cc: yarn-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; 
hdfs-...@hadoop.apache.org
Subject: IMPORTANT: testing patches for branches


Hey gang, 

Just so everyone is aware, if you are working on a patch for either a 
feature branch or a major branch, if you name the patch with the branch name 
following the spec in HowToContribute (and a few other ways... test-patch tries 
to figure it out!), test-patch.sh *should* be switching the repo over to that 
branch for testing. 

For example,  naming a patch foo-branch-2.01.patch should get tested on 
branch-2.  Naming a patch foo-HDFS-7285.00.patch should get tested on the 
HDFS-7285 branch.

This hopefully means that there should really be no more 'blind' +1's 
to patches that go to branches.  The we only test against trunk argument is 
no longer valid. :)

RE: Looking to a Hadoop 3 release

[Zheng, Kai]

Might I have some comments for this, just providing my thought. Thanks.

 If we start now, it might make it out by 2016. If we start now, 
 downstreamers can start aligning themselves to land versions that suit at 
 about the same time.
Not only for down streamers to align with the long term release, but also for 
contributors like me to align with their future effort, maybe.

In addition to the JDK8 support and classpath isolation, might we add more 
possible candidate considerations. 
How would you like this one, HADOOP-9797 Pluggable and compatible UGI change ?
https://issues.apache.org/jira/browse/HADOOP-9797

The benefits: 
1) allow multiple login sessions/contexts and authentication methods to be used 
in the same Java application/process without conflicts, providing good 
isolation by getting rid of globals and statics.
2) allow to pluggable new authentication methods for UGI, in modular, 
manageable and maintainable manner.

Another, we would also push the first release of Apache Kerby, preparing for a 
strong dedicated and clean Kerberos library in Java for both client and KDC 
sides, and by leveraging the library, 
update Hadoop-MiniKDC and perform more security tests.
https://issues.apache.org/jira/browse/DIRKRB-102

Hope this makes sense. Thanks.

Regards,
Kai

-Original Message-
From: saint@gmail.com [mailto:saint@gmail.com] On Behalf Of Stack
Sent: Thursday, March 05, 2015 2:47 AM
To: common-...@hadoop.apache.org
Cc: mapreduce-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org
Subject: Re: Looking to a Hadoop 3 release

In general +1 on 3.0.0. Its time. If we start now, it might make it out by 
2016. If we start now, downstreamers can start aligning themselves to land 
versions that suit at about the same time.

While two big items have been called out as possible incompatible changes, and 
there is ongoing discussion as to whether they are or not*, is there any chance 
of getting a longer list of big differences between the branches? In particular 
I'd be interested in improvements that are 'off' by default that would be 
better defaulted 'on'.

Thanks,
St.Ack

* Let me note that 'compatible' around these parts is a trampled concept 
seemingly open to interpretation with a definition that is other than prevails 
elsewhere in software. See Allen's list above, and in our downstream project, 
the recent HBASE-13149 HBase server MR tools are broken on Hadoop 2.5+ Yarn, 
among others.  Let 3.x be incompatible with 2.x if only so we can leave behind 
all current notions of 'compatibility'
and just start over (as per Allen).


On Mon, Mar 2, 2015 at 3:19 PM, Andrew Wang andrew.w...@cloudera.com
wrote:

 Hi devs,

 It's been a year and a half since 2.x went GA, and I think we're about 
 due for a 3.x release.
 Notably, there are two incompatible changes I'd like to call out, that 
 will have a tremendous positive impact for our users.

 First, classpath isolation being done at HADOOP-11656, which has been 
 a long-standing request from many downstreams and Hadoop users.

 Second, bumping the source and target JDK version to JDK8 (related to 
 HADOOP-11090), which is important since JDK7 is EOL in April 2015 (two 
 months from now). In the past, we've had issues with our dependencies 
 discontinuing support for old JDKs, so this will future-proof us.

 Between the two, we'll also have quite an opportunity to clean up and 
 upgrade our dependencies, another common user and developer request.

 I'd like to propose that we start rolling a series of monthly-ish 
 series of
 3.0 alpha releases ASAP, with myself volunteering to take on the RM 
 and other cat herding responsibilities. There are already quite a few 
 changes slated for 3.0 besides the above (for instance the shell 
 script rewrite) so there's already value in a 3.0 alpha, and the more 
 time we give downstreams to integrate, the better.

 This opens up discussion about inclusion of other changes, but I'm 
 hoping to freeze incompatible changes after maybe two alphas, do a 
 beta (with no further incompat changes allowed), and then finally a 
 3.x GA. For those keeping track, that means a 3.x GA in about four months.

 I would also like to stress though that this is not intended to be a 
 big bang release. For instance, it would be great if we could maintain 
 wire compatibility between 2.x and 3.x, so rolling upgrades work. 
 Keeping
 branch-2 and branch-3 similar also makes backports easier, since we're 
 likely maintaining 2.x for a while yet.

 Please let me know any comments / concerns related to the above. If 
 people are friendly to the idea, I'd like to cut a branch-3 and start 
 working on the first alpha.

 Best,
 Andrew

RE: 2.7 status

[Zheng, Kai]

Thanks Vinod for the hints. 

I have updated the both patches aligning with latest codes, and added more unit 
tests. The building results look reasonable. Thanks anyone that would give them 
more review and I would update in timely manner. 

Regards,
Kai

-Original Message-
From: Vinod Kumar Vavilapalli [mailto:vino...@hortonworks.com] 
Sent: Tuesday, March 03, 2015 11:31 AM
To: Zheng, Kai
Cc: mapreduce-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org; Hadoop Common; 
yarn-...@hadoop.apache.org
Subject: Re: 2.7 status

Kai, please ping the reviewers that were already looking at your patches 
before. If the patches go in by end of this week, we can include them.

Thanks,
+Vinod

On Mar 2, 2015, at 7:04 PM, Zheng, Kai kai.zh...@intel.com wrote:

 Is it interested to get the following issues in the release ? Thanks !
 
 HADOOP-10670
 HADOOP-10671
 
 Regards,
 Kai
 
 -Original Message-
 From: Yongjun Zhang [mailto:yzh...@cloudera.com]
 Sent: Monday, March 02, 2015 4:46 AM
 To: hdfs-...@hadoop.apache.org
 Cc: Vinod Kumar Vavilapalli; Hadoop Common; 
 mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
 Subject: Re: 2.7 status
 
 Hi,
 
 Thanks for working on 2.7 release.
 
 Currently the fallback from KerberosAuthenticator to PseudoAuthenticator  is 
 enabled by default in a hardcoded way. HAOOP-10895 changes the default and 
 requires applications (such as oozie) to set a config property or call an API 
 to enable the fallback.
 
 This jira has been reviewed, and almost ready to get in. However, there is 
 a concern that we have to change the relevant applications. Please see my 
 comment here:
 
 https://issues.apache.org/jira/browse/HADOOP-10895?focusedCommentId=14
 321823page=com.atlassian.jira.plugin.system.issuetabpanels:comment-ta
 bpanel#comment-14321823
 
 Any of your comments will be highly appreciated. This jira was postponed from 
 2.6. I think it should be no problem to skip 2.7. But your comments would 
 help us to decide what to do with this jira for future releases.
 
 Thanks.
 
 --Yongjun
 
 
 On Sun, Mar 1, 2015 at 11:58 AM, Arun Murthy a...@hortonworks.com wrote:
 
 Sounds good, thanks for the help Vinod!
 
 Arun
 
 
 From: Vinod Kumar Vavilapalli
 Sent: Sunday, March 01, 2015 11:43 AM
 To: Hadoop Common; Jason Lowe; Arun Murthy
 Subject: Re: 2.7 status
 
 Agreed. How about we roll an RC end of this week? As a Java 7+ 
 release with features, patches that already got in?
 
 Here's a filter tracking blocker tickets - 
 https://issues.apache.org/jira/issues/?filter=12330598. Nine open now.
 
 +Arun
 Arun, I'd like to help get 2.7 out without further delay. Do you mind 
 me taking over release duties?
 
 Thanks,
 +Vinod
 
 From: Jason Lowe jl...@yahoo-inc.com.INVALID
 Sent: Friday, February 13, 2015 8:11 AM
 To: common-...@hadoop.apache.org
 Subject: Re: 2.7 status
 
 I'd like to see a 2.7 release sooner than later.  It has been almost 
 3 months since Hadoop 2.6 was released, and there have already been 
 634 JIRAs committed to 2.7.  That's a lot of changes waiting for an official 
 release.
 
 https://issues.apache.org/jira/issues/?jql=project%20in%20%28hadoop%2
 C 
 hdfs%2Cyarn%2Cmapreduce%29%20AND%20fixversion%3D2.7.0%20AND%20resolut
 i
 on%3DFixed
 Jason
 
  From: Sangjin Lee sj...@apache.org
 To: common-...@hadoop.apache.org common-...@hadoop.apache.org
 Sent: Tuesday, February 10, 2015 1:30 PM
 Subject: 2.7 status
 
 Folks,
 
 What is the current status of the 2.7 release? I know initially it 
 started out as a java-7 only release, but looking at the JIRAs that 
 is very much not the case.
 
 Do we have a certain timeframe for 2.7 or is it time to discuss it?
 
 Thanks,
 Sangjin
 
 

RE: Looking to a Hadoop 3 release

[Zheng, Kai]

JDK8 support is in the consideration, looks like many issues were reported and 
resolved already.

https://issues.apache.org/jira/browse/HADOOP-11090


-Original Message-
From: Andrew Wang [mailto:andrew.w...@cloudera.com] 
Sent: Tuesday, March 03, 2015 7:20 AM
To: common-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; 
hdfs-...@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Looking to a Hadoop 3 release

Hi devs,

It's been a year and a half since 2.x went GA, and I think we're about due for 
a 3.x release.
Notably, there are two incompatible changes I'd like to call out, that will 
have a tremendous positive impact for our users.

First, classpath isolation being done at HADOOP-11656, which has been a 
long-standing request from many downstreams and Hadoop users.

Second, bumping the source and target JDK version to JDK8 (related to 
HADOOP-11090), which is important since JDK7 is EOL in April 2015 (two months 
from now). In the past, we've had issues with our dependencies discontinuing 
support for old JDKs, so this will future-proof us.

Between the two, we'll also have quite an opportunity to clean up and upgrade 
our dependencies, another common user and developer request.

I'd like to propose that we start rolling a series of monthly-ish series of
3.0 alpha releases ASAP, with myself volunteering to take on the RM and other 
cat herding responsibilities. There are already quite a few changes slated for 
3.0 besides the above (for instance the shell script rewrite) so there's 
already value in a 3.0 alpha, and the more time we give downstreams to 
integrate, the better.

This opens up discussion about inclusion of other changes, but I'm hoping to 
freeze incompatible changes after maybe two alphas, do a beta (with no further 
incompat changes allowed), and then finally a 3.x GA. For those keeping track, 
that means a 3.x GA in about four months.

I would also like to stress though that this is not intended to be a big bang 
release. For instance, it would be great if we could maintain wire 
compatibility between 2.x and 3.x, so rolling upgrades work. Keeping
branch-2 and branch-3 similar also makes backports easier, since we're likely 
maintaining 2.x for a while yet.

Please let me know any comments / concerns related to the above. If people are 
friendly to the idea, I'd like to cut a branch-3 and start working on the first 
alpha.

Best,
Andrew

RE: Looking to a Hadoop 3 release

[Zheng, Kai]

Sorry for the bad. I thought it was sending to my colleagues. 

By the way, for the JDK8 support, we (Intel) would like to investigate further 
and help, thanks.

Regards,
Kai

-Original Message-
From: Zheng, Kai 
Sent: Tuesday, March 03, 2015 8:49 AM
To: common-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; 
hdfs-...@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: RE: Looking to a Hadoop 3 release

JDK8 support is in the consideration, looks like many issues were reported and 
resolved already.

https://issues.apache.org/jira/browse/HADOOP-11090


-Original Message-
From: Andrew Wang [mailto:andrew.w...@cloudera.com] 
Sent: Tuesday, March 03, 2015 7:20 AM
To: common-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; 
hdfs-...@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Looking to a Hadoop 3 release

Hi devs,

It's been a year and a half since 2.x went GA, and I think we're about due for 
a 3.x release.
Notably, there are two incompatible changes I'd like to call out, that will 
have a tremendous positive impact for our users.

First, classpath isolation being done at HADOOP-11656, which has been a 
long-standing request from many downstreams and Hadoop users.

Second, bumping the source and target JDK version to JDK8 (related to 
HADOOP-11090), which is important since JDK7 is EOL in April 2015 (two months 
from now). In the past, we've had issues with our dependencies discontinuing 
support for old JDKs, so this will future-proof us.

Between the two, we'll also have quite an opportunity to clean up and upgrade 
our dependencies, another common user and developer request.

I'd like to propose that we start rolling a series of monthly-ish series of
3.0 alpha releases ASAP, with myself volunteering to take on the RM and other 
cat herding responsibilities. There are already quite a few changes slated for 
3.0 besides the above (for instance the shell script rewrite) so there's 
already value in a 3.0 alpha, and the more time we give downstreams to 
integrate, the better.

This opens up discussion about inclusion of other changes, but I'm hoping to 
freeze incompatible changes after maybe two alphas, do a beta (with no further 
incompat changes allowed), and then finally a 3.x GA. For those keeping track, 
that means a 3.x GA in about four months.

I would also like to stress though that this is not intended to be a big bang 
release. For instance, it would be great if we could maintain wire 
compatibility between 2.x and 3.x, so rolling upgrades work. Keeping
branch-2 and branch-3 similar also makes backports easier, since we're likely 
maintaining 2.x for a while yet.

Please let me know any comments / concerns related to the above. If people are 
friendly to the idea, I'd like to cut a branch-3 and start working on the first 
alpha.

Best,
Andrew

RE: 2.7 status

[Zheng, Kai]

Is it interested to get the following issues in the release ? Thanks !

HADOOP-10670
HADOOP-10671

Regards,
Kai

-Original Message-
From: Yongjun Zhang [mailto:yzh...@cloudera.com] 
Sent: Monday, March 02, 2015 4:46 AM
To: hdfs-...@hadoop.apache.org
Cc: Vinod Kumar Vavilapalli; Hadoop Common; mapreduce-dev@hadoop.apache.org; 
yarn-...@hadoop.apache.org
Subject: Re: 2.7 status

Hi,

Thanks for working on 2.7 release.

Currently the fallback from KerberosAuthenticator to PseudoAuthenticator  is 
enabled by default in a hardcoded way. HAOOP-10895 changes the default and 
requires applications (such as oozie) to set a config property or call an API 
to enable the fallback.

This jira has been reviewed, and almost ready to get in. However, there is a 
concern that we have to change the relevant applications. Please see my comment 
here:

https://issues.apache.org/jira/browse/HADOOP-10895?focusedCommentId=14321823page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14321823

Any of your comments will be highly appreciated. This jira was postponed from 
2.6. I think it should be no problem to skip 2.7. But your comments would help 
us to decide what to do with this jira for future releases.

Thanks.

--Yongjun


On Sun, Mar 1, 2015 at 11:58 AM, Arun Murthy a...@hortonworks.com wrote:

 Sounds good, thanks for the help Vinod!

 Arun

 
 From: Vinod Kumar Vavilapalli
 Sent: Sunday, March 01, 2015 11:43 AM
 To: Hadoop Common; Jason Lowe; Arun Murthy
 Subject: Re: 2.7 status

 Agreed. How about we roll an RC end of this week? As a Java 7+ release 
 with features, patches that already got in?

 Here's a filter tracking blocker tickets - 
 https://issues.apache.org/jira/issues/?filter=12330598. Nine open now.

 +Arun
 Arun, I'd like to help get 2.7 out without further delay. Do you mind 
 me taking over release duties?

 Thanks,
 +Vinod
 
 From: Jason Lowe jl...@yahoo-inc.com.INVALID
 Sent: Friday, February 13, 2015 8:11 AM
 To: common-...@hadoop.apache.org
 Subject: Re: 2.7 status

 I'd like to see a 2.7 release sooner than later.  It has been almost 3 
 months since Hadoop 2.6 was released, and there have already been 634 
 JIRAs committed to 2.7.  That's a lot of changes waiting for an official 
 release.

 https://issues.apache.org/jira/issues/?jql=project%20in%20%28hadoop%2C
 hdfs%2Cyarn%2Cmapreduce%29%20AND%20fixversion%3D2.7.0%20AND%20resoluti
 on%3DFixed
 Jason

   From: Sangjin Lee sj...@apache.org
  To: common-...@hadoop.apache.org common-...@hadoop.apache.org
  Sent: Tuesday, February 10, 2015 1:30 PM
  Subject: 2.7 status

 Folks,

 What is the current status of the 2.7 release? I know initially it 
 started out as a java-7 only release, but looking at the JIRAs that 
 is very much not the case.

 Do we have a certain timeframe for 2.7 or is it time to discuss it?

 Thanks,
 Sangjin