RE: Planning Hadoop 2.6.1 release

[Brahma Reddy Battula]

HI vinod

any update on this..? are we planning to give 2.6.1 Or can we make 2.7.1 as 
stable give..?


Thanks & Regards
 Brahma Reddy Battula


From: Zhihai Xu [z...@cloudera.com]
Sent: Wednesday, May 13, 2015 12:04 PM
To: mapreduce-...@hadoop.apache.org
Cc: common-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; 
hdfs-...@hadoop.apache.org
Subject: Re: Planning Hadoop 2.6.1 release

Hi Akira,

Can we also include YARN-3242? YARN-3242 fixed a critical ZKRMStateStore
bug.
It will work better with YARN-2992.

thanks
zhihai


On Tue, May 12, 2015 at 10:38 PM, Akira AJISAKA 
wrote:

> Thanks all for collecting jiras for 2.6.1 release. In addition, I'd like
> to include the following:
>
> * HADOOP-11343. Overflow is not properly handled in calculating final iv
> for AES CTR
> * YARN-2874. Dead lock in "DelegationTokenRenewer" which blocks RM to
> execute any further apps
> * YARN-2992. ZKRMStateStore crashes due to session expiry
> * YARN-3013. AMRMClientImpl does not update AMRM token properly
> * YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to
> die
> * MAPREDUCE-6303. Read timeout when retrying a fetch error can be fatal to
> a reducer
>
> All of these are marked as blocker bug for 2.7.0 but not fixed in 2.6.0.
>
> Regards,
> Akira
>
>
> On 5/4/15 11:15, Brahma Reddy Battula wrote:
>
>> Hello Vinod,
>>
>> I am thinking,can we include HADOOP-11491 also..? wihout this jira harfs
>> will not be usable when cluster installed in HA mode and try to get
>> filecontext like below..
>>
>>
>> Path path = new
>> Path("har:///archivedLogs/application_1428917727658_0005-application_1428917727658_0008-1428927448352.har");
>> FileSystem fs = path.getFileSystem(new Configuration());
>> path = fs.makeQualified(path);
>> FileContext fc = FileContext.getFileContext(path.toUri(),new
>> Configuration());
>>
>>
>>
>> Thanks & Regards
>> Brahma Reddy Battula
>> 
>> From: Chris Nauroth [cnaur...@hortonworks.com]
>> Sent: Friday, May 01, 2015 4:32 AM
>> To: mapreduce-...@hadoop.apache.org; common-dev@hadoop.apache.org;
>> yarn-...@hadoop.apache.org; hdfs-...@hadoop.apache.org
>> Subject: Re: Planning Hadoop 2.6.1 release
>>
>> Thank you, Arpit.  In addition, I suggest we include the following:
>>
>> HADOOP-11333. Fix deadlock in DomainSocketWatcher when the notification
>> pipe is full
>> HADOOP-11604. Prevent ConcurrentModificationException while closing domain
>> sockets during shutdown of DomainSocketWatcher thread.
>> HADOOP-11648. Set DomainSocketWatcher thread name explicitly
>> HADOOP-11802. DomainSocketWatcher thread terminates sometimes after there
>> is an I/O error during requestShortCircuitShm
>>
>> HADOOP-11604 and 11648 are not critical by themselves, but they are
>> pre-requisites to getting a clean cherry-pick of 11802, which we believe
>> finally fixes the root cause of this issue.
>>
>>
>> --Chris Nauroth
>>
>>
>>
>>
>> On 4/30/15, 3:55 PM, "Arpit Agarwal"  wrote:
>>
>>  HDFS candidates for back-porting to Hadoop 2.6.1. The first two were
>>> requested in [1].
>>>
>>> HADOOP-11674. oneByteBuf in CryptoInputStream and CryptoOutputStream
>>> should be non static
>>> HADOOP-11710. Make CryptoOutputStream behave like DFSOutputStream wrt
>>> synchronization
>>>
>>> HDFS-7009. Active NN and standby NN have different live nodes.
>>> HDFS-7035. Make adding a new data directory to the DataNode an atomic and
>>> improve error handling
>>> HDFS-7425. NameNode block deletion logging uses incorrect appender.
>>> HDFS-7443. Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate
>>> block files are present in the same volume.
>>> HDFS-7489. Incorrect locking in FsVolumeList#checkDirs can hang datanodes
>>> HDFS-7503. Namenode restart after large deletions can cause slow
>>> processReport.
>>> HDFS-7575. Upgrade should generate a unique storage ID for each volume.
>>> HDFS-7579. Improve log reporting during block report rpc failure.
>>> HDFS-7587. Edit log corruption can happen if append fails with a quota
>>> violation.
>>> HDFS-7596. NameNode should prune dead storages from storageMap.
>>> HDFS-7611. deleteSnapshot and delete of a file can leave orphaned blocks
>>> in the blocksMap on NameNode restart.
>>> HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause
>>> DataNode to r

Re: Planning Hadoop 2.6.1 release

[Sean Busbey]

Any update on a release plan for 2.6.1?

On Wed, Jun 10, 2015 at 1:25 AM, Brahma Reddy Battula <
brahmareddy.batt...@huawei.com> wrote:

> HI vinod
>
> any update on this..? are we planning to give 2.6.1 Or can we make 2.7.1
> as stable give..?
>
>
> Thanks & Regards
>  Brahma Reddy Battula
>
> 
> From: Zhihai Xu [z...@cloudera.com]
> Sent: Wednesday, May 13, 2015 12:04 PM
> To: mapreduce-...@hadoop.apache.org
> Cc: common-dev@hadoop.apache.org; yarn-...@hadoop.apache.org;
> hdfs-...@hadoop.apache.org
> Subject: Re: Planning Hadoop 2.6.1 release
>
> Hi Akira,
>
> Can we also include YARN-3242? YARN-3242 fixed a critical ZKRMStateStore
> bug.
> It will work better with YARN-2992.
>
> thanks
> zhihai
>
>
> On Tue, May 12, 2015 at 10:38 PM, Akira AJISAKA <
> ajisa...@oss.nttdata.co.jp>
> wrote:
>
> > Thanks all for collecting jiras for 2.6.1 release. In addition, I'd like
> > to include the following:
> >
> > * HADOOP-11343. Overflow is not properly handled in calculating final iv
> > for AES CTR
> > * YARN-2874. Dead lock in "DelegationTokenRenewer" which blocks RM to
> > execute any further apps
> > * YARN-2992. ZKRMStateStore crashes due to session expiry
> > * YARN-3013. AMRMClientImpl does not update AMRM token properly
> > * YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to
> > die
> > * MAPREDUCE-6303. Read timeout when retrying a fetch error can be fatal
> to
> > a reducer
> >
> > All of these are marked as blocker bug for 2.7.0 but not fixed in 2.6.0.
> >
> > Regards,
> > Akira
> >
> >
> > On 5/4/15 11:15, Brahma Reddy Battula wrote:
> >
> >> Hello Vinod,
> >>
> >> I am thinking,can we include HADOOP-11491 also..? wihout this jira harfs
> >> will not be usable when cluster installed in HA mode and try to get
> >> filecontext like below..
> >>
> >>
> >> Path path = new
> >>
> Path("har:///archivedLogs/application_1428917727658_0005-application_1428917727658_0008-1428927448352.har");
> >> FileSystem fs = path.getFileSystem(new Configuration());
> >> path = fs.makeQualified(path);
> >> FileContext fc = FileContext.getFileContext(path.toUri(),new
> >> Configuration());
> >>
> >>
> >>
> >> Thanks & Regards
> >> Brahma Reddy Battula
> >> 
> >> From: Chris Nauroth [cnaur...@hortonworks.com]
> >> Sent: Friday, May 01, 2015 4:32 AM
> >> To: mapreduce-...@hadoop.apache.org; common-dev@hadoop.apache.org;
> >> yarn-...@hadoop.apache.org; hdfs-...@hadoop.apache.org
> >> Subject: Re: Planning Hadoop 2.6.1 release
> >>
> >> Thank you, Arpit.  In addition, I suggest we include the following:
> >>
> >> HADOOP-11333. Fix deadlock in DomainSocketWatcher when the notification
> >> pipe is full
> >> HADOOP-11604. Prevent ConcurrentModificationException while closing
> domain
> >> sockets during shutdown of DomainSocketWatcher thread.
> >> HADOOP-11648. Set DomainSocketWatcher thread name explicitly
> >> HADOOP-11802. DomainSocketWatcher thread terminates sometimes after
> there
> >> is an I/O error during requestShortCircuitShm
> >>
> >> HADOOP-11604 and 11648 are not critical by themselves, but they are
> >> pre-requisites to getting a clean cherry-pick of 11802, which we believe
> >> finally fixes the root cause of this issue.
> >>
> >>
> >> --Chris Nauroth
> >>
> >>
> >>
> >>
> >> On 4/30/15, 3:55 PM, "Arpit Agarwal"  wrote:
> >>
> >>  HDFS candidates for back-porting to Hadoop 2.6.1. The first two were
> >>> requested in [1].
> >>>
> >>> HADOOP-11674. oneByteBuf in CryptoInputStream and CryptoOutputStream
> >>> should be non static
> >>> HADOOP-11710. Make CryptoOutputStream behave like DFSOutputStream wrt
> >>> synchronization
> >>>
> >>> HDFS-7009. Active NN and standby NN have different live nodes.
> >>> HDFS-7035. Make adding a new data directory to the DataNode an atomic
> and
> >>> improve error handling
> >>> HDFS-7425. NameNode block deletion logging uses incorrect appender.
> >>> HDFS-7443. Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate
> >>> block files are present in the same volume.
> >>> HDFS-7489. Incorr

Re: Planning Hadoop 2.6.1 release

[Vinod Kumar Vavilapalli]

Got pinged on a recent thread on this one.

As I mentioned there, I had many offline discussions re 2.6.1.

The biggest problem I found offline was about what bug-fixes are acceptable and 
what aren’t for everyone wishing to consume 2.6.1. Given the number of 
bug-fixes that went into 2.7.x and into branch-2.8, figuring out a set of 
patches that is acceptable for everyone is a huge challenge which kind of 
stalled my attempts.

Thanks
+Vinod


> On Jul 1, 2015, at 12:41 PM, Sean Busbey  wrote:
> 
> Any update on a release plan for 2.6.1?
> 
> On Wed, Jun 10, 2015 at 1:25 AM, Brahma Reddy Battula <
> brahmareddy.batt...@huawei.com> wrote:
> 
>> HI vinod
>> 
>> any update on this..? are we planning to give 2.6.1 Or can we make 2.7.1
>> as stable give..?
>> 
>> 
>> Thanks & Regards
>> Brahma Reddy Battula
>> 
>> 
>> From: Zhihai Xu [z...@cloudera.com]
>> Sent: Wednesday, May 13, 2015 12:04 PM
>> To: mapreduce-...@hadoop.apache.org
>> Cc: common-dev@hadoop.apache.org; yarn-...@hadoop.apache.org;
>> hdfs-...@hadoop.apache.org
>> Subject: Re: Planning Hadoop 2.6.1 release
>> 
>> Hi Akira,
>> 
>> Can we also include YARN-3242? YARN-3242 fixed a critical ZKRMStateStore
>> bug.
>> It will work better with YARN-2992.
>> 
>> thanks
>> zhihai
>> 
>> 
>> On Tue, May 12, 2015 at 10:38 PM, Akira AJISAKA <
>> ajisa...@oss.nttdata.co.jp>
>> wrote:
>> 
>>> Thanks all for collecting jiras for 2.6.1 release. In addition, I'd like
>>> to include the following:
>>> 
>>> * HADOOP-11343. Overflow is not properly handled in calculating final iv
>>> for AES CTR
>>> * YARN-2874. Dead lock in "DelegationTokenRenewer" which blocks RM to
>>> execute any further apps
>>> * YARN-2992. ZKRMStateStore crashes due to session expiry
>>> * YARN-3013. AMRMClientImpl does not update AMRM token properly
>>> * YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to
>>> die
>>> * MAPREDUCE-6303. Read timeout when retrying a fetch error can be fatal
>> to
>>> a reducer
>>> 
>>> All of these are marked as blocker bug for 2.7.0 but not fixed in 2.6.0.
>>> 
>>> Regards,
>>> Akira
>>> 
>>> 
>>> On 5/4/15 11:15, Brahma Reddy Battula wrote:
>>> 
>>>> Hello Vinod,
>>>> 
>>>> I am thinking,can we include HADOOP-11491 also..? wihout this jira harfs
>>>> will not be usable when cluster installed in HA mode and try to get
>>>> filecontext like below..
>>>> 
>>>> 
>>>> Path path = new
>>>> 
>> Path("har:///archivedLogs/application_1428917727658_0005-application_1428917727658_0008-1428927448352.har");
>>>> FileSystem fs = path.getFileSystem(new Configuration());
>>>> path = fs.makeQualified(path);
>>>> FileContext fc = FileContext.getFileContext(path.toUri(),new
>>>> Configuration());
>>>> 
>>>> 
>>>> 
>>>> Thanks & Regards
>>>> Brahma Reddy Battula
>>>> 
>>>> From: Chris Nauroth [cnaur...@hortonworks.com]
>>>> Sent: Friday, May 01, 2015 4:32 AM
>>>> To: mapreduce-...@hadoop.apache.org; common-dev@hadoop.apache.org;
>>>> yarn-...@hadoop.apache.org; hdfs-...@hadoop.apache.org
>>>> Subject: Re: Planning Hadoop 2.6.1 release
>>>> 
>>>> Thank you, Arpit.  In addition, I suggest we include the following:
>>>> 
>>>> HADOOP-11333. Fix deadlock in DomainSocketWatcher when the notification
>>>> pipe is full
>>>> HADOOP-11604. Prevent ConcurrentModificationException while closing
>> domain
>>>> sockets during shutdown of DomainSocketWatcher thread.
>>>> HADOOP-11648. Set DomainSocketWatcher thread name explicitly
>>>> HADOOP-11802. DomainSocketWatcher thread terminates sometimes after
>> there
>>>> is an I/O error during requestShortCircuitShm
>>>> 
>>>> HADOOP-11604 and 11648 are not critical by themselves, but they are
>>>> pre-requisites to getting a clean cherry-pick of 11802, which we believe
>>>> finally fixes the root cause of this issue.
>>>> 
>>>> 
>>>> --Chris Nauroth
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On 4/30/15, 3:55 PM, "Arpit Agarwal"  wrote:
>

Re: Planning Hadoop 2.6.1 release

[Vinod Kumar Vavilapalli]

Alright, I’d like to make progress while the issue is hot.

I created a label to discuss on the candidate list of patches: 
https://issues.apache.org/jira/issues/?jql=labels%20%3D%202.6.1-candidate<https://issues.apache.org/jira/issues/?jql=labels%20=%202.6.1-candidate>

Next steps, I’ll do the following
 - Review 2.7 and 2.8 blocker/critical tickets and see what makes sense for 
2.6.1 and add as candidates
 - I haven’t reviewed the current list yet, the seed list is from this email 
thread. Will review them.
 - I also have a bunch of patches that I’d like to include, will update them 
right away.

Others, please look at the current list and let me know what else you’d like to 
include.

I’d like to keep this ‘candidate-collection’ cycle’ for a max of a week and 
then start the release process. @Akira, let’s sync up offline on how to take 
this forward in terms of the release process.

Thanks
+Vinod


On Jul 15, 2015, at 1:12 PM, Vinod Kumar Vavilapalli 
mailto:vino...@hortonworks.com>> wrote:

Got pinged on a recent thread on this one.

As I mentioned there, I had many offline discussions re 2.6.1.

The biggest problem I found offline was about what bug-fixes are acceptable and 
what aren’t for everyone wishing to consume 2.6.1. Given the number of 
bug-fixes that went into 2.7.x and into branch-2.8, figuring out a set of 
patches that is acceptable for everyone is a huge challenge which kind of 
stalled my attempts.

Thanks
+Vinod


On Jul 1, 2015, at 12:41 PM, Sean Busbey 
mailto:bus...@cloudera.com>> wrote:

Any update on a release plan for 2.6.1?

On Wed, Jun 10, 2015 at 1:25 AM, Brahma Reddy Battula <
brahmareddy.batt...@huawei.com<mailto:brahmareddy.batt...@huawei.com>> wrote:

HI vinod

any update on this..? are we planning to give 2.6.1 Or can we make 2.7.1
as stable give..?


Thanks & Regards
Brahma Reddy Battula


From: Zhihai Xu [z...@cloudera.com<mailto:z...@cloudera.com>]
Sent: Wednesday, May 13, 2015 12:04 PM
To: mapreduce-...@hadoop.apache.org<mailto:mapreduce-...@hadoop.apache.org>
Cc: common-dev@hadoop.apache.org<mailto:common-dev@hadoop.apache.org>; 
yarn-...@hadoop.apache.org<mailto:yarn-...@hadoop.apache.org>;
hdfs-...@hadoop.apache.org<mailto:hdfs-...@hadoop.apache.org>
Subject: Re: Planning Hadoop 2.6.1 release

Hi Akira,

Can we also include YARN-3242? YARN-3242 fixed a critical ZKRMStateStore
bug.
It will work better with YARN-2992.

thanks
zhihai


On Tue, May 12, 2015 at 10:38 PM, Akira AJISAKA <
ajisa...@oss.nttdata.co.jp>
wrote:

Thanks all for collecting jiras for 2.6.1 release. In addition, I'd like
to include the following:

* HADOOP-11343. Overflow is not properly handled in calculating final iv
for AES CTR
* YARN-2874. Dead lock in "DelegationTokenRenewer" which blocks RM to
execute any further apps
* YARN-2992. ZKRMStateStore crashes due to session expiry
* YARN-3013. AMRMClientImpl does not update AMRM token properly
* YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to
die
* MAPREDUCE-6303. Read timeout when retrying a fetch error can be fatal
to
a reducer

All of these are marked as blocker bug for 2.7.0 but not fixed in 2.6.0.

Regards,
Akira


On 5/4/15 11:15, Brahma Reddy Battula wrote:

Hello Vinod,

I am thinking,can we include HADOOP-11491 also..? wihout this jira harfs
will not be usable when cluster installed in HA mode and try to get
filecontext like below..


Path path = new

Path("har:///archivedLogs/application_1428917727658_0005-application_1428917727658_0008-1428927448352.har");
FileSystem fs = path.getFileSystem(new Configuration());
path = fs.makeQualified(path);
FileContext fc = FileContext.getFileContext(path.toUri(),new
Configuration());



Thanks & Regards
Brahma Reddy Battula

From: Chris Nauroth [cnaur...@hortonworks.com]
Sent: Friday, May 01, 2015 4:32 AM
To: mapreduce-...@hadoop.apache.org; common-dev@hadoop.apache.org;
yarn-...@hadoop.apache.org; hdfs-...@hadoop.apache.org
Subject: Re: Planning Hadoop 2.6.1 release

Thank you, Arpit.  In addition, I suggest we include the following:

HADOOP-11333. Fix deadlock in DomainSocketWatcher when the notification
pipe is full
HADOOP-11604. Prevent ConcurrentModificationException while closing
domain
sockets during shutdown of DomainSocketWatcher thread.
HADOOP-11648. Set DomainSocketWatcher thread name explicitly
HADOOP-11802. DomainSocketWatcher thread terminates sometimes after
there
is an I/O error during requestShortCircuitShm

HADOOP-11604 and 11648 are not critical by themselves, but they are
pre-requisites to getting a clean cherry-pick of 11802, which we believe
finally fixes the root cause of this issue.


--Chris Nauroth




On 4/30/15, 3:55 PM, "Arpit Agarwal"  wrote:

HDFS candidates for back-porting to Hadoop 2.6.1. The first two were
requested in [1].

HADOOP-11674. oneByteBuf in Cr

Re: Planning Hadoop 2.6.1 release

[Akira AJISAKA]

Thanks Vinod for creating the list!

> @Akira, let’s sync up offline on how to take this forward in terms of 
the release process.


Okay.

Thanks,
Akira

On 7/16/15 10:24, Vinod Kumar Vavilapalli wrote:

Alright, I’d like to make progress while the issue is hot.

I created a label to discuss on the candidate list of patches: 
https://issues.apache.org/jira/issues/?jql=labels%20%3D%202.6.1-candidate<https://issues.apache.org/jira/issues/?jql=labels%20=%202.6.1-candidate>

Next steps, I’ll do the following
  - Review 2.7 and 2.8 blocker/critical tickets and see what makes sense for 
2.6.1 and add as candidates
  - I haven’t reviewed the current list yet, the seed list is from this email 
thread. Will review them.
  - I also have a bunch of patches that I’d like to include, will update them 
right away.

Others, please look at the current list and let me know what else you’d like to 
include.

I’d like to keep this ‘candidate-collection’ cycle’ for a max of a week and 
then start the release process. @Akira, let’s sync up offline on how to take 
this forward in terms of the release process.

Thanks
+Vinod


On Jul 15, 2015, at 1:12 PM, Vinod Kumar Vavilapalli 
mailto:vino...@hortonworks.com>> wrote:

Got pinged on a recent thread on this one.

As I mentioned there, I had many offline discussions re 2.6.1.

The biggest problem I found offline was about what bug-fixes are acceptable and 
what aren’t for everyone wishing to consume 2.6.1. Given the number of 
bug-fixes that went into 2.7.x and into branch-2.8, figuring out a set of 
patches that is acceptable for everyone is a huge challenge which kind of 
stalled my attempts.

Thanks
+Vinod


On Jul 1, 2015, at 12:41 PM, Sean Busbey 
mailto:bus...@cloudera.com>> wrote:

Any update on a release plan for 2.6.1?

On Wed, Jun 10, 2015 at 1:25 AM, Brahma Reddy Battula <
brahmareddy.batt...@huawei.com<mailto:brahmareddy.batt...@huawei.com>> wrote:

HI vinod

any update on this..? are we planning to give 2.6.1 Or can we make 2.7.1
as stable give..?


Thanks & Regards
Brahma Reddy Battula


From: Zhihai Xu [z...@cloudera.com<mailto:z...@cloudera.com>]
Sent: Wednesday, May 13, 2015 12:04 PM
To: mapreduce-...@hadoop.apache.org<mailto:mapreduce-...@hadoop.apache.org>
Cc: common-dev@hadoop.apache.org<mailto:common-dev@hadoop.apache.org>; 
yarn-...@hadoop.apache.org<mailto:yarn-...@hadoop.apache.org>;
hdfs-...@hadoop.apache.org<mailto:hdfs-...@hadoop.apache.org>
Subject: Re: Planning Hadoop 2.6.1 release

Hi Akira,

Can we also include YARN-3242? YARN-3242 fixed a critical ZKRMStateStore
bug.
It will work better with YARN-2992.

thanks
zhihai


On Tue, May 12, 2015 at 10:38 PM, Akira AJISAKA <
ajisa...@oss.nttdata.co.jp>
wrote:

Thanks all for collecting jiras for 2.6.1 release. In addition, I'd like
to include the following:

* HADOOP-11343. Overflow is not properly handled in calculating final iv
for AES CTR
* YARN-2874. Dead lock in "DelegationTokenRenewer" which blocks RM to
execute any further apps
* YARN-2992. ZKRMStateStore crashes due to session expiry
* YARN-3013. AMRMClientImpl does not update AMRM token properly
* YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to
die
* MAPREDUCE-6303. Read timeout when retrying a fetch error can be fatal
to
a reducer

All of these are marked as blocker bug for 2.7.0 but not fixed in 2.6.0.

Regards,
Akira


On 5/4/15 11:15, Brahma Reddy Battula wrote:

Hello Vinod,

I am thinking,can we include HADOOP-11491 also..? wihout this jira harfs
will not be usable when cluster installed in HA mode and try to get
filecontext like below..


Path path = new

Path("har:///archivedLogs/application_1428917727658_0005-application_1428917727658_0008-1428927448352.har");
FileSystem fs = path.getFileSystem(new Configuration());
path = fs.makeQualified(path);
FileContext fc = FileContext.getFileContext(path.toUri(),new
Configuration());



Thanks & Regards
Brahma Reddy Battula

From: Chris Nauroth [cnaur...@hortonworks.com]
Sent: Friday, May 01, 2015 4:32 AM
To: mapreduce-...@hadoop.apache.org; common-dev@hadoop.apache.org;
yarn-...@hadoop.apache.org; hdfs-...@hadoop.apache.org
Subject: Re: Planning Hadoop 2.6.1 release

Thank you, Arpit.  In addition, I suggest we include the following:

HADOOP-11333. Fix deadlock in DomainSocketWatcher when the notification
pipe is full
HADOOP-11604. Prevent ConcurrentModificationException while closing
domain
sockets during shutdown of DomainSocketWatcher thread.
HADOOP-11648. Set DomainSocketWatcher thread name explicitly
HADOOP-11802. DomainSocketWatcher thread terminates sometimes after
there
is an I/O error during requestShortCircuitShm

HADOOP-11604 and 11648 are not critical by themselves, but they are
pre-requisites to getting a clean cherry-pick of 11802, which we believe
finally fixes the root cause 

Re: Planning Hadoop 2.6.1 release

[Vinod Kumar Vavilapalli]

 - I also have a bunch of patches that I’d like to include, will update them 
right away.

I’ve just finished this. The latest 2.6.1-candidate list is up at 64 JIRAs.

Others, please look at the list and post anything else you’d like to get 
included for 2.6.1.

Thanks
+Vinod


On Jul 15, 2015, at 6:24 PM, Vinod Kumar Vavilapalli 
mailto:vino...@hortonworks.com>> wrote:

Alright, I’d like to make progress while the issue is hot.

I created a label to discuss on the candidate list of patches: 
https://issues.apache.org/jira/issues/?jql=labels%20%3D%202.6.1-candidate

Next steps, I’ll do the following
 - Review 2.7 and 2.8 blocker/critical tickets and see what makes sense for 
2.6.1 and add as candidates
 - I haven’t reviewed the current list yet, the seed list is from this email 
thread. Will review them.
 - I also have a bunch of patches that I’d like to include, will update them 
right away.

Others, please look at the current list and let me know what else you’d like to 
include.

I’d like to keep this ‘candidate-collection’ cycle’ for a max of a week and 
then start the release process. @Akira, let’s sync up offline on how to take 
this forward in terms of the release process.

Thanks
+Vinod

Re: Planning Hadoop 2.6.1 release

[Akira AJISAKA]

Thanks Vinod for updating the candidate list.
I'd like to include the followings 12 JIRAs:

* YARN-3641
* YARN-3585
* YARN-2910
* HDFS-8431
* HDFS-7830
* HDFS-7763
* HDFS-7742
* HDFS-7235
* HDFS-7225
* MAPREDUCE-6324
* HADOOP-11934
* HADOOP-11491

Thanks,
Akira

On 7/18/15 11:13, Vinod Kumar Vavilapalli wrote:

  - I also have a bunch of patches that I’d like to include, will update them 
right away.

I’ve just finished this. The latest 2.6.1-candidate list is up at 64 JIRAs.

Others, please look at the list and post anything else you’d like to get 
included for 2.6.1.

Thanks
+Vinod


On Jul 15, 2015, at 6:24 PM, Vinod Kumar Vavilapalli 
mailto:vino...@hortonworks.com>> wrote:

Alright, I’d like to make progress while the issue is hot.

I created a label to discuss on the candidate list of patches: 
https://issues.apache.org/jira/issues/?jql=labels%20%3D%202.6.1-candidate

Next steps, I’ll do the following
  - Review 2.7 and 2.8 blocker/critical tickets and see what makes sense for 
2.6.1 and add as candidates
  - I haven’t reviewed the current list yet, the seed list is from this email 
thread. Will review them.
  - I also have a bunch of patches that I’d like to include, will update them 
right away.

Others, please look at the current list and let me know what else you’d like to 
include.

I’d like to keep this ‘candidate-collection’ cycle’ for a max of a week and 
then start the release process. @Akira, let’s sync up offline on how to take 
this forward in terms of the release process.

Thanks
+Vinod

Re: Planning Hadoop 2.6.1 release

[J. Rottinghuis]

Hi Vinod,

We've gone through the various lists of upstream jiras and wanted to
provide our take on what we'd like to see in 2.6.1 release.
In addition, we have ~58 jiras (some a group of work such as DN maintenance
state) that we already have in production
in a pre-2.6 release. I've gone through and selected those that seem
relevant for 2.6.1.

Jiras already marked with 2.6.1-candidate (we agree should be part of
2.6.1):
HADOOP-11802
HDFS-7489
HDFS-7533
HDFS-7587
HDFS-7596
HDFS-7707
HDFS-7742
HDFS-8046
HDFS-8072
HDFS-8127
MAPREDUCE-6303
MAPREDUCE-6361
YARN-2414
YARN-2905
YARN-3369
YARN-3485
YARN-3585
YARN-3641
HDFS-7443
HDFS-7575


Jiras not yet marked with 2.6.1-candidate that we'd like to see in 2.6.1:
HDFS-7213
HDFS-7446
HDFS-7704
HDFS-7788
HDFS-7884
HDFS-7894
HDFS-7916
HDFS-7929
HDFS-7930
HDFS-7980
HDFS-8245
HDFS-8270
HDFS-8404
HDFS-8480
HDFS-8486
MAPREDUCE-6238
YARN-2856
MAPREDUCE-6300
YARN-2952
YARN-2997
YARN-3094
YARN-3222
YARN-3238
YARN-3464
YARN-3526
YARN-3850

Jiras that we're already running in production pre-2.6 that we'd like in
2.6.1-candidate:

HADOOP-11812 (committed in 2.8)
MAPREDUCE-5649 (committed in 2.8)
YARN-3231 (committed in 2.7)
MAPREDUCE-6166 (committed in 2.7)
HADOOP-11295 (committed in 2.7)
HDFS-7314 (committed in 2.8)
HDFS-7182 (committed in 2.7)
MAPREDUCE-5465 (committed in 2.8)

Lower priority jira that we're already running in production in pre-2.6
that are more a nice to have:
HDFS-7281 (committed in 3.0)
YARN-3176 (not yet committed in OSS)

Note that these lists are a culmination of lots of work by many people in
our team, so credit goes to all of them.
Any possible typo or mistake in copying jira ids are entirely to be blamed
on me.

Thanks,

Joep


On Tue, Jul 21, 2015 at 2:15 AM, Akira AJISAKA 
wrote:

> Thanks Vinod for updating the candidate list.
> I'd like to include the followings 12 JIRAs:
>
> * YARN-3641
> * YARN-3585
> * YARN-2910
> * HDFS-8431
> * HDFS-7830
> * HDFS-7763
> * HDFS-7742
> * HDFS-7235
> * HDFS-7225
> * MAPREDUCE-6324
> * HADOOP-11934
> * HADOOP-11491
>
> Thanks,
> Akira
>
> On 7/18/15 11:13, Vinod Kumar Vavilapalli wrote:
>
>>   - I also have a bunch of patches that I’d like to include, will update
>> them right away.
>>
>> I’ve just finished this. The latest 2.6.1-candidate list is up at 64
>> JIRAs.
>>
>> Others, please look at the list and post anything else you’d like to get
>> included for 2.6.1.
>>
>> Thanks
>> +Vinod
>>
>>
>> On Jul 15, 2015, at 6:24 PM, Vinod Kumar Vavilapalli <
>> vino...@hortonworks.com> wrote:
>>
>> Alright, I’d like to make progress while the issue is hot.
>>
>> I created a label to discuss on the candidate list of patches:
>> https://issues.apache.org/jira/issues/?jql=labels%20%3D%202.6.1-candidate
>> 
>>
>> Next steps, I’ll do the following
>>   - Review 2.7 and 2.8 blocker/critical tickets and see what makes sense
>> for 2.6.1 and add as candidates
>>   - I haven’t reviewed the current list yet, the seed list is from this
>> email thread. Will review them.
>>   - I also have a bunch of patches that I’d like to include, will update
>> them right away.
>>
>> Others, please look at the current list and let me know what else you’d
>> like to include.
>>
>> I’d like to keep this ‘candidate-collection’ cycle’ for a max of a week
>> and then start the release process. @Akira, let’s sync up offline on how to
>> take this forward in terms of the release process.
>>
>> Thanks
>> +Vinod
>>
>>
>>
>

Re: Planning Hadoop 2.6.1 release

[Vinod Kumar Vavilapalli]

I’ve added them all to the 2.6.1-candidate list. I included everything even 
though some of them are major tickets. The list is getting large, we will have 
to cut these down once we get down to the next phase of figuring out what to 
include and what not to.

Thanks
+Vinod

> On Jul 21, 2015, at 2:15 AM, Akira AJISAKA  wrote:
> 
> Thanks Vinod for updating the candidate list.
> I'd like to include the followings 12 JIRAs:
> 
> * YARN-3641
> * YARN-3585
> * YARN-2910
> * HDFS-8431
> * HDFS-7830
> * HDFS-7763
> * HDFS-7742
> * HDFS-7235
> * HDFS-7225
> * MAPREDUCE-6324
> * HADOOP-11934
> * HADOOP-11491
> 
> Thanks,
> Akira
> 
> On 7/18/15 11:13, Vinod Kumar Vavilapalli wrote:
>>  - I also have a bunch of patches that I’d like to include, will update them 
>> right away.
>> 
>> I’ve just finished this. The latest 2.6.1-candidate list is up at 64 JIRAs.
>> 
>> Others, please look at the list and post anything else you’d like to get 
>> included for 2.6.1.
>> 
>> Thanks
>> +Vinod
>> 
>> 
>> On Jul 15, 2015, at 6:24 PM, Vinod Kumar Vavilapalli 
>> mailto:vino...@hortonworks.com>> wrote:
>> 
>> Alright, I’d like to make progress while the issue is hot.
>> 
>> I created a label to discuss on the candidate list of patches: 
>> https://issues.apache.org/jira/issues/?jql=labels%20%3D%202.6.1-candidate
>> 
>> Next steps, I’ll do the following
>>  - Review 2.7 and 2.8 blocker/critical tickets and see what makes sense for 
>> 2.6.1 and add as candidates
>>  - I haven’t reviewed the current list yet, the seed list is from this email 
>> thread. Will review them.
>>  - I also have a bunch of patches that I’d like to include, will update them 
>> right away.
>> 
>> Others, please look at the current list and let me know what else you’d like 
>> to include.
>> 
>> I’d like to keep this ‘candidate-collection’ cycle’ for a max of a week and 
>> then start the release process. @Akira, let’s sync up offline on how to take 
>> this forward in terms of the release process.
>> 
>> Thanks
>> +Vinod
>> 
>> 
> 
> 

Re: Planning Hadoop 2.6.1 release

[Sangjin Lee]

Out of the JIRAs we proposed, please remove HDFS-7916. I don't think it
applies to 2.6.

Thanks,
Sangjin

On Wed, Jul 22, 2015 at 4:02 PM, Vinod Kumar Vavilapalli <
vino...@hortonworks.com> wrote:

> I’ve added them all to the 2.6.1-candidate list. I included everything
> even though some of them are major tickets. The list is getting large, we
> will have to cut these down once we get down to the next phase of figuring
> out what to include and what not to.
>
> Thanks
> +Vinod
>
> > On Jul 21, 2015, at 2:15 AM, Akira AJISAKA 
> wrote:
> >
> > Thanks Vinod for updating the candidate list.
> > I'd like to include the followings 12 JIRAs:
> >
> > * YARN-3641
> > * YARN-3585
> > * YARN-2910
> > * HDFS-8431
> > * HDFS-7830
> > * HDFS-7763
> > * HDFS-7742
> > * HDFS-7235
> > * HDFS-7225
> > * MAPREDUCE-6324
> > * HADOOP-11934
> > * HADOOP-11491
> >
> > Thanks,
> > Akira
> >
> > On 7/18/15 11:13, Vinod Kumar Vavilapalli wrote:
> >>  - I also have a bunch of patches that I’d like to include, will update
> them right away.
> >>
> >> I’ve just finished this. The latest 2.6.1-candidate list is up at 64
> JIRAs.
> >>
> >> Others, please look at the list and post anything else you’d like to
> get included for 2.6.1.
> >>
> >> Thanks
> >> +Vinod
> >>
> >>
> >> On Jul 15, 2015, at 6:24 PM, Vinod Kumar Vavilapalli <
> vino...@hortonworks.com> wrote:
> >>
> >> Alright, I’d like to make progress while the issue is hot.
> >>
> >> I created a label to discuss on the candidate list of patches:
> https://issues.apache.org/jira/issues/?jql=labels%20%3D%202.6.1-candidate<
> https://issues.apache.org/jira/issues/?jql=labels%20=%202.6.1-candidate>
> >>
> >> Next steps, I’ll do the following
> >>  - Review 2.7 and 2.8 blocker/critical tickets and see what makes sense
> for 2.6.1 and add as candidates
> >>  - I haven’t reviewed the current list yet, the seed list is from this
> email thread. Will review them.
> >>  - I also have a bunch of patches that I’d like to include, will update
> them right away.
> >>
> >> Others, please look at the current list and let me know what else you’d
> like to include.
> >>
> >> I’d like to keep this ‘candidate-collection’ cycle’ for a max of a week
> and then start the release process. @Akira, let’s sync up offline on how to
> take this forward in terms of the release process.
> >>
> >> Thanks
> >> +Vinod
> >>
> >>
> >
> >
>
>

Re: Planning Hadoop 2.6.1 release

[Akira AJISAKA]

Thanks Joep and your team members for creating the list. I really 
appreciate your work. I looked your 'not yet marked with 
2.6.1-candidate' list and categorized them.


1) Now marked as 2.6.1-candidate and I agree with you to keep it marked.

* HDFS-7213
* HDFS-7788
* HDFS-7884
* HDFS-7930
* YARN-2856
* YARN-3222
* YARN-3238
* YARN-3464
* YARN-3526
* YARN-3850

Thanks Sangjin for creating patches for branch-2.6.

2) Not yet marked as 2.6.1-candidate that I'd like to see in 2.6.1

* HDFS-7182
* HDFS-7314
* HDFS-7704
* HDFS-7894
* HDFS-7929 and HDFS-8480 (they are related)
* HDFS-7980
* HDFS-8245
* HDFS-8270
* HDFS-8404
* HDFS-8486
* MAPREDUCE-5465
* MAPREDUCE-5649
* MAPREDUCE-6166
* MAPREDUCE-6238
* MAPREDUCE-6300
* YARN-2952
* YARN-2997
* YARN-3094
* YARN-3176 (to be fixed soon, I think)
* YARN-3231
* HADOOP-11295
* HADOOP-11812

3) Not yet marked as 2.6.1-candidate. I'd like to drop

* HDFS-7281 (incompatible change)
* HDFS-7446 (this looks to be an improvement)
* HDFS-7916 (cannot apply to branch-2.6 as Sangjin mentioned)

Hi Vinod, could you mark the issues in 2) as 2.6.1-candidate?

I'd like to freeze the candidate list in about 7 days and start 
backporting them. Do you have any thoughts?


Regards,
Akira

On 7/25/15 10:32, Sangjin Lee wrote:

Out of the JIRAs we proposed, please remove HDFS-7916. I don't think it
applies to 2.6.

Thanks,
Sangjin

On Wed, Jul 22, 2015 at 4:02 PM, Vinod Kumar Vavilapalli <
vino...@hortonworks.com> wrote:


I’ve added them all to the 2.6.1-candidate list. I included everything
even though some of them are major tickets. The list is getting large, we
will have to cut these down once we get down to the next phase of figuring
out what to include and what not to.

Thanks
+Vinod


On Jul 21, 2015, at 2:15 AM, Akira AJISAKA 

wrote:


Thanks Vinod for updating the candidate list.
I'd like to include the followings 12 JIRAs:

* YARN-3641
* YARN-3585
* YARN-2910
* HDFS-8431
* HDFS-7830
* HDFS-7763
* HDFS-7742
* HDFS-7235
* HDFS-7225
* MAPREDUCE-6324
* HADOOP-11934
* HADOOP-11491

Thanks,
Akira

On 7/18/15 11:13, Vinod Kumar Vavilapalli wrote:

  - I also have a bunch of patches that I’d like to include, will update

them right away.


I’ve just finished this. The latest 2.6.1-candidate list is up at 64

JIRAs.


Others, please look at the list and post anything else you’d like to

get included for 2.6.1.


Thanks
+Vinod


On Jul 15, 2015, at 6:24 PM, Vinod Kumar Vavilapalli <

vino...@hortonworks.com> wrote:


Alright, I’d like to make progress while the issue is hot.

I created a label to discuss on the candidate list of patches:

https://issues.apache.org/jira/issues/?jql=labels%20%3D%202.6.1-candidate<
https://issues.apache.org/jira/issues/?jql=labels%20=%202.6.1-candidate>


Next steps, I’ll do the following
  - Review 2.7 and 2.8 blocker/critical tickets and see what makes sense

for 2.6.1 and add as candidates

  - I haven’t reviewed the current list yet, the seed list is from this

email thread. Will review them.

  - I also have a bunch of patches that I’d like to include, will update

them right away.


Others, please look at the current list and let me know what else you’d

like to include.


I’d like to keep this ‘candidate-collection’ cycle’ for a max of a week

and then start the release process. @Akira, let’s sync up offline on how to
take this forward in terms of the release process.


Thanks
+Vinod

Re: Planning Hadoop 2.6.1 release

[Sangjin Lee]

Thanks Akira.

I'd like to make one small correction. If we're getting HDFS-7704, then we
should also get HDFS-7916. My earlier comment was assuming HDFS-7704 was
not included in the list. But if is (and I think it should), then we should
also get HDFS-7916 as it addresses an important issue related to HDFS-7704.
Hope it makes it clear.

Sangjin

On Fri, Jul 31, 2015 at 10:01 AM, Akira AJISAKA 
wrote:

> Thanks Joep and your team members for creating the list. I really
> appreciate your work. I looked your 'not yet marked with 2.6.1-candidate'
> list and categorized them.
>
> 1) Now marked as 2.6.1-candidate and I agree with you to keep it marked.
>
> * HDFS-7213
> * HDFS-7788
> * HDFS-7884
> * HDFS-7930
> * YARN-2856
> * YARN-3222
> * YARN-3238
> * YARN-3464
> * YARN-3526
> * YARN-3850
>
> Thanks Sangjin for creating patches for branch-2.6.
>
> 2) Not yet marked as 2.6.1-candidate that I'd like to see in 2.6.1
>
> * HDFS-7182
> * HDFS-7314
> * HDFS-7704
> * HDFS-7894
> * HDFS-7929 and HDFS-8480 (they are related)
> * HDFS-7980
> * HDFS-8245
> * HDFS-8270
> * HDFS-8404
> * HDFS-8486
> * MAPREDUCE-5465
> * MAPREDUCE-5649
> * MAPREDUCE-6166
> * MAPREDUCE-6238
> * MAPREDUCE-6300
> * YARN-2952
> * YARN-2997
> * YARN-3094
> * YARN-3176 (to be fixed soon, I think)
> * YARN-3231
> * HADOOP-11295
> * HADOOP-11812
>
> 3) Not yet marked as 2.6.1-candidate. I'd like to drop
>
> * HDFS-7281 (incompatible change)
> * HDFS-7446 (this looks to be an improvement)
> * HDFS-7916 (cannot apply to branch-2.6 as Sangjin mentioned)
>
> Hi Vinod, could you mark the issues in 2) as 2.6.1-candidate?
>
> I'd like to freeze the candidate list in about 7 days and start
> backporting them. Do you have any thoughts?
>
> Regards,
> Akira
>
>
> On 7/25/15 10:32, Sangjin Lee wrote:
>
>> Out of the JIRAs we proposed, please remove HDFS-7916. I don't think it
>> applies to 2.6.
>>
>> Thanks,
>> Sangjin
>>
>> On Wed, Jul 22, 2015 at 4:02 PM, Vinod Kumar Vavilapalli <
>> vino...@hortonworks.com> wrote:
>>
>> I’ve added them all to the 2.6.1-candidate list. I included everything
>>> even though some of them are major tickets. The list is getting large, we
>>> will have to cut these down once we get down to the next phase of
>>> figuring
>>> out what to include and what not to.
>>>
>>> Thanks
>>> +Vinod
>>>
>>> On Jul 21, 2015, at 2:15 AM, Akira AJISAKA 

>>> wrote:
>>>

 Thanks Vinod for updating the candidate list.
 I'd like to include the followings 12 JIRAs:

 * YARN-3641
 * YARN-3585
 * YARN-2910
 * HDFS-8431
 * HDFS-7830
 * HDFS-7763
 * HDFS-7742
 * HDFS-7235
 * HDFS-7225
 * MAPREDUCE-6324
 * HADOOP-11934
 * HADOOP-11491

 Thanks,
 Akira

 On 7/18/15 11:13, Vinod Kumar Vavilapalli wrote:

>   - I also have a bunch of patches that I’d like to include, will
> update
>
 them right away.
>>>

> I’ve just finished this. The latest 2.6.1-candidate list is up at 64
>
 JIRAs.
>>>

> Others, please look at the list and post anything else you’d like to
>
 get included for 2.6.1.
>>>

> Thanks
> +Vinod
>
>
> On Jul 15, 2015, at 6:24 PM, Vinod Kumar Vavilapalli <
>
 vino...@hortonworks.com> wrote:
>>>

> Alright, I’d like to make progress while the issue is hot.
>
> I created a label to discuss on the candidate list of patches:
>

>>> https://issues.apache.org/jira/issues/?jql=labels%20%3D%202.6.1-candidate
>>> <
>>> https://issues.apache.org/jira/issues/?jql=labels%20=%202.6.1-candidate>
>>>

> Next steps, I’ll do the following
>   - Review 2.7 and 2.8 blocker/critical tickets and see what makes
> sense
>
 for 2.6.1 and add as candidates
>>>
   - I haven’t reviewed the current list yet, the seed list is from this
>
 email thread. Will review them.
>>>
   - I also have a bunch of patches that I’d like to include, will update
>
 them right away.
>>>

> Others, please look at the current list and let me know what else you’d
>
 like to include.
>>>

> I’d like to keep this ‘candidate-collection’ cycle’ for a max of a week
>
 and then start the release process. @Akira, let’s sync up offline on
>>> how to
>>> take this forward in terms of the release process.
>>>

> Thanks
> +Vinod
>
>
>


>>>
>>>
>>
>

Re: Planning Hadoop 2.6.1 release

[Sangjin Lee]

Just for the completeness, the following JIRAs have already been committed
to branch-2.6 and thus will be part of 2.6.1:

HADOOP-11307
YARN-2375
HDFS-7425
HDFS-4882
HDFS-7489
HDFS-7503
HDFS-7443
HDFS-3443
HADOOP-11466
HDFS-7733
MAPREDUCE-6237
YARN-3251
HDFS-6153 reverted


On Fri, Jul 31, 2015 at 10:07 AM, Sangjin Lee  wrote:

> Thanks Akira.
>
> I'd like to make one small correction. If we're getting HDFS-7704, then we
> should also get HDFS-7916. My earlier comment was assuming HDFS-7704 was
> not included in the list. But if is (and I think it should), then we should
> also get HDFS-7916 as it addresses an important issue related to HDFS-7704.
> Hope it makes it clear.
>
> Sangjin
>
> On Fri, Jul 31, 2015 at 10:01 AM, Akira AJISAKA <
> ajisa...@oss.nttdata.co.jp> wrote:
>
>> Thanks Joep and your team members for creating the list. I really
>> appreciate your work. I looked your 'not yet marked with 2.6.1-candidate'
>> list and categorized them.
>>
>> 1) Now marked as 2.6.1-candidate and I agree with you to keep it marked.
>>
>> * HDFS-7213
>> * HDFS-7788
>> * HDFS-7884
>> * HDFS-7930
>> * YARN-2856
>> * YARN-3222
>> * YARN-3238
>> * YARN-3464
>> * YARN-3526
>> * YARN-3850
>>
>> Thanks Sangjin for creating patches for branch-2.6.
>>
>> 2) Not yet marked as 2.6.1-candidate that I'd like to see in 2.6.1
>>
>> * HDFS-7182
>> * HDFS-7314
>> * HDFS-7704
>> * HDFS-7894
>> * HDFS-7929 and HDFS-8480 (they are related)
>> * HDFS-7980
>> * HDFS-8245
>> * HDFS-8270
>> * HDFS-8404
>> * HDFS-8486
>> * MAPREDUCE-5465
>> * MAPREDUCE-5649
>> * MAPREDUCE-6166
>> * MAPREDUCE-6238
>> * MAPREDUCE-6300
>> * YARN-2952
>> * YARN-2997
>> * YARN-3094
>> * YARN-3176 (to be fixed soon, I think)
>> * YARN-3231
>> * HADOOP-11295
>> * HADOOP-11812
>>
>> 3) Not yet marked as 2.6.1-candidate. I'd like to drop
>>
>> * HDFS-7281 (incompatible change)
>> * HDFS-7446 (this looks to be an improvement)
>> * HDFS-7916 (cannot apply to branch-2.6 as Sangjin mentioned)
>>
>> Hi Vinod, could you mark the issues in 2) as 2.6.1-candidate?
>>
>> I'd like to freeze the candidate list in about 7 days and start
>> backporting them. Do you have any thoughts?
>>
>> Regards,
>> Akira
>>
>>
>> On 7/25/15 10:32, Sangjin Lee wrote:
>>
>>> Out of the JIRAs we proposed, please remove HDFS-7916. I don't think it
>>> applies to 2.6.
>>>
>>> Thanks,
>>> Sangjin
>>>
>>> On Wed, Jul 22, 2015 at 4:02 PM, Vinod Kumar Vavilapalli <
>>> vino...@hortonworks.com> wrote:
>>>
>>> I’ve added them all to the 2.6.1-candidate list. I included everything
 even though some of them are major tickets. The list is getting large,
 we
 will have to cut these down once we get down to the next phase of
 figuring
 out what to include and what not to.

 Thanks
 +Vinod

 On Jul 21, 2015, at 2:15 AM, Akira AJISAKA 
>
 wrote:

>
> Thanks Vinod for updating the candidate list.
> I'd like to include the followings 12 JIRAs:
>
> * YARN-3641
> * YARN-3585
> * YARN-2910
> * HDFS-8431
> * HDFS-7830
> * HDFS-7763
> * HDFS-7742
> * HDFS-7235
> * HDFS-7225
> * MAPREDUCE-6324
> * HADOOP-11934
> * HADOOP-11491
>
> Thanks,
> Akira
>
> On 7/18/15 11:13, Vinod Kumar Vavilapalli wrote:
>
>>   - I also have a bunch of patches that I’d like to include, will
>> update
>>
> them right away.

>
>> I’ve just finished this. The latest 2.6.1-candidate list is up at 64
>>
> JIRAs.

>
>> Others, please look at the list and post anything else you’d like to
>>
> get included for 2.6.1.

>
>> Thanks
>> +Vinod
>>
>>
>> On Jul 15, 2015, at 6:24 PM, Vinod Kumar Vavilapalli <
>>
> vino...@hortonworks.com> wrote:

>
>> Alright, I’d like to make progress while the issue is hot.
>>
>> I created a label to discuss on the candidate list of patches:
>>
>
 https://issues.apache.org/jira/issues/?jql=labels%20%3D%202.6.1-candidate
 <
 https://issues.apache.org/jira/issues/?jql=labels%20=%202.6.1-candidate
 >

>
>> Next steps, I’ll do the following
>>   - Review 2.7 and 2.8 blocker/critical tickets and see what makes
>> sense
>>
> for 2.6.1 and add as candidates

>   - I haven’t reviewed the current list yet, the seed list is from this
>>
> email thread. Will review them.

>   - I also have a bunch of patches that I’d like to include, will
>> update
>>
> them right away.

>
>> Others, please look at the current list and let me know what else
>> you’d
>>
> like to include.

>
>> I’d like to keep this ‘candidate-collection’ cycle’ for a max of a
>> week
>>
> and then start the release process. @Akira, let’s sync up offline on
 how to
 take this forward in terms of the release process.

>
>> Thanks
>>>

Re: Planning Hadoop 2.6.1 release

[Vinod Kumar Vavilapalli]

Tx for the list, Joep!

> Jiras not yet marked with 2.6.1-candidate that we'd like to see in 2.6.1:

A whole bunch of them already are marked so, presumably from my list. I’ve 
added the remaining for discussion.

> HDFS-7281 (committed in 3.0)

This is an incompatible change, so not adding it.


> YARN-3176 (not yet committed in OSS)

We should let it naturally go into 2.6.1 and 2.7.2 depending on when it goes 
in. We shouldn't block the release for this till it gets reviewed/committed in 
case the later can’t come in time.

Thanks
+Vinod

Re: Planning Hadoop 2.6.1 release

[Vinod Kumar Vavilapalli]

Makes sense, it was caused by HDFS-7704 which got into 2.7.0 only and is not 
part of the candidate list. Removed HDFS-7916 from the list.

Thanks
+Vinod

> On Jul 24, 2015, at 6:32 PM, Sangjin Lee  wrote:
> 
> Out of the JIRAs we proposed, please remove HDFS-7916. I don't think it
> applies to 2.6.
> 
> Thanks,
> Sangjin

Re: Planning Hadoop 2.6.1 release

[Sangjin Lee]

See my later update in the thread. HDFS-7704 is in the list.

Thanks,
Sangjin

On Mon, Aug 3, 2015 at 1:19 PM, Vinod Kumar Vavilapalli <
vino...@hortonworks.com> wrote:

> Makes sense, it was caused by HDFS-7704 which got into 2.7.0 only and is
> not part of the candidate list. Removed HDFS-7916 from the list.
>
> Thanks
> +Vinod
>
> > On Jul 24, 2015, at 6:32 PM, Sangjin Lee  wrote:
> >
> > Out of the JIRAs we proposed, please remove HDFS-7916. I don't think it
> > applies to 2.6.
> >
> > Thanks,
> > Sangjin
>
>

Re: Planning Hadoop 2.6.1 release

[Wangda Tan]

Can we add following two fixes to 2.6.1?

https://issues.apache.org/jira/browse/YARN-2922 and
https://issues.apache.org/jira/browse/YARN-3487.

They're not fatal issue, but they can cause lots of issue in a large
cluster.

Thanks,
Wangda


On Mon, Aug 3, 2015 at 1:21 PM, Sangjin Lee  wrote:

> See my later update in the thread. HDFS-7704 is in the list.
>
> Thanks,
> Sangjin
>
> On Mon, Aug 3, 2015 at 1:19 PM, Vinod Kumar Vavilapalli <
> vino...@hortonworks.com> wrote:
>
> > Makes sense, it was caused by HDFS-7704 which got into 2.7.0 only and is
> > not part of the candidate list. Removed HDFS-7916 from the list.
> >
> > Thanks
> > +Vinod
> >
> > > On Jul 24, 2015, at 6:32 PM, Sangjin Lee  wrote:
> > >
> > > Out of the JIRAs we proposed, please remove HDFS-7916. I don't think it
> > > applies to 2.6.
> > >
> > > Thanks,
> > > Sangjin
> >
> >
>

Re: Planning Hadoop 2.6.1 release

[Zhijie Shen]

+YARN-3011: without which, NM is likely to crash, and cannot come back with 
recovery enabled.

Thanks,
Zhijie

From: Wangda Tan 
Sent: Wednesday, August 05, 2015 10:42 AM
To: yarn-...@hadoop.apache.org
Cc: mapreduce-...@hadoop.apache.org; common-dev@hadoop.apache.org; 
hdfs-...@hadoop.apache.org
Subject: Re: Planning Hadoop 2.6.1 release

Can we add following two fixes to 2.6.1?

https://issues.apache.org/jira/browse/YARN-2922 and
https://issues.apache.org/jira/browse/YARN-3487.

They're not fatal issue, but they can cause lots of issue in a large
cluster.

Thanks,
Wangda


On Mon, Aug 3, 2015 at 1:21 PM, Sangjin Lee  wrote:

> See my later update in the thread. HDFS-7704 is in the list.
>
> Thanks,
> Sangjin
>
> On Mon, Aug 3, 2015 at 1:19 PM, Vinod Kumar Vavilapalli <
> vino...@hortonworks.com> wrote:
>
> > Makes sense, it was caused by HDFS-7704 which got into 2.7.0 only and is
> > not part of the candidate list. Removed HDFS-7916 from the list.
> >
> > Thanks
> > +Vinod
> >
> > > On Jul 24, 2015, at 6:32 PM, Sangjin Lee  wrote:
> > >
> > > Out of the JIRAs we proposed, please remove HDFS-7916. I don't think it
> > > applies to 2.6.
> > >
> > > Thanks,
> > > Sangjin
> >
> >
>

Re: Planning Hadoop 2.6.1 release

[Rich Haase]

+1 to add those fixes.


Rich Haase | Sr. Software Engineer | Pandora
m 303.887.1146 | rha...@pandora.com




On 8/5/15, 11:42 AM, "Wangda Tan"  wrote:

>Can we add following two fixes to 2.6.1?
>
>https://issues.apache.org/jira/browse/YARN-2922 and
>https://issues.apache.org/jira/browse/YARN-3487.
>
>They're not fatal issue, but they can cause lots of issue in a large
>cluster.
>
>Thanks,
>Wangda
>
>
>On Mon, Aug 3, 2015 at 1:21 PM, Sangjin Lee  wrote:
>
>> See my later update in the thread. HDFS-7704 is in the list.
>>
>> Thanks,
>> Sangjin
>>
>> On Mon, Aug 3, 2015 at 1:19 PM, Vinod Kumar Vavilapalli <
>> vino...@hortonworks.com> wrote:
>>
>> > Makes sense, it was caused by HDFS-7704 which got into 2.7.0 only and
>>is
>> > not part of the candidate list. Removed HDFS-7916 from the list.
>> >
>> > Thanks
>> > +Vinod
>> >
>> > > On Jul 24, 2015, at 6:32 PM, Sangjin Lee  wrote:
>> > >
>> > > Out of the JIRAs we proposed, please remove HDFS-7916. I don't
>>think it
>> > > applies to 2.6.
>> > >
>> > > Thanks,
>> > > Sangjin
>> >
>> >
>>

Re: Planning Hadoop 2.6.1 release

[Allan Wilson]

Another +1 to add those fixes.

YARN-3487 bug can grind a large cluster to a halt repeatedly



-Allan


Allan Wilson | Sr. Software Engineer | Pandora
m 919.841.2449 |  awils...@pandora.com





On 8/5/15, 1:52 PM, "Rich Haase"  wrote:

>+1 to add those fixes.
>
>
>Rich Haase | Sr. Software Engineer | Pandora
>m 303.887.1146 | rha...@pandora.com
>
>
>
>
>On 8/5/15, 11:42 AM, "Wangda Tan"  wrote:
>
>>Can we add following two fixes to 2.6.1?
>>
>>https://issues.apache.org/jira/browse/YARN-2922 and
>>https://issues.apache.org/jira/browse/YARN-3487.
>>
>>They're not fatal issue, but they can cause lots of issue in a large
>>cluster.
>>
>>Thanks,
>>Wangda
>>
>>
>>On Mon, Aug 3, 2015 at 1:21 PM, Sangjin Lee  wrote:
>>
>>> See my later update in the thread. HDFS-7704 is in the list.
>>>
>>> Thanks,
>>> Sangjin
>>>
>>> On Mon, Aug 3, 2015 at 1:19 PM, Vinod Kumar Vavilapalli <
>>> vino...@hortonworks.com> wrote:
>>>
>>> > Makes sense, it was caused by HDFS-7704 which got into 2.7.0 only and
>>>is
>>> > not part of the candidate list. Removed HDFS-7916 from the list.
>>> >
>>> > Thanks
>>> > +Vinod
>>> >
>>> > > On Jul 24, 2015, at 6:32 PM, Sangjin Lee  wrote:
>>> > >
>>> > > Out of the JIRAs we proposed, please remove HDFS-7916. I don't
>>>think it
>>> > > applies to 2.6.
>>> > >
>>> > > Thanks,
>>> > > Sangjin
>>> >
>>> >
>>>
>

Re: Planning Hadoop 2.6.1 release

[Junping Du]

I would like to nominate YARN-3832 as 2.6.1 candidate which is critical and I 
also saw it happened recently on a 2.6.0 cluster. Hope this is not too late.

Thanks,

Junping

From: Rich Haase 
Sent: Thursday, August 06, 2015 1:52 AM
To: mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org
Cc: common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org
Subject: Re: Planning Hadoop 2.6.1 release

+1 to add those fixes.


Rich Haase | Sr. Software Engineer | Pandora
m 303.887.1146 | rha...@pandora.com




On 8/5/15, 11:42 AM, "Wangda Tan"  wrote:

>Can we add following two fixes to 2.6.1?
>
>https://issues.apache.org/jira/browse/YARN-2922 and
>https://issues.apache.org/jira/browse/YARN-3487.
>
>They're not fatal issue, but they can cause lots of issue in a large
>cluster.
>
>Thanks,
>Wangda
>
>
>On Mon, Aug 3, 2015 at 1:21 PM, Sangjin Lee  wrote:
>
>> See my later update in the thread. HDFS-7704 is in the list.
>>
>> Thanks,
>> Sangjin
>>
>> On Mon, Aug 3, 2015 at 1:19 PM, Vinod Kumar Vavilapalli <
>> vino...@hortonworks.com> wrote:
>>
>> > Makes sense, it was caused by HDFS-7704 which got into 2.7.0 only and
>>is
>> > not part of the candidate list. Removed HDFS-7916 from the list.
>> >
>> > Thanks
>> > +Vinod
>> >
>> > > On Jul 24, 2015, at 6:32 PM, Sangjin Lee  wrote:
>> > >
>> > > Out of the JIRAs we proposed, please remove HDFS-7916. I don't
>>think it
>> > > applies to 2.6.
>> > >
>> > > Thanks,
>> > > Sangjin
>> >
>> >
>>

Re: Planning Hadoop 2.6.1 release

[Sean Busbey]

If we haven't frozen yet, HDFS-8850 is a straight forward fix that is
currently only in 2.8+ and would benefit 2.6 and 2.7.

On Wed, Aug 5, 2015 at 2:56 PM, Junping Du  wrote:

> I would like to nominate YARN-3832 as 2.6.1 candidate which is critical
> and I also saw it happened recently on a 2.6.0 cluster. Hope this is not
> too late.
>
> Thanks,
>
> Junping
> 
> From: Rich Haase 
> Sent: Thursday, August 06, 2015 1:52 AM
> To: mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org
> Cc: common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org
> Subject: Re: Planning Hadoop 2.6.1 release
>
> +1 to add those fixes.
>
>
> Rich Haase | Sr. Software Engineer | Pandora
> m 303.887.1146 | rha...@pandora.com
>
>
>
>
> On 8/5/15, 11:42 AM, "Wangda Tan"  wrote:
>
> >Can we add following two fixes to 2.6.1?
> >
> >https://issues.apache.org/jira/browse/YARN-2922 and
> >https://issues.apache.org/jira/browse/YARN-3487.
> >
> >They're not fatal issue, but they can cause lots of issue in a large
> >cluster.
> >
> >Thanks,
> >Wangda
> >
> >
> >On Mon, Aug 3, 2015 at 1:21 PM, Sangjin Lee  wrote:
> >
> >> See my later update in the thread. HDFS-7704 is in the list.
> >>
> >> Thanks,
> >> Sangjin
> >>
> >> On Mon, Aug 3, 2015 at 1:19 PM, Vinod Kumar Vavilapalli <
> >> vino...@hortonworks.com> wrote:
> >>
> >> > Makes sense, it was caused by HDFS-7704 which got into 2.7.0 only and
> >>is
> >> > not part of the candidate list. Removed HDFS-7916 from the list.
> >> >
> >> > Thanks
> >> > +Vinod
> >> >
> >> > > On Jul 24, 2015, at 6:32 PM, Sangjin Lee  wrote:
> >> > >
> >> > > Out of the JIRAs we proposed, please remove HDFS-7916. I don't
> >>think it
> >> > > applies to 2.6.
> >> > >
> >> > > Thanks,
> >> > > Sangjin
> >> >
> >> >
> >>
>
>


-- 
Sean

RE: Planning Hadoop 2.6.1 release

[Rohith Sharma K S]

Can we add following fixes to 2.6.1?

YARN-3733 
YARN-2865
YARN-3990
YARN-2894


Thanks & Regards
Rohith Sharma K S


-Original Message-
From: Allan Wilson [mailto:awils...@pandora.com] 
Sent: 05 August 2015 23:25
To: common-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org
Cc: hdfs-...@hadoop.apache.org
Subject: Re: Planning Hadoop 2.6.1 release

Another +1 to add those fixes.

YARN-3487 bug can grind a large cluster to a halt repeatedly



-Allan


Allan Wilson | Sr. Software Engineer | Pandora m 919.841.2449 |  
awils...@pandora.com





On 8/5/15, 1:52 PM, "Rich Haase"  wrote:

>+1 to add those fixes.
>
>
>Rich Haase | Sr. Software Engineer | Pandora m 303.887.1146 | 
>rha...@pandora.com
>
>
>
>
>On 8/5/15, 11:42 AM, "Wangda Tan"  wrote:
>
>>Can we add following two fixes to 2.6.1?
>>
>>https://issues.apache.org/jira/browse/YARN-2922 and 
>>https://issues.apache.org/jira/browse/YARN-3487.
>>
>>They're not fatal issue, but they can cause lots of issue in a large 
>>cluster.
>>
>>Thanks,
>>Wangda
>>
>>
>>On Mon, Aug 3, 2015 at 1:21 PM, Sangjin Lee  wrote:
>>
>>> See my later update in the thread. HDFS-7704 is in the list.
>>>
>>> Thanks,
>>> Sangjin
>>>
>>> On Mon, Aug 3, 2015 at 1:19 PM, Vinod Kumar Vavilapalli < 
>>> vino...@hortonworks.com> wrote:
>>>
>>> > Makes sense, it was caused by HDFS-7704 which got into 2.7.0 only 
>>> > and
>>>is
>>> > not part of the candidate list. Removed HDFS-7916 from the list.
>>> >
>>> > Thanks
>>> > +Vinod
>>> >
>>> > > On Jul 24, 2015, at 6:32 PM, Sangjin Lee  wrote:
>>> > >
>>> > > Out of the JIRAs we proposed, please remove HDFS-7916. I don't
>>>think it
>>> > > applies to 2.6.
>>> > >
>>> > > Thanks,
>>> > > Sangjin
>>> >
>>> >
>>>
>

Re: Planning Hadoop 2.6.1 release

[Vinod Kumar Vavilapalli]

Added them both. Tx Wangda!

+Vinod

> On Aug 5, 2015, at 10:42 AM, Wangda Tan  wrote:
> 
> Can we add following two fixes to 2.6.1?
> 
> https://issues.apache.org/jira/browse/YARN-2922 and
> https://issues.apache.org/jira/browse/YARN-3487.
> 
> They're not fatal issue, but they can cause lots of issue in a large
> cluster.
> 
> Thanks,
> Wangda
> 
> 
> On Mon, Aug 3, 2015 at 1:21 PM, Sangjin Lee  wrote:
> 
>> See my later update in the thread. HDFS-7704 is in the list.
>> 
>> Thanks,
>> Sangjin
>> 
>> On Mon, Aug 3, 2015 at 1:19 PM, Vinod Kumar Vavilapalli <
>> vino...@hortonworks.com> wrote:
>> 
>>> Makes sense, it was caused by HDFS-7704 which got into 2.7.0 only and is
>>> not part of the candidate list. Removed HDFS-7916 from the list.
>>> 
>>> Thanks
>>> +Vinod
>>> 
 On Jul 24, 2015, at 6:32 PM, Sangjin Lee  wrote:
 
 Out of the JIRAs we proposed, please remove HDFS-7916. I don't think it
 applies to 2.6.
 
 Thanks,
 Sangjin
>>> 
>>> 
>> 

Re: Planning Hadoop 2.6.1 release

[Vinod Kumar Vavilapalli]

Added this, Tx Zhijie!

+Vinod

> On Aug 5, 2015, at 10:49 AM, Zhijie Shen  wrote:
> 
> +YARN-3011: without which, NM is likely to crash, and cannot come back with 
> recovery enabled.
> 
> Thanks,
> Zhijie
> 
> From: Wangda Tan 
> Sent: Wednesday, August 05, 2015 10:42 AM
> To: yarn-...@hadoop.apache.org
> Cc: mapreduce-...@hadoop.apache.org; common-dev@hadoop.apache.org; 
> hdfs-...@hadoop.apache.org
> Subject: Re: Planning Hadoop 2.6.1 release
> 
> Can we add following two fixes to 2.6.1?
> 
> https://issues.apache.org/jira/browse/YARN-2922 and
> https://issues.apache.org/jira/browse/YARN-3487.
> 
> They're not fatal issue, but they can cause lots of issue in a large
> cluster.
> 
> Thanks,
> Wangda
> 
> 
> On Mon, Aug 3, 2015 at 1:21 PM, Sangjin Lee  wrote:
> 
>> See my later update in the thread. HDFS-7704 is in the list.
>> 
>> Thanks,
>> Sangjin
>> 
>> On Mon, Aug 3, 2015 at 1:19 PM, Vinod Kumar Vavilapalli <
>> vino...@hortonworks.com> wrote:
>> 
>>> Makes sense, it was caused by HDFS-7704 which got into 2.7.0 only and is
>>> not part of the candidate list. Removed HDFS-7916 from the list.
>>> 
>>> Thanks
>>> +Vinod
>>> 
>>>> On Jul 24, 2015, at 6:32 PM, Sangjin Lee  wrote:
>>>> 
>>>> Out of the JIRAs we proposed, please remove HDFS-7916. I don't think it
>>>> applies to 2.6.
>>>> 
>>>> Thanks,
>>>> Sangjin
>>> 
>>> 
>> 
> 

Re: Planning Hadoop 2.6.1 release

[Vinod Kumar Vavilapalli]

Added, tx Junping.

+Vinod

> On Aug 5, 2015, at 12:56 PM, Junping Du  wrote:
> 
> I would like to nominate YARN-3832 as 2.6.1 candidate which is critical and I 
> also saw it happened recently on a 2.6.0 cluster. Hope this is not too late.
> 
> Thanks,
> 
> Junping
> 
> From: Rich Haase 
> Sent: Thursday, August 06, 2015 1:52 AM
> To: mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org
> Cc: common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org
> Subject: Re: Planning Hadoop 2.6.1 release
> 
> +1 to add those fixes.
> 
> 
> Rich Haase | Sr. Software Engineer | Pandora
> m 303.887.1146 | rha...@pandora.com
> 
> 
> 
> 
> On 8/5/15, 11:42 AM, "Wangda Tan"  wrote:
> 
>> Can we add following two fixes to 2.6.1?
>> 
>> https://issues.apache.org/jira/browse/YARN-2922 and
>> https://issues.apache.org/jira/browse/YARN-3487.
>> 
>> They're not fatal issue, but they can cause lots of issue in a large
>> cluster.
>> 
>> Thanks,
>> Wangda
>> 
>> 
>> On Mon, Aug 3, 2015 at 1:21 PM, Sangjin Lee  wrote:
>> 
>>> See my later update in the thread. HDFS-7704 is in the list.
>>> 
>>> Thanks,
>>> Sangjin
>>> 
>>> On Mon, Aug 3, 2015 at 1:19 PM, Vinod Kumar Vavilapalli <
>>> vino...@hortonworks.com> wrote:
>>> 
>>>> Makes sense, it was caused by HDFS-7704 which got into 2.7.0 only and
>>> is
>>>> not part of the candidate list. Removed HDFS-7916 from the list.
>>>> 
>>>> Thanks
>>>> +Vinod
>>>> 
>>>>> On Jul 24, 2015, at 6:32 PM, Sangjin Lee  wrote:
>>>>> 
>>>>> Out of the JIRAs we proposed, please remove HDFS-7916. I don't
>>> think it
>>>>> applies to 2.6.
>>>>> 
>>>>> Thanks,
>>>>> Sangjin
>>>> 
>>>> 
>>> 
> 
> 

Re: Planning Hadoop 2.6.1 release

[Vinod Kumar Vavilapalli]

Added to the list. Can you get it committed to 2.7.2 please?

Thanks
+Vinod

> On Aug 5, 2015, at 2:34 PM, Sean Busbey  wrote:
> 
> If we haven't frozen yet, HDFS-8850 is a straight forward fix that is
> currently only in 2.8+ and would benefit 2.6 and 2.7.
> 
> On Wed, Aug 5, 2015 at 2:56 PM, Junping Du  wrote:
> 
>> I would like to nominate YARN-3832 as 2.6.1 candidate which is critical
>> and I also saw it happened recently on a 2.6.0 cluster. Hope this is not
>> too late.
>> 
>> Thanks,
>> 
>> Junping
>> 
>> From: Rich Haase 
>> Sent: Thursday, August 06, 2015 1:52 AM
>> To: mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org
>> Cc: common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org
>> Subject: Re: Planning Hadoop 2.6.1 release
>> 
>> +1 to add those fixes.
>> 
>> 
>> Rich Haase | Sr. Software Engineer | Pandora
>> m 303.887.1146 | rha...@pandora.com
>> 
>> 
>> 
>> 
>> On 8/5/15, 11:42 AM, "Wangda Tan"  wrote:
>> 
>>> Can we add following two fixes to 2.6.1?
>>> 
>>> https://issues.apache.org/jira/browse/YARN-2922 and
>>> https://issues.apache.org/jira/browse/YARN-3487.
>>> 
>>> They're not fatal issue, but they can cause lots of issue in a large
>>> cluster.
>>> 
>>> Thanks,
>>> Wangda
>>> 
>>> 
>>> On Mon, Aug 3, 2015 at 1:21 PM, Sangjin Lee  wrote:
>>> 
>>>> See my later update in the thread. HDFS-7704 is in the list.
>>>> 
>>>> Thanks,
>>>> Sangjin
>>>> 
>>>> On Mon, Aug 3, 2015 at 1:19 PM, Vinod Kumar Vavilapalli <
>>>> vino...@hortonworks.com> wrote:
>>>> 
>>>>> Makes sense, it was caused by HDFS-7704 which got into 2.7.0 only and
>>>> is
>>>>> not part of the candidate list. Removed HDFS-7916 from the list.
>>>>> 
>>>>> Thanks
>>>>> +Vinod
>>>>> 
>>>>>> On Jul 24, 2015, at 6:32 PM, Sangjin Lee  wrote:
>>>>>> 
>>>>>> Out of the JIRAs we proposed, please remove HDFS-7916. I don't
>>>> think it
>>>>>> applies to 2.6.
>>>>>> 
>>>>>> Thanks,
>>>>>> Sangjin
>>>>> 
>>>>> 
>>>> 
>> 
>> 
> 
> 
> -- 
> Sean

Re: Planning Hadoop 2.6.1 release

[Vinod Kumar Vavilapalli]

Added them all, tx Rohit.

+Vinod

> On Aug 5, 2015, at 9:51 PM, Rohith Sharma K S  
> wrote:
> 
> Can we add following fixes to 2.6.1?
> 
> YARN-3733 
> YARN-2865
> YARN-3990
> YARN-2894
> 
> 
> Thanks & Regards
> Rohith Sharma K S
> 
> 
> -Original Message-
> From: Allan Wilson [mailto:awils...@pandora.com] 
> Sent: 05 August 2015 23:25
> To: common-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org; 
> yarn-...@hadoop.apache.org
> Cc: hdfs-...@hadoop.apache.org
> Subject: Re: Planning Hadoop 2.6.1 release
> 
> Another +1 to add those fixes.
> 
> YARN-3487 bug can grind a large cluster to a halt repeatedly
> 
> 
> 
> -Allan
> 
> 
> Allan Wilson | Sr. Software Engineer | Pandora m 919.841.2449 |  
> awils...@pandora.com
> 
> 
> 
> 
> 
> On 8/5/15, 1:52 PM, "Rich Haase"  wrote:
> 
>> +1 to add those fixes.
>> 
>> 
>> Rich Haase | Sr. Software Engineer | Pandora m 303.887.1146 | 
>> rha...@pandora.com
>> 
>> 
>> 
>> 
>> On 8/5/15, 11:42 AM, "Wangda Tan"  wrote:
>> 
>>> Can we add following two fixes to 2.6.1?
>>> 
>>> https://issues.apache.org/jira/browse/YARN-2922 and 
>>> https://issues.apache.org/jira/browse/YARN-3487.
>>> 
>>> They're not fatal issue, but they can cause lots of issue in a large 
>>> cluster.
>>> 
>>> Thanks,
>>> Wangda
>>> 
>>> 
>>> On Mon, Aug 3, 2015 at 1:21 PM, Sangjin Lee  wrote:
>>> 
>>>> See my later update in the thread. HDFS-7704 is in the list.
>>>> 
>>>> Thanks,
>>>> Sangjin
>>>> 
>>>> On Mon, Aug 3, 2015 at 1:19 PM, Vinod Kumar Vavilapalli < 
>>>> vino...@hortonworks.com> wrote:
>>>> 
>>>>> Makes sense, it was caused by HDFS-7704 which got into 2.7.0 only 
>>>>> and
>>>> is
>>>>> not part of the candidate list. Removed HDFS-7916 from the list.
>>>>> 
>>>>> Thanks
>>>>> +Vinod
>>>>> 
>>>>>> On Jul 24, 2015, at 6:32 PM, Sangjin Lee  wrote:
>>>>>> 
>>>>>> Out of the JIRAs we proposed, please remove HDFS-7916. I don't
>>>> think it
>>>>>> applies to 2.6.
>>>>>> 
>>>>>> Thanks,
>>>>>> Sangjin
>>>>> 
>>>>> 
>>>> 
>> 
> 
> 

Re: Planning Hadoop 2.6.1 release

[Vinod Kumar Vavilapalli]

As of now, I am freezing the list. We have 139 tickets to go through, find the 
right order, get them committed etc.

I’ll start a separate thread for this one-time long release-process of 2.6.1.

Thanks everyone for pitching in.
+Vinod

> On Aug 10, 2015, at 4:13 PM, Vinod Kumar Vavilapalli 
>  wrote:
> 
> Added them all, tx Rohit.
> 
> +Vinod
> 
>> On Aug 5, 2015, at 9:51 PM, Rohith Sharma K S  
>> wrote:
>> 
>> Can we add following fixes to 2.6.1?
>> 
>> YARN-3733 
>> YARN-2865
>> YARN-3990
>> YARN-2894
>> 
>> 
>> Thanks & Regards
>> Rohith Sharma K S
>> 
>> 
>> -Original Message-
>> From: Allan Wilson [mailto:awils...@pandora.com] 
>> Sent: 05 August 2015 23:25
>> To: common-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org; 
>> yarn-...@hadoop.apache.org
>> Cc: hdfs-...@hadoop.apache.org
>> Subject: Re: Planning Hadoop 2.6.1 release
>> 
>> Another +1 to add those fixes.
>> 
>> YARN-3487 bug can grind a large cluster to a halt repeatedly
>> 
>> 
>> 
>> -Allan
>> 
>> 
>> Allan Wilson | Sr. Software Engineer | Pandora m 919.841.2449 |  
>> awils...@pandora.com
>> 
>> 
>> 
>> 
>> 
>> On 8/5/15, 1:52 PM, "Rich Haase"  wrote:
>> 
>>> +1 to add those fixes.
>>> 
>>> 
>>> Rich Haase | Sr. Software Engineer | Pandora m 303.887.1146 | 
>>> rha...@pandora.com
>>> 
>>> 
>>> 
>>> 
>>> On 8/5/15, 11:42 AM, "Wangda Tan"  wrote:
>>> 
>>>> Can we add following two fixes to 2.6.1?
>>>> 
>>>> https://issues.apache.org/jira/browse/YARN-2922 and 
>>>> https://issues.apache.org/jira/browse/YARN-3487.
>>>> 
>>>> They're not fatal issue, but they can cause lots of issue in a large 
>>>> cluster.
>>>> 
>>>> Thanks,
>>>> Wangda
>>>> 
>>>> 
>>>> On Mon, Aug 3, 2015 at 1:21 PM, Sangjin Lee  wrote:
>>>> 
>>>>> See my later update in the thread. HDFS-7704 is in the list.
>>>>> 
>>>>> Thanks,
>>>>> Sangjin
>>>>> 
>>>>> On Mon, Aug 3, 2015 at 1:19 PM, Vinod Kumar Vavilapalli < 
>>>>> vino...@hortonworks.com> wrote:
>>>>> 
>>>>>> Makes sense, it was caused by HDFS-7704 which got into 2.7.0 only 
>>>>>> and
>>>>> is
>>>>>> not part of the candidate list. Removed HDFS-7916 from the list.
>>>>>> 
>>>>>> Thanks
>>>>>> +Vinod
>>>>>> 
>>>>>>> On Jul 24, 2015, at 6:32 PM, Sangjin Lee  wrote:
>>>>>>> 
>>>>>>> Out of the JIRAs we proposed, please remove HDFS-7916. I don't
>>>>> think it
>>>>>>> applies to 2.6.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Sangjin
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> 
>> 
> 

Re: Planning Hadoop 2.6.1 release

[Hitesh Shah]

Jason Lowe just filed YARN-4047 ( patch available too ) which probably is also 
a good candidate for 2.6.1 or 2.6.2 if the list is frozen. I have labelled the 
jira as a 2.6.1 candidate. Feel free to switch to 2.6.2 if needed. 

thanks
— Hitesh

On Aug 10, 2015, at 4:15 PM, Vinod Kumar Vavilapalli  
wrote:

> As of now, I am freezing the list. We have 139 tickets to go through, find 
> the right order, get them committed etc.
> 
> I’ll start a separate thread for this one-time long release-process of 2.6.1.
> 
> Thanks everyone for pitching in.
> +Vinod
> 
>> On Aug 10, 2015, at 4:13 PM, Vinod Kumar Vavilapalli 
>>  wrote:
>> 
>> Added them all, tx Rohit.
>> 
>> +Vinod
>> 
>>> On Aug 5, 2015, at 9:51 PM, Rohith Sharma K S  
>>> wrote:
>>> 
>>> Can we add following fixes to 2.6.1?
>>> 
>>> YARN-3733 
>>> YARN-2865
>>> YARN-3990
>>> YARN-2894
>>> 
>>> 
>>> Thanks & Regards
>>> Rohith Sharma K S
>>> 
>>> 
>>> -Original Message-
>>> From: Allan Wilson [mailto:awils...@pandora.com] 
>>> Sent: 05 August 2015 23:25
>>> To: common-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org; 
>>> yarn-...@hadoop.apache.org
>>> Cc: hdfs-...@hadoop.apache.org
>>> Subject: Re: Planning Hadoop 2.6.1 release
>>> 
>>> Another +1 to add those fixes.
>>> 
>>> YARN-3487 bug can grind a large cluster to a halt repeatedly
>>> 
>>> 
>>> 
>>> -Allan
>>> 
>>> 
>>> Allan Wilson | Sr. Software Engineer | Pandora m 919.841.2449 |  
>>> awils...@pandora.com
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On 8/5/15, 1:52 PM, "Rich Haase"  wrote:
>>> 
>>>> +1 to add those fixes.
>>>> 
>>>> 
>>>> Rich Haase | Sr. Software Engineer | Pandora m 303.887.1146 | 
>>>> rha...@pandora.com
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On 8/5/15, 11:42 AM, "Wangda Tan"  wrote:
>>>> 
>>>>> Can we add following two fixes to 2.6.1?
>>>>> 
>>>>> https://issues.apache.org/jira/browse/YARN-2922 and 
>>>>> https://issues.apache.org/jira/browse/YARN-3487.
>>>>> 
>>>>> They're not fatal issue, but they can cause lots of issue in a large 
>>>>> cluster.
>>>>> 
>>>>> Thanks,
>>>>> Wangda
>>>>> 
>>>>> 
>>>>> On Mon, Aug 3, 2015 at 1:21 PM, Sangjin Lee  wrote:
>>>>> 
>>>>>> See my later update in the thread. HDFS-7704 is in the list.
>>>>>> 
>>>>>> Thanks,
>>>>>> Sangjin
>>>>>> 
>>>>>> On Mon, Aug 3, 2015 at 1:19 PM, Vinod Kumar Vavilapalli < 
>>>>>> vino...@hortonworks.com> wrote:
>>>>>> 
>>>>>>> Makes sense, it was caused by HDFS-7704 which got into 2.7.0 only 
>>>>>>> and
>>>>>> is
>>>>>>> not part of the candidate list. Removed HDFS-7916 from the list.
>>>>>>> 
>>>>>>> Thanks
>>>>>>> +Vinod
>>>>>>> 
>>>>>>>> On Jul 24, 2015, at 6:32 PM, Sangjin Lee  wrote:
>>>>>>>> 
>>>>>>>> Out of the JIRAs we proposed, please remove HDFS-7916. I don't
>>>>>> think it
>>>>>>>> applies to 2.6.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Sangjin
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>> 
>>> 
>>> 
>> 
> 

Re: Planning Hadoop 2.6.1 release

[Sangjin Lee]

HDFS-8850 <https://issues.apache.org/jira/browse/HDFS-8850> is a fix for
VolumeScanner which was added to 2.7 (HDFS-7430
<https://issues.apache.org/jira/browse/HDFS-7430>). I don't see it
applicable to 2.6. Could you please confirm?

Thanks,
Sangjin

On Wed, Aug 5, 2015 at 2:34 PM, Sean Busbey  wrote:

> If we haven't frozen yet, HDFS-8850 is a straight forward fix that is
> currently only in 2.8+ and would benefit 2.6 and 2.7.
>
> On Wed, Aug 5, 2015 at 2:56 PM, Junping Du  wrote:
>
> > I would like to nominate YARN-3832 as 2.6.1 candidate which is critical
> > and I also saw it happened recently on a 2.6.0 cluster. Hope this is not
> > too late.
> >
> > Thanks,
> >
> > Junping
> > 
> > From: Rich Haase 
> > Sent: Thursday, August 06, 2015 1:52 AM
> > To: mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org
> > Cc: common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org
> > Subject: Re: Planning Hadoop 2.6.1 release
> >
> > +1 to add those fixes.
> >
> >
> > Rich Haase | Sr. Software Engineer | Pandora
> > m 303.887.1146 | rha...@pandora.com
> >
> >
> >
> >
> > On 8/5/15, 11:42 AM, "Wangda Tan"  wrote:
> >
> > >Can we add following two fixes to 2.6.1?
> > >
> > >https://issues.apache.org/jira/browse/YARN-2922 and
> > >https://issues.apache.org/jira/browse/YARN-3487.
> > >
> > >They're not fatal issue, but they can cause lots of issue in a large
> > >cluster.
> > >
> > >Thanks,
> > >Wangda
> > >
> > >
> > >On Mon, Aug 3, 2015 at 1:21 PM, Sangjin Lee  wrote:
> > >
> > >> See my later update in the thread. HDFS-7704 is in the list.
> > >>
> > >> Thanks,
> > >> Sangjin
> > >>
> > >> On Mon, Aug 3, 2015 at 1:19 PM, Vinod Kumar Vavilapalli <
> > >> vino...@hortonworks.com> wrote:
> > >>
> > >> > Makes sense, it was caused by HDFS-7704 which got into 2.7.0 only
> and
> > >>is
> > >> > not part of the candidate list. Removed HDFS-7916 from the list.
> > >> >
> > >> > Thanks
> > >> > +Vinod
> > >> >
> > >> > > On Jul 24, 2015, at 6:32 PM, Sangjin Lee 
> wrote:
> > >> > >
> > >> > > Out of the JIRAs we proposed, please remove HDFS-7916. I don't
> > >>think it
> > >> > > applies to 2.6.
> > >> > >
> > >> > > Thanks,
> > >> > > Sangjin
> > >> >
> > >> >
> > >>
> >
> >
>
>
> --
> Sean
>

Re: Planning Hadoop 2.6.1 release

[Vinod Kumar Vavilapalli]

Everyone,

Please stop committing to 2.6.1 (branch-2.6 or branch-2.6.1), we have a fairly 
elaborate parallel release-process going on for 2.6.1 (with me / Akira and 
Sangjin participating offline) and any outside cherry-picks are going to 
disrupt our progress.

If you want something added to 2.6.1, please post it in the mailing lists and 
we will consider it.

Thanks.
+Vinod

> On Aug 14, 2015, at 12:17 AM, Vinayakumar B  wrote:
> 
> FYI, Some of the issues in
> https://wiki.apache.org/hadoop/Release-2.6.1-Working-Notes are already
> merged in 2.6.1.
> 
> 
> 
> Regards,
> Vinay
> 
> On Fri, Aug 14, 2015 at 12:43 PM, Sangjin Lee  wrote:
> 
>> HDFS-8850 <https://issues.apache.org/jira/browse/HDFS-8850> is a fix for
>> VolumeScanner which was added to 2.7 (HDFS-7430
>> <https://issues.apache.org/jira/browse/HDFS-7430>). I don't see it
>> applicable to 2.6. Could you please confirm?
>> 
>> Thanks,
>> Sangjin
>> 
>> On Wed, Aug 5, 2015 at 2:34 PM, Sean Busbey  wrote:
>> 
>>> If we haven't frozen yet, HDFS-8850 is a straight forward fix that is
>>> currently only in 2.8+ and would benefit 2.6 and 2.7.
>>> 
>>> On Wed, Aug 5, 2015 at 2:56 PM, Junping Du  wrote:
>>> 
>>>> I would like to nominate YARN-3832 as 2.6.1 candidate which is critical
>>>> and I also saw it happened recently on a 2.6.0 cluster. Hope this is
>> not
>>>> too late.
>>>> 
>>>> Thanks,
>>>> 
>>>> Junping
>>>> ____________
>>>> From: Rich Haase 
>>>> Sent: Thursday, August 06, 2015 1:52 AM
>>>> To: mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org
>>>> Cc: common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org
>>>> Subject: Re: Planning Hadoop 2.6.1 release
>>>> 
>>>> +1 to add those fixes.
>>>> 
>>>> 
>>>> Rich Haase | Sr. Software Engineer | Pandora
>>>> m 303.887.1146 | rha...@pandora.com
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On 8/5/15, 11:42 AM, "Wangda Tan"  wrote:
>>>> 
>>>>> Can we add following two fixes to 2.6.1?
>>>>> 
>>>>> https://issues.apache.org/jira/browse/YARN-2922 and
>>>>> https://issues.apache.org/jira/browse/YARN-3487.
>>>>> 
>>>>> They're not fatal issue, but they can cause lots of issue in a large
>>>>> cluster.
>>>>> 
>>>>> Thanks,
>>>>> Wangda
>>>>> 
>>>>> 
>>>>> On Mon, Aug 3, 2015 at 1:21 PM, Sangjin Lee  wrote:
>>>>> 
>>>>>> See my later update in the thread. HDFS-7704 is in the list.
>>>>>> 
>>>>>> Thanks,
>>>>>> Sangjin
>>>>>> 
>>>>>> On Mon, Aug 3, 2015 at 1:19 PM, Vinod Kumar Vavilapalli <
>>>>>> vino...@hortonworks.com> wrote:
>>>>>> 
>>>>>>> Makes sense, it was caused by HDFS-7704 which got into 2.7.0 only
>>> and
>>>>>> is
>>>>>>> not part of the candidate list. Removed HDFS-7916 from the list.
>>>>>>> 
>>>>>>> Thanks
>>>>>>> +Vinod
>>>>>>> 
>>>>>>>> On Jul 24, 2015, at 6:32 PM, Sangjin Lee 
>>> wrote:
>>>>>>>> 
>>>>>>>> Out of the JIRAs we proposed, please remove HDFS-7916. I don't
>>>>>> think it
>>>>>>>> applies to 2.6.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Sangjin
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> --
>>> Sean
>>> 
>> 

Re: Planning Hadoop 2.6.1 release

[Vinod Kumar Vavilapalli]

To clarify, there is already a branch-2.6.1 created separately. So, if you want 
something in 2.6.2, you can continue putting it in branch-2.6.

But no 2.6.1 related commits please.

Thanks
+Vinod

> On Aug 14, 2015, at 12:09 PM, Vinod Kumar Vavilapalli 
>  wrote:
> 
> Everyone,
> 
> Please stop committing to 2.6.1 (branch-2.6 or branch-2.6.1), we have a 
> fairly elaborate parallel release-process going on for 2.6.1 (with me / Akira 
> and Sangjin participating offline) and any outside cherry-picks are going to 
> disrupt our progress.
> 
> If you want something added to 2.6.1, please post it in the mailing lists and 
> we will consider it.
> 
> Thanks.
> +Vinod
> 
>> On Aug 14, 2015, at 12:17 AM, Vinayakumar B  wrote:
>> 
>> FYI, Some of the issues in
>> https://wiki.apache.org/hadoop/Release-2.6.1-Working-Notes are already
>> merged in 2.6.1.
>> 
>> 
>> 
>> Regards,
>> Vinay
>> 
>> On Fri, Aug 14, 2015 at 12:43 PM, Sangjin Lee  wrote:
>> 
>>> HDFS-8850 <https://issues.apache.org/jira/browse/HDFS-8850> is a fix for
>>> VolumeScanner which was added to 2.7 (HDFS-7430
>>> <https://issues.apache.org/jira/browse/HDFS-7430>). I don't see it
>>> applicable to 2.6. Could you please confirm?
>>> 
>>> Thanks,
>>> Sangjin
>>> 
>>> On Wed, Aug 5, 2015 at 2:34 PM, Sean Busbey  wrote:
>>> 
>>>> If we haven't frozen yet, HDFS-8850 is a straight forward fix that is
>>>> currently only in 2.8+ and would benefit 2.6 and 2.7.
>>>> 
>>>> On Wed, Aug 5, 2015 at 2:56 PM, Junping Du  wrote:
>>>> 
>>>>> I would like to nominate YARN-3832 as 2.6.1 candidate which is critical
>>>>> and I also saw it happened recently on a 2.6.0 cluster. Hope this is
>>> not
>>>>> too late.
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> Junping
>>>>> 
>>>>> From: Rich Haase 
>>>>> Sent: Thursday, August 06, 2015 1:52 AM
>>>>> To: mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org
>>>>> Cc: common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org
>>>>> Subject: Re: Planning Hadoop 2.6.1 release
>>>>> 
>>>>> +1 to add those fixes.
>>>>> 
>>>>> 
>>>>> Rich Haase | Sr. Software Engineer | Pandora
>>>>> m 303.887.1146 | rha...@pandora.com
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> On 8/5/15, 11:42 AM, "Wangda Tan"  wrote:
>>>>> 
>>>>>> Can we add following two fixes to 2.6.1?
>>>>>> 
>>>>>> https://issues.apache.org/jira/browse/YARN-2922 and
>>>>>> https://issues.apache.org/jira/browse/YARN-3487.
>>>>>> 
>>>>>> They're not fatal issue, but they can cause lots of issue in a large
>>>>>> cluster.
>>>>>> 
>>>>>> Thanks,
>>>>>> Wangda
>>>>>> 
>>>>>> 
>>>>>> On Mon, Aug 3, 2015 at 1:21 PM, Sangjin Lee  wrote:
>>>>>> 
>>>>>>> See my later update in the thread. HDFS-7704 is in the list.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Sangjin
>>>>>>> 
>>>>>>> On Mon, Aug 3, 2015 at 1:19 PM, Vinod Kumar Vavilapalli <
>>>>>>> vino...@hortonworks.com> wrote:
>>>>>>> 
>>>>>>>> Makes sense, it was caused by HDFS-7704 which got into 2.7.0 only
>>>> and
>>>>>>> is
>>>>>>>> not part of the candidate list. Removed HDFS-7916 from the list.
>>>>>>>> 
>>>>>>>> Thanks
>>>>>>>> +Vinod
>>>>>>>> 
>>>>>>>>> On Jul 24, 2015, at 6:32 PM, Sangjin Lee 
>>>> wrote:
>>>>>>>>> 
>>>>>>>>> Out of the JIRAs we proposed, please remove HDFS-7916. I don't
>>>>>>> think it
>>>>>>>>> applies to 2.6.
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> Sangjin
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> --
>>>> Sean
>>>> 
>>> 
> 

Re: Planning Hadoop 2.6.1 release

[Vinayakumar B]

Apologies for the disruption by commits to branch-2.6

I thought since the branch-2.6 already have the setup made for 2.6.1 its
ready for merging the commits from the list
https://wiki.apache.org/hadoop/Release-2.6.1-Working-Notes.

Cherry-picked only issues from the list and which were cleanly appliable.

-Vinay

On Sat, Aug 15, 2015 at 12:53 AM, Vinod Kumar Vavilapalli <
vino...@hortonworks.com> wrote:

> To clarify, there is already a branch-2.6.1 created separately. So, if you
> want something in 2.6.2, you can continue putting it in branch-2.6.
>
> But no 2.6.1 related commits please.
>
> Thanks
> +Vinod
>
> > On Aug 14, 2015, at 12:09 PM, Vinod Kumar Vavilapalli <
> vino...@hortonworks.com> wrote:
> >
> > Everyone,
> >
> > Please stop committing to 2.6.1 (branch-2.6 or branch-2.6.1), we have a
> fairly elaborate parallel release-process going on for 2.6.1 (with me /
> Akira and Sangjin participating offline) and any outside cherry-picks are
> going to disrupt our progress.
> >
> > If you want something added to 2.6.1, please post it in the mailing
> lists and we will consider it.
> >
> > Thanks.
> > +Vinod
> >
> >> On Aug 14, 2015, at 12:17 AM, Vinayakumar B 
> wrote:
> >>
> >> FYI, Some of the issues in
> >> https://wiki.apache.org/hadoop/Release-2.6.1-Working-Notes are already
> >> merged in 2.6.1.
> >>
> >>
> >>
> >> Regards,
> >> Vinay
> >>
> >> On Fri, Aug 14, 2015 at 12:43 PM, Sangjin Lee  wrote:
> >>
> >>> HDFS-8850 <https://issues.apache.org/jira/browse/HDFS-8850> is a fix
> for
> >>> VolumeScanner which was added to 2.7 (HDFS-7430
> >>> <https://issues.apache.org/jira/browse/HDFS-7430>). I don't see it
> >>> applicable to 2.6. Could you please confirm?
> >>>
> >>> Thanks,
> >>> Sangjin
> >>>
> >>> On Wed, Aug 5, 2015 at 2:34 PM, Sean Busbey 
> wrote:
> >>>
> >>>> If we haven't frozen yet, HDFS-8850 is a straight forward fix that is
> >>>> currently only in 2.8+ and would benefit 2.6 and 2.7.
> >>>>
> >>>> On Wed, Aug 5, 2015 at 2:56 PM, Junping Du 
> wrote:
> >>>>
> >>>>> I would like to nominate YARN-3832 as 2.6.1 candidate which is
> critical
> >>>>> and I also saw it happened recently on a 2.6.0 cluster. Hope this is
> >>> not
> >>>>> too late.
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> Junping
> >>>>> 
> >>>>> From: Rich Haase 
> >>>>> Sent: Thursday, August 06, 2015 1:52 AM
> >>>>> To: mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org
> >>>>> Cc: common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org
> >>>>> Subject: Re: Planning Hadoop 2.6.1 release
> >>>>>
> >>>>> +1 to add those fixes.
> >>>>>
> >>>>>
> >>>>> Rich Haase | Sr. Software Engineer | Pandora
> >>>>> m 303.887.1146 | rha...@pandora.com
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On 8/5/15, 11:42 AM, "Wangda Tan"  wrote:
> >>>>>
> >>>>>> Can we add following two fixes to 2.6.1?
> >>>>>>
> >>>>>> https://issues.apache.org/jira/browse/YARN-2922 and
> >>>>>> https://issues.apache.org/jira/browse/YARN-3487.
> >>>>>>
> >>>>>> They're not fatal issue, but they can cause lots of issue in a large
> >>>>>> cluster.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Wangda
> >>>>>>
> >>>>>>
> >>>>>> On Mon, Aug 3, 2015 at 1:21 PM, Sangjin Lee 
> wrote:
> >>>>>>
> >>>>>>> See my later update in the thread. HDFS-7704 is in the list.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Sangjin
> >>>>>>>
> >>>>>>> On Mon, Aug 3, 2015 at 1:19 PM, Vinod Kumar Vavilapalli <
> >>>>>>> vino...@hortonworks.com> wrote:
> >>>>>>>
> >>>>>>>> Makes sense, it was caused by HDFS-7704 which got into 2.7.0 only
> >>>> and
> >>>>>>> is
> >>>>>>>> not part of the candidate list. Removed HDFS-7916 from the list.
> >>>>>>>>
> >>>>>>>> Thanks
> >>>>>>>> +Vinod
> >>>>>>>>
> >>>>>>>>> On Jul 24, 2015, at 6:32 PM, Sangjin Lee 
> >>>> wrote:
> >>>>>>>>>
> >>>>>>>>> Out of the JIRAs we proposed, please remove HDFS-7916. I don't
> >>>>>>> think it
> >>>>>>>>> applies to 2.6.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> Sangjin
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> Sean
> >>>>
> >>>
> >
>
>

Re: Planning Hadoop 2.6.1 release

[Nikhil]

Hi Vinod,

I am wondering if there is any news on 2.6.1 release. Any dates planned for
the release?

Nikhil


On Fri, Aug 14, 2015 at 12:23 PM, Vinod Kumar Vavilapalli <
vino...@hortonworks.com> wrote:

> To clarify, there is already a branch-2.6.1 created separately. So, if you
> want something in 2.6.2, you can continue putting it in branch-2.6.
>
> But no 2.6.1 related commits please.
>
> Thanks
> +Vinod
>
> > On Aug 14, 2015, at 12:09 PM, Vinod Kumar Vavilapalli <
> vino...@hortonworks.com> wrote:
> >
> > Everyone,
> >
> > Please stop committing to 2.6.1 (branch-2.6 or branch-2.6.1), we have a
> fairly elaborate parallel release-process going on for 2.6.1 (with me /
> Akira and Sangjin participating offline) and any outside cherry-picks are
> going to disrupt our progress.
> >
> > If you want something added to 2.6.1, please post it in the mailing
> lists and we will consider it.
> >
> > Thanks.
> > +Vinod
> >
> >> On Aug 14, 2015, at 12:17 AM, Vinayakumar B 
> wrote:
> >>
> >> FYI, Some of the issues in
> >> https://wiki.apache.org/hadoop/Release-2.6.1-Working-Notes are already
> >> merged in 2.6.1.
> >>
> >>
> >>
> >> Regards,
> >> Vinay
> >>
> >> On Fri, Aug 14, 2015 at 12:43 PM, Sangjin Lee  wrote:
> >>
> >>> HDFS-8850 <https://issues.apache.org/jira/browse/HDFS-8850> is a fix
> for
> >>> VolumeScanner which was added to 2.7 (HDFS-7430
> >>> <https://issues.apache.org/jira/browse/HDFS-7430>). I don't see it
> >>> applicable to 2.6. Could you please confirm?
> >>>
> >>> Thanks,
> >>> Sangjin
> >>>
> >>> On Wed, Aug 5, 2015 at 2:34 PM, Sean Busbey 
> wrote:
> >>>
> >>>> If we haven't frozen yet, HDFS-8850 is a straight forward fix that is
> >>>> currently only in 2.8+ and would benefit 2.6 and 2.7.
> >>>>
> >>>> On Wed, Aug 5, 2015 at 2:56 PM, Junping Du 
> wrote:
> >>>>
> >>>>> I would like to nominate YARN-3832 as 2.6.1 candidate which is
> critical
> >>>>> and I also saw it happened recently on a 2.6.0 cluster. Hope this is
> >>> not
> >>>>> too late.
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> Junping
> >>>>> 
> >>>>> From: Rich Haase 
> >>>>> Sent: Thursday, August 06, 2015 1:52 AM
> >>>>> To: mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org
> >>>>> Cc: common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org
> >>>>> Subject: Re: Planning Hadoop 2.6.1 release
> >>>>>
> >>>>> +1 to add those fixes.
> >>>>>
> >>>>>
> >>>>> Rich Haase | Sr. Software Engineer | Pandora
> >>>>> m 303.887.1146 | rha...@pandora.com
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On 8/5/15, 11:42 AM, "Wangda Tan"  wrote:
> >>>>>
> >>>>>> Can we add following two fixes to 2.6.1?
> >>>>>>
> >>>>>> https://issues.apache.org/jira/browse/YARN-2922 and
> >>>>>> https://issues.apache.org/jira/browse/YARN-3487.
> >>>>>>
> >>>>>> They're not fatal issue, but they can cause lots of issue in a large
> >>>>>> cluster.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Wangda
> >>>>>>
> >>>>>>
> >>>>>> On Mon, Aug 3, 2015 at 1:21 PM, Sangjin Lee 
> wrote:
> >>>>>>
> >>>>>>> See my later update in the thread. HDFS-7704 is in the list.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Sangjin
> >>>>>>>
> >>>>>>> On Mon, Aug 3, 2015 at 1:19 PM, Vinod Kumar Vavilapalli <
> >>>>>>> vino...@hortonworks.com> wrote:
> >>>>>>>
> >>>>>>>> Makes sense, it was caused by HDFS-7704 which got into 2.7.0 only
> >>>> and
> >>>>>>> is
> >>>>>>>> not part of the candidate list. Removed HDFS-7916 from the list.
> >>>>>>>>
> >>>>>>>> Thanks
> >>>>>>>> +Vinod
> >>>>>>>>
> >>>>>>>>> On Jul 24, 2015, at 6:32 PM, Sangjin Lee 
> >>>> wrote:
> >>>>>>>>>
> >>>>>>>>> Out of the JIRAs we proposed, please remove HDFS-7916. I don't
> >>>>>>> think it
> >>>>>>>>> applies to 2.6.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> Sangjin
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> Sean
> >>>>
> >>>
> >
>
>

Re: Planning Hadoop 2.6.1 release

[Vinod Kumar Vavilapalli]

Funny you ask, I committed the last of the fixes today and rolled an RC - 
http://markmail.org/thread/dsvlaxj6vhxjg3i4

Thanks
+Vinod

On Sep 9, 2015, at 4:22 PM, Nikhil 
mailto:mnik...@gmail.com>> wrote:

Hi Vinod,

I am wondering if there is any news on 2.6.1 release. Any dates planned for
the release?

Nikhil


On Fri, Aug 14, 2015 at 12:23 PM, Vinod Kumar Vavilapalli <
vino...@hortonworks.com<mailto:vino...@hortonworks.com>> wrote:

To clarify, there is already a branch-2.6.1 created separately. So, if you
want something in 2.6.2, you can continue putting it in branch-2.6.

But no 2.6.1 related commits please.

Thanks
+Vinod

On Aug 14, 2015, at 12:09 PM, Vinod Kumar Vavilapalli <
vino...@hortonworks.com<mailto:vino...@hortonworks.com>> wrote:

Everyone,

Please stop committing to 2.6.1 (branch-2.6 or branch-2.6.1), we have a
fairly elaborate parallel release-process going on for 2.6.1 (with me /
Akira and Sangjin participating offline) and any outside cherry-picks are
going to disrupt our progress.

If you want something added to 2.6.1, please post it in the mailing
lists and we will consider it.

Thanks.
+Vinod

On Aug 14, 2015, at 12:17 AM, Vinayakumar B 
mailto:vinayakum...@apache.org>>
wrote:

FYI, Some of the issues in
https://wiki.apache.org/hadoop/Release-2.6.1-Working-Notes are already
merged in 2.6.1.



Regards,
Vinay

On Fri, Aug 14, 2015 at 12:43 PM, Sangjin Lee 
mailto:sj...@apache.org>> wrote:

HDFS-8850 <https://issues.apache.org/jira/browse/HDFS-8850> is a fix
for
VolumeScanner which was added to 2.7 (HDFS-7430
<https://issues.apache.org/jira/browse/HDFS-7430>). I don't see it
applicable to 2.6. Could you please confirm?

Thanks,
Sangjin

On Wed, Aug 5, 2015 at 2:34 PM, Sean Busbey 
mailto:bus...@cloudera.com>>
wrote:

If we haven't frozen yet, HDFS-8850 is a straight forward fix that is
currently only in 2.8+ and would benefit 2.6 and 2.7.

On Wed, Aug 5, 2015 at 2:56 PM, Junping Du 
mailto:j...@hortonworks.com>>
wrote:

I would like to nominate YARN-3832 as 2.6.1 candidate which is
critical
and I also saw it happened recently on a 2.6.0 cluster. Hope this is
not
too late.

Thanks,

Junping

From: Rich Haase mailto:rha...@pandora.com>>
Sent: Thursday, August 06, 2015 1:52 AM
To: mapreduce-...@hadoop.apache.org<mailto:mapreduce-...@hadoop.apache.org>; 
yarn-...@hadoop.apache.org<mailto:yarn-...@hadoop.apache.org>
Cc: common-dev@hadoop.apache.org<mailto:common-dev@hadoop.apache.org>; 
hdfs-...@hadoop.apache.org<mailto:hdfs-...@hadoop.apache.org>
Subject: Re: Planning Hadoop 2.6.1 release

+1 to add those fixes.


Rich Haase | Sr. Software Engineer | Pandora
m 303.887.1146 | rha...@pandora.com<mailto:rha...@pandora.com>




On 8/5/15, 11:42 AM, "Wangda Tan" 
mailto:wheele...@gmail.com>> wrote:

Can we add following two fixes to 2.6.1?

https://issues.apache.org/jira/browse/YARN-2922 and
https://issues.apache.org/jira/browse/YARN-3487.

They're not fatal issue, but they can cause lots of issue in a large
cluster.

Thanks,
Wangda


On Mon, Aug 3, 2015 at 1:21 PM, Sangjin Lee 
mailto:sj...@apache.org>>
wrote:

See my later update in the thread. HDFS-7704 is in the list.

Thanks,
Sangjin

On Mon, Aug 3, 2015 at 1:19 PM, Vinod Kumar Vavilapalli <
vino...@hortonworks.com<mailto:vino...@hortonworks.com>> wrote:

Makes sense, it was caused by HDFS-7704 which got into 2.7.0 only
and
is
not part of the candidate list. Removed HDFS-7916 from the list.

Thanks
+Vinod

On Jul 24, 2015, at 6:32 PM, Sangjin Lee 
mailto:sj...@apache.org>>
wrote:

Out of the JIRAs we proposed, please remove HDFS-7916. I don't
think it
applies to 2.6.

Thanks,
Sangjin







--
Sean

Re: Planning Hadoop 2.6.1 release

[Arpit Agarwal]

HDFS candidates for back-porting to Hadoop 2.6.1. The first two were requested 
in [1].

HADOOP-11674. oneByteBuf in CryptoInputStream and CryptoOutputStream should be 
non static
HADOOP-11710. Make CryptoOutputStream behave like DFSOutputStream wrt 
synchronization

HDFS-7009. Active NN and standby NN have different live nodes.
HDFS-7035. Make adding a new data directory to the DataNode an atomic and 
improve error handling
HDFS-7425. NameNode block deletion logging uses incorrect appender.
HDFS-7443. Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate block 
files are present in the same volume.
HDFS-7489. Incorrect locking in FsVolumeList#checkDirs can hang datanodes
HDFS-7503. Namenode restart after large deletions can cause slow processReport.
HDFS-7575. Upgrade should generate a unique storage ID for each volume.
HDFS-7579. Improve log reporting during block report rpc failure.
HDFS-7587. Edit log corruption can happen if append fails with a quota 
violation.
HDFS-7596. NameNode should prune dead storages from storageMap.
HDFS-7611. deleteSnapshot and delete of a file can leave orphaned blocks in the 
blocksMap on NameNode restart.
HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause DataNode 
to register successfully with only one NameNode.
HDFS-7733. NFS: readdir/readdirplus return null directory attribute on failure.
HDFS-7831. Fix the starting index and end condition of the loop in 
FileDiffList.findEarlierSnapshotBlocks().
HDFS-7885. Datanode should not trust the generation stamp provided by client.
HDFS-7960. The full block report should prune zombie storages even if they're 
not empty.
HDFS-8072. Reserved RBW space is not released if client terminates while 
writing block.
HDFS-8127. NameNode Failover during HA upgrade can cause DataNode to finalize 
upgrade.


Arpit

[1] Will Hadoop 2.6.1 be released soon? 
http://markmail.org/thread/zlsr6prejyogdyvh



On 4/27/15, 11:47 AM, "Vinod Kumar Vavilapalli"  wrote:

>There were several requests on the user lists [1] for a 2.6.1 release. I
>got many offline comments too.
>
>Planning to do a 2.6.1 release in a few weeks time. We already have a bunch
>of tickets committed to 2.7.1. I created a filter [2] to tracking pending
>tickets.
>
>We need to collectively come up with a list of critical issues. We can use
>the JIRA Target Version field for the same. I see some but not a whole lot
>of new work for this release, most of it is likely going to be pulling in
>critical patches from 2.7.1/2.8 etc.
>
>Thoughts?
>
>Thanks
>+Vinod
>
>[1] Will Hadoop 2.6.1 be released soon?
>http://markmail.org/thread/zlsr6prejyogdyvh
>[2] 2.6.1 pending tickets
>https://issues.apache.org/jira/issues/?filter=12331711
>

Re: Planning Hadoop 2.6.1 release

[Chris Nauroth]

Thank you, Arpit.  In addition, I suggest we include the following:

HADOOP-11333. Fix deadlock in DomainSocketWatcher when the notification
pipe is full
HADOOP-11604. Prevent ConcurrentModificationException while closing domain
sockets during shutdown of DomainSocketWatcher thread.
HADOOP-11648. Set DomainSocketWatcher thread name explicitly
HADOOP-11802. DomainSocketWatcher thread terminates sometimes after there
is an I/O error during requestShortCircuitShm

HADOOP-11604 and 11648 are not critical by themselves, but they are
pre-requisites to getting a clean cherry-pick of 11802, which we believe
finally fixes the root cause of this issue.


--Chris Nauroth




On 4/30/15, 3:55 PM, "Arpit Agarwal"  wrote:

>HDFS candidates for back-porting to Hadoop 2.6.1. The first two were
>requested in [1].
>
>HADOOP-11674. oneByteBuf in CryptoInputStream and CryptoOutputStream
>should be non static
>HADOOP-11710. Make CryptoOutputStream behave like DFSOutputStream wrt
>synchronization
>
>HDFS-7009. Active NN and standby NN have different live nodes.
>HDFS-7035. Make adding a new data directory to the DataNode an atomic and
>improve error handling
>HDFS-7425. NameNode block deletion logging uses incorrect appender.
>HDFS-7443. Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate
>block files are present in the same volume.
>HDFS-7489. Incorrect locking in FsVolumeList#checkDirs can hang datanodes
>HDFS-7503. Namenode restart after large deletions can cause slow
>processReport.
>HDFS-7575. Upgrade should generate a unique storage ID for each volume.
>HDFS-7579. Improve log reporting during block report rpc failure.
>HDFS-7587. Edit log corruption can happen if append fails with a quota
>violation.
>HDFS-7596. NameNode should prune dead storages from storageMap.
>HDFS-7611. deleteSnapshot and delete of a file can leave orphaned blocks
>in the blocksMap on NameNode restart.
>HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause
>DataNode to register successfully with only one NameNode.
>HDFS-7733. NFS: readdir/readdirplus return null directory attribute on
>failure.
>HDFS-7831. Fix the starting index and end condition of the loop in
>FileDiffList.findEarlierSnapshotBlocks().
>HDFS-7885. Datanode should not trust the generation stamp provided by
>client.
>HDFS-7960. The full block report should prune zombie storages even if
>they're not empty.
>HDFS-8072. Reserved RBW space is not released if client terminates while
>writing block.
>HDFS-8127. NameNode Failover during HA upgrade can cause DataNode to
>finalize upgrade.
>
>
>Arpit
>
>[1] Will Hadoop 2.6.1 be released soon?
>http://markmail.org/thread/zlsr6prejyogdyvh
>
>
>
>On 4/27/15, 11:47 AM, "Vinod Kumar Vavilapalli" 
>wrote:
>
>>There were several requests on the user lists [1] for a 2.6.1 release. I
>>got many offline comments too.
>>
>>Planning to do a 2.6.1 release in a few weeks time. We already have a
>>bunch
>>of tickets committed to 2.7.1. I created a filter [2] to tracking pending
>>tickets.
>>
>>We need to collectively come up with a list of critical issues. We can
>>use
>>the JIRA Target Version field for the same. I see some but not a whole
>>lot
>>of new work for this release, most of it is likely going to be pulling in
>>critical patches from 2.7.1/2.8 etc.
>>
>>Thoughts?
>>
>>Thanks
>>+Vinod
>>
>>[1] Will Hadoop 2.6.1 be released soon?
>>http://markmail.org/thread/zlsr6prejyogdyvh
>>[2] 2.6.1 pending tickets
>>https://issues.apache.org/jira/issues/?filter=12331711
>>
>
>

RE: Planning Hadoop 2.6.1 release

[Brahma Reddy Battula]

Hello Vinod,

I am thinking,can we include HADOOP-11491 also..? wihout this jira harfs will 
not be usable when cluster installed in HA mode and try to get filecontext like 
below..


Path path = new 
Path("har:///archivedLogs/application_1428917727658_0005-application_1428917727658_0008-1428927448352.har");
FileSystem fs = path.getFileSystem(new Configuration());
path = fs.makeQualified(path);
FileContext fc = FileContext.getFileContext(path.toUri(),new Configuration());



Thanks & Regards
Brahma Reddy Battula

From: Chris Nauroth [cnaur...@hortonworks.com]
Sent: Friday, May 01, 2015 4:32 AM
To: mapreduce-...@hadoop.apache.org; common-dev@hadoop.apache.org; 
yarn-...@hadoop.apache.org; hdfs-...@hadoop.apache.org
Subject: Re: Planning Hadoop 2.6.1 release

Thank you, Arpit.  In addition, I suggest we include the following:

HADOOP-11333. Fix deadlock in DomainSocketWatcher when the notification
pipe is full
HADOOP-11604. Prevent ConcurrentModificationException while closing domain
sockets during shutdown of DomainSocketWatcher thread.
HADOOP-11648. Set DomainSocketWatcher thread name explicitly
HADOOP-11802. DomainSocketWatcher thread terminates sometimes after there
is an I/O error during requestShortCircuitShm

HADOOP-11604 and 11648 are not critical by themselves, but they are
pre-requisites to getting a clean cherry-pick of 11802, which we believe
finally fixes the root cause of this issue.


--Chris Nauroth




On 4/30/15, 3:55 PM, "Arpit Agarwal"  wrote:

>HDFS candidates for back-porting to Hadoop 2.6.1. The first two were
>requested in [1].
>
>HADOOP-11674. oneByteBuf in CryptoInputStream and CryptoOutputStream
>should be non static
>HADOOP-11710. Make CryptoOutputStream behave like DFSOutputStream wrt
>synchronization
>
>HDFS-7009. Active NN and standby NN have different live nodes.
>HDFS-7035. Make adding a new data directory to the DataNode an atomic and
>improve error handling
>HDFS-7425. NameNode block deletion logging uses incorrect appender.
>HDFS-7443. Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate
>block files are present in the same volume.
>HDFS-7489. Incorrect locking in FsVolumeList#checkDirs can hang datanodes
>HDFS-7503. Namenode restart after large deletions can cause slow
>processReport.
>HDFS-7575. Upgrade should generate a unique storage ID for each volume.
>HDFS-7579. Improve log reporting during block report rpc failure.
>HDFS-7587. Edit log corruption can happen if append fails with a quota
>violation.
>HDFS-7596. NameNode should prune dead storages from storageMap.
>HDFS-7611. deleteSnapshot and delete of a file can leave orphaned blocks
>in the blocksMap on NameNode restart.
>HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause
>DataNode to register successfully with only one NameNode.
>HDFS-7733. NFS: readdir/readdirplus return null directory attribute on
>failure.
>HDFS-7831. Fix the starting index and end condition of the loop in
>FileDiffList.findEarlierSnapshotBlocks().
>HDFS-7885. Datanode should not trust the generation stamp provided by
>client.
>HDFS-7960. The full block report should prune zombie storages even if
>they're not empty.
>HDFS-8072. Reserved RBW space is not released if client terminates while
>writing block.
>HDFS-8127. NameNode Failover during HA upgrade can cause DataNode to
>finalize upgrade.
>
>
>Arpit
>
>[1] Will Hadoop 2.6.1 be released soon?
>http://markmail.org/thread/zlsr6prejyogdyvh
>
>
>
>On 4/27/15, 11:47 AM, "Vinod Kumar Vavilapalli" 
>wrote:
>
>>There were several requests on the user lists [1] for a 2.6.1 release. I
>>got many offline comments too.
>>
>>Planning to do a 2.6.1 release in a few weeks time. We already have a
>>bunch
>>of tickets committed to 2.7.1. I created a filter [2] to tracking pending
>>tickets.
>>
>>We need to collectively come up with a list of critical issues. We can
>>use
>>the JIRA Target Version field for the same. I see some but not a whole
>>lot
>>of new work for this release, most of it is likely going to be pulling in
>>critical patches from 2.7.1/2.8 etc.
>>
>>Thoughts?
>>
>>Thanks
>>+Vinod
>>
>>[1] Will Hadoop 2.6.1 be released soon?
>>http://markmail.org/thread/zlsr6prejyogdyvh
>>[2] 2.6.1 pending tickets
>>https://issues.apache.org/jira/issues/?filter=12331711
>>
>
>

Re: Planning Hadoop 2.6.1 release

[Akira AJISAKA]

Thanks all for collecting jiras for 2.6.1 release. In addition, I'd like 
to include the following:


* HADOOP-11343. Overflow is not properly handled in calculating final iv 
for AES CTR
* YARN-2874. Dead lock in "DelegationTokenRenewer" which blocks RM to 
execute any further apps

* YARN-2992. ZKRMStateStore crashes due to session expiry
* YARN-3013. AMRMClientImpl does not update AMRM token properly
* YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to die
* MAPREDUCE-6303. Read timeout when retrying a fetch error can be fatal 
to a reducer


All of these are marked as blocker bug for 2.7.0 but not fixed in 2.6.0.

Regards,
Akira

On 5/4/15 11:15, Brahma Reddy Battula wrote:

Hello Vinod,

I am thinking,can we include HADOOP-11491 also..? wihout this jira harfs will 
not be usable when cluster installed in HA mode and try to get filecontext like 
below..


Path path = new 
Path("har:///archivedLogs/application_1428917727658_0005-application_1428917727658_0008-1428927448352.har");
FileSystem fs = path.getFileSystem(new Configuration());
path = fs.makeQualified(path);
FileContext fc = FileContext.getFileContext(path.toUri(),new Configuration());



Thanks & Regards
Brahma Reddy Battula

From: Chris Nauroth [cnaur...@hortonworks.com]
Sent: Friday, May 01, 2015 4:32 AM
To: mapreduce-...@hadoop.apache.org; common-dev@hadoop.apache.org; 
yarn-...@hadoop.apache.org; hdfs-...@hadoop.apache.org
Subject: Re: Planning Hadoop 2.6.1 release

Thank you, Arpit.  In addition, I suggest we include the following:

HADOOP-11333. Fix deadlock in DomainSocketWatcher when the notification
pipe is full
HADOOP-11604. Prevent ConcurrentModificationException while closing domain
sockets during shutdown of DomainSocketWatcher thread.
HADOOP-11648. Set DomainSocketWatcher thread name explicitly
HADOOP-11802. DomainSocketWatcher thread terminates sometimes after there
is an I/O error during requestShortCircuitShm

HADOOP-11604 and 11648 are not critical by themselves, but they are
pre-requisites to getting a clean cherry-pick of 11802, which we believe
finally fixes the root cause of this issue.


--Chris Nauroth




On 4/30/15, 3:55 PM, "Arpit Agarwal"  wrote:


HDFS candidates for back-porting to Hadoop 2.6.1. The first two were
requested in [1].

HADOOP-11674. oneByteBuf in CryptoInputStream and CryptoOutputStream
should be non static
HADOOP-11710. Make CryptoOutputStream behave like DFSOutputStream wrt
synchronization

HDFS-7009. Active NN and standby NN have different live nodes.
HDFS-7035. Make adding a new data directory to the DataNode an atomic and
improve error handling
HDFS-7425. NameNode block deletion logging uses incorrect appender.
HDFS-7443. Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate
block files are present in the same volume.
HDFS-7489. Incorrect locking in FsVolumeList#checkDirs can hang datanodes
HDFS-7503. Namenode restart after large deletions can cause slow
processReport.
HDFS-7575. Upgrade should generate a unique storage ID for each volume.
HDFS-7579. Improve log reporting during block report rpc failure.
HDFS-7587. Edit log corruption can happen if append fails with a quota
violation.
HDFS-7596. NameNode should prune dead storages from storageMap.
HDFS-7611. deleteSnapshot and delete of a file can leave orphaned blocks
in the blocksMap on NameNode restart.
HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause
DataNode to register successfully with only one NameNode.
HDFS-7733. NFS: readdir/readdirplus return null directory attribute on
failure.
HDFS-7831. Fix the starting index and end condition of the loop in
FileDiffList.findEarlierSnapshotBlocks().
HDFS-7885. Datanode should not trust the generation stamp provided by
client.
HDFS-7960. The full block report should prune zombie storages even if
they're not empty.
HDFS-8072. Reserved RBW space is not released if client terminates while
writing block.
HDFS-8127. NameNode Failover during HA upgrade can cause DataNode to
finalize upgrade.


Arpit

[1] Will Hadoop 2.6.1 be released soon?
http://markmail.org/thread/zlsr6prejyogdyvh



On 4/27/15, 11:47 AM, "Vinod Kumar Vavilapalli" 
wrote:


There were several requests on the user lists [1] for a 2.6.1 release. I
got many offline comments too.

Planning to do a 2.6.1 release in a few weeks time. We already have a
bunch
of tickets committed to 2.7.1. I created a filter [2] to tracking pending
tickets.

We need to collectively come up with a list of critical issues. We can
use
the JIRA Target Version field for the same. I see some but not a whole
lot
of new work for this release, most of it is likely going to be pulling in
critical patches from 2.7.1/2.8 etc.

Thoughts?

Thanks
+Vinod

[1] Will Hadoop 2.6.1 be released soon?
http://markmail.org/thread/zlsr6prejyogdyvh
[2] 2.6.1 pending tickets
https://issues.apache.org/jira/issues/?filter=12331711

Re: Planning Hadoop 2.6.1 release

[Zhihai Xu]

Hi Akira,

Can we also include YARN-3242? YARN-3242 fixed a critical ZKRMStateStore
bug.
It will work better with YARN-2992.

thanks
zhihai


On Tue, May 12, 2015 at 10:38 PM, Akira AJISAKA 
wrote:

> Thanks all for collecting jiras for 2.6.1 release. In addition, I'd like
> to include the following:
>
> * HADOOP-11343. Overflow is not properly handled in calculating final iv
> for AES CTR
> * YARN-2874. Dead lock in "DelegationTokenRenewer" which blocks RM to
> execute any further apps
> * YARN-2992. ZKRMStateStore crashes due to session expiry
> * YARN-3013. AMRMClientImpl does not update AMRM token properly
> * YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to
> die
> * MAPREDUCE-6303. Read timeout when retrying a fetch error can be fatal to
> a reducer
>
> All of these are marked as blocker bug for 2.7.0 but not fixed in 2.6.0.
>
> Regards,
> Akira
>
>
> On 5/4/15 11:15, Brahma Reddy Battula wrote:
>
>> Hello Vinod,
>>
>> I am thinking,can we include HADOOP-11491 also..? wihout this jira harfs
>> will not be usable when cluster installed in HA mode and try to get
>> filecontext like below..
>>
>>
>> Path path = new
>> Path("har:///archivedLogs/application_1428917727658_0005-application_1428917727658_0008-1428927448352.har");
>> FileSystem fs = path.getFileSystem(new Configuration());
>> path = fs.makeQualified(path);
>> FileContext fc = FileContext.getFileContext(path.toUri(),new
>> Configuration());
>>
>>
>>
>> Thanks & Regards
>> Brahma Reddy Battula
>> 
>> From: Chris Nauroth [cnaur...@hortonworks.com]
>> Sent: Friday, May 01, 2015 4:32 AM
>> To: mapreduce-...@hadoop.apache.org; common-dev@hadoop.apache.org;
>> yarn-...@hadoop.apache.org; hdfs-...@hadoop.apache.org
>> Subject: Re: Planning Hadoop 2.6.1 release
>>
>> Thank you, Arpit.  In addition, I suggest we include the following:
>>
>> HADOOP-11333. Fix deadlock in DomainSocketWatcher when the notification
>> pipe is full
>> HADOOP-11604. Prevent ConcurrentModificationException while closing domain
>> sockets during shutdown of DomainSocketWatcher thread.
>> HADOOP-11648. Set DomainSocketWatcher thread name explicitly
>> HADOOP-11802. DomainSocketWatcher thread terminates sometimes after there
>> is an I/O error during requestShortCircuitShm
>>
>> HADOOP-11604 and 11648 are not critical by themselves, but they are
>> pre-requisites to getting a clean cherry-pick of 11802, which we believe
>> finally fixes the root cause of this issue.
>>
>>
>> --Chris Nauroth
>>
>>
>>
>>
>> On 4/30/15, 3:55 PM, "Arpit Agarwal"  wrote:
>>
>>  HDFS candidates for back-porting to Hadoop 2.6.1. The first two were
>>> requested in [1].
>>>
>>> HADOOP-11674. oneByteBuf in CryptoInputStream and CryptoOutputStream
>>> should be non static
>>> HADOOP-11710. Make CryptoOutputStream behave like DFSOutputStream wrt
>>> synchronization
>>>
>>> HDFS-7009. Active NN and standby NN have different live nodes.
>>> HDFS-7035. Make adding a new data directory to the DataNode an atomic and
>>> improve error handling
>>> HDFS-7425. NameNode block deletion logging uses incorrect appender.
>>> HDFS-7443. Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate
>>> block files are present in the same volume.
>>> HDFS-7489. Incorrect locking in FsVolumeList#checkDirs can hang datanodes
>>> HDFS-7503. Namenode restart after large deletions can cause slow
>>> processReport.
>>> HDFS-7575. Upgrade should generate a unique storage ID for each volume.
>>> HDFS-7579. Improve log reporting during block report rpc failure.
>>> HDFS-7587. Edit log corruption can happen if append fails with a quota
>>> violation.
>>> HDFS-7596. NameNode should prune dead storages from storageMap.
>>> HDFS-7611. deleteSnapshot and delete of a file can leave orphaned blocks
>>> in the blocksMap on NameNode restart.
>>> HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause
>>> DataNode to register successfully with only one NameNode.
>>> HDFS-7733. NFS: readdir/readdirplus return null directory attribute on
>>> failure.
>>> HDFS-7831. Fix the starting index and end condition of the loop in
>>> FileDiffList.findEarlierSnapshotBlocks().
>>> HDFS-7885. Datanode should not trust the generation stamp provided by
>>> client.
>>> HDFS-7960. The full block report should prune z