Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/601/

[Nov 21, 2017 5:09:16 PM] (haibochen) YARN-7531. ResourceRequest.equal does not 
check
[Nov 21, 2017 6:33:34 PM] (yufei) YARN-7513. Remove the scheduler lock in 
FSAppAttempt.getWeight()
[Nov 22, 2017 4:42:28 AM] (cdouglas) MAPREDUCE-7011.




-1 overall


The following subsystems voted -1:
asflicense findbugs unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
   org.apache.hadoop.yarn.api.records.Resource.getResources() may expose 
internal representation by returning Resource.resources At Resource.java:by 
returning Resource.resources At Resource.java:[line 208] 

Failed junit tests :

   hadoop.security.TestRaceWhenRelogin 
   hadoop.hdfs.TestDistributedFileSystemWithECFile 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure 
   hadoop.hdfs.TestDFSRemove 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120 
   hadoop.hdfs.TestErasureCodingPoliciesWithRandomECPolicy 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210 
   hadoop.hdfs.TestDFSStripedOutputStream 
   hadoop.hdfs.TestDFSStripedOutputStreamWithRandomECPolicy 
   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration 
   hadoop.hdfs.TestGetFileChecksum 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 
   hadoop.hdfs.server.balancer.TestBalancerRPCDelay 
   hadoop.hdfs.TestUnsetAndChangeDirectoryEcPolicy 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090 
   hadoop.hdfs.TestErasureCodingPolicies 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060 
   hadoop.fs.TestUnbuffer 
   hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks 
   
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/601/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/601/artifact/out/diff-compile-javac-root.txt
  [276K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/601/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/601/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/601/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/601/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/601/artifact/out/whitespace-eol.txt
  [8.8M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/601/artifact/out/whitespace-tabs.txt
  [288K]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/601/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html
  [8.0K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/601/artifact/out/diff-javadoc-javadoc-root.txt
  [760K]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/601/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
  [156K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/601/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [492K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/601/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [80K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/601/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
  [84K]

   asflicense:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/601/artifact/out/patch-asflicense-problems.txt
  [4.0K]

Powered by Apache Yetus 0.7.0-SNAPSHOT   http://yetus.apache.org

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-7559) TestNodeLabelContainerAllocation failing intermittently

[Arun Suresh (JIRA)]

Arun Suresh created YARN-7559:
-

 Summary: TestNodeLabelContainerAllocation failing intermittently
 Key: YARN-7559
 URL: https://issues.apache.org/jira/browse/YARN-7559
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Arun Suresh






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-7558) YARN log command fails to get logs for running containers if the url authentication is enabled.

[Xuan Gong (JIRA)]

Xuan Gong created YARN-7558:
---

 Summary: YARN log command fails to get logs for running containers 
if the url authentication is enabled.
 Key: YARN-7558
 URL: https://issues.apache.org/jira/browse/YARN-7558
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Xuan Gong






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC3)

[Konstantin Shvachko]

Thanks for the links, Arun. I see I missed this part.
Glad the decision was conscious.
Thanks for the release.
--Konstantin

On Wed, Nov 22, 2017 at 11:39 AM, Arun Suresh  wrote:

> Hello Konstantin
>
> We actually did discuss a bit about this:
> https://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg28407.html
> https://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg28403.html
>
> We finally decided to add the following disclaimer on the release page (
> https://hadoop.apache.org/releases.html):
> "Please note: Although this release has been tested on fairly large
> clusters, production users can wait for a subsequent point release which
> will contain fixes from further stabilization and downstream adoption."
>
> Hope this suffices.
>
>
>
>
> On Wed, Nov 22, 2017 at 10:30 AM, Konstantin Shvachko <
> shv.had...@gmail.com> wrote:
>
>> Hey guys,
>>
>> I don't think this has been discussed, pardon if it was.
>> As it stands today hadoop 2.9.0 is marked as stable release. Isn't that
>> deceptive for users?
>> Not to diminish the quality and not to understate the effort, which was
>> huge and very much appreciated.
>> But it is the first in the series, with quite a few experimental features,
>> and big delta from the last stable, ...
>>
>> Thanks,
>> --Konstantin
>>
>>
>> On Fri, Nov 17, 2017 at 1:56 PM, Subru Krishnan  wrote:
>>
>> > Wrapping up the vote with my +1.
>> >
>> > Deployed RC3 on a federated YARN cluster with 6 sub-clusters:
>> > - ran multiple sample jobs
>> > - enabled opportunistic containers and submitted more samples
>> > - configured HDFS federation and reran jobs.
>> >
>> >
>> > With 13 binding +1s and 7 non-binding +1s and no +/-1s, pleased to
>> announce
>> > the vote is passed successfully.
>> >
>> > Thanks to the many of you who contributed to the release and made this
>> > possible and to everyone in this thread who took the time/effort to
>> > validate and vote!
>> >
>> > We’ll push the release bits and send out an announcement for 2.9.0 soon.
>> >
>> > Cheers,
>> > Subru
>> >
>> > On Fri, Nov 17, 2017 at 12:41 PM Eric Payne 
>> > wrote:
>> >
>> > > Thanks Arun and Subru for the hard work on this release.
>> > >
>> > > +1 (binding)
>> > >
>> > > Built from source and stood up a pseudo cluster with 4 NMs
>> > >
>> > > Tested the following:
>> > >
>> > > o User limits are honored during In-queue preemption
>> > >
>> > > o Priorities are honored during In-queue preemption
>> > >
>> > > o Can kill applications from the command line
>> > >
>> > > o Users with different weights are assigned resources proportional to
>> > > their weights.
>> > >
>> > > Thanks,
>> > > -Eric Payne
>> > >
>> > >
>> > > --
>> > > *From:* Arun Suresh 
>> > > *To:* yarn-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org;
>> > Hadoop
>> > > Common ; Hdfs-dev <
>> > > hdfs-...@hadoop.apache.org>
>> > > *Cc:* Subramaniam Krishnan 
>> > > *Sent:* Monday, November 13, 2017 6:10 PM
>> > >
>> > > *Subject:* [VOTE] Release Apache Hadoop 2.9.0 (RC3)
>> > >
>> > > Hi Folks,
>> > >
>> > > Apache Hadoop 2.9.0 is the first release of Hadoop 2.9 line and will
>> be
>> > the
>> > > starting release for Apache Hadoop 2.9.x line - it includes 30 New
>> > Features
>> > > with 500+ subtasks, 407 Improvements, 790 Bug fixes new fixed issues
>> > since
>> > > 2.8.2.
>> > >
>> > > More information about the 2.9.0 release plan can be found here:
>> > > *
>> > > https://cwiki.apache.org/confluence/display/HADOOP/
>> > Roadmap#Roadmap-Version2.9
>> > > <
>> > > https://cwiki.apache.org/confluence/display/HADOOP/
>> > Roadmap#Roadmap-Version2.9
>> > > >*
>> > >
>> > > New RC is available at: *
>> > > https://home.apache.org/~asuresh/hadoop-2.9.0-RC3/
>> > > *
>> > >
>> > > The RC tag in git is: release-2.9.0-RC3, and the latest commit id is:
>> > > 756ebc8394e473ac25feac05fa493f6d612e6c50.
>> > >
>> > > The maven artifacts are available via repository.apache.org at:
>> > > <
>> > > https://www.google.com/url?q=https%3A%2F%2Frepository.
>> > apache.org%2Fcontent%2Frepositories%2Forgapachehadoop-1066=D&
>> > sntz=1=AFQjCNFcern4uingMV_sEreko_zeLlgdlg
>> > > >*https://repository.apache.org/content/repositories/
>> > orgapachehadoop-1068/
>> > > > > orgapachehadoop-1068/
>> > > >*
>> > >
>> > > We are carrying over the votes from the previous RC given that the
>> delta
>> > is
>> > > the license fix.
>> > >
>> > > Given the above - we are also going to stick with the original
>> deadline
>> > for
>> > > the vote : ending on Friday 17th November 2017 2pm PT time.
>> > >
>> > > Thanks,
>> > > -Arun/Subru
>> > >
>> > >
>> > >
>> >
>>
>
>

[jira] [Created] (YARN-7557) It should be possible to specify resource types in the fair scheduler increment value

[Daniel Templeton (JIRA)]

Daniel Templeton created YARN-7557:
--

 Summary: It should be possible to specify resource types in the 
fair scheduler increment value
 Key: YARN-7557
 URL: https://issues.apache.org/jira/browse/YARN-7557
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: fairscheduler
Affects Versions: 3.0.0-beta1
Reporter: Daniel Templeton
Priority: Critical






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Apache Hadoop 2.8.3 Release Plan

[Junping Du]

Thanks Konstantin and Weiwei for calling attention. These two HDFS issues looks 
to be important to be fixed which are on my radar now.

I will hold on RC cut until we figure them out.


Thanks,


Junping



From: Weiwei Yang 
Sent: Tuesday, November 21, 2017 7:45 PM
To: Junping Du; Konstantin Shvachko
Cc: Wangda Tan; Andrew Wang; Zheng, Kai; common-...@hadoop.apache.org; 
hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.3 Release Plan

Agree with Konstantin. This two issues has been opened for a while but could 
not reach a consensus on the fix, hope this gets enough attention from the 
community and get them resolved.

Thanks

--
Weiwei

On 22 Nov 2017, 11:18 AM +0800, Konstantin Shvachko , 
wrote:
I would consider these two blockers for 2.8.3 as they crash NN:

https://issues.apache.org/jira/browse/HDFS-12638
https://issues.apache.org/jira/browse/HDFS-12832

Thanks,
--Konstantin

On Tue, Nov 21, 2017 at 11:16 AM, Junping Du  wrote:

Thanks Andrew and Wangda for comments!

To me, an improvement with 17 patches is not a big problem as this is
self-contained and I didn't see a single line of delete/update on existing
code - well, arguably, patches with only adding code can also have big
impact but not likely the case here.

While the dependency discussions on HADOOP-14964 are still going on, I
will leave the decision to JIRA discussion based on which approach we will
choose(shaded?) and impact. If we cannot make consensus in short term,
probably we have to miss this in 2.8.3 release.


Okay. Last call for blocker/critical fixes landing on branch-2.8.3. RC0
will get cut-off shortly.



Thanks,


Junping



From: Wangda Tan > wrote:
The Aliyun OSS code isn't a small improvement. If you look at Sammi's
comment
,
it's
a 17 patch series that is being backported in one shot. What we're talking
about is equivalent to merging a feature branch in a maintenance release. I
see that Kai and Chris are having a discussion about the dependency
changes, which indicates this is not a zero-risk change either. We really
should not be changing dependency versions in a maintenance unless it's
because of a bug.

It's unfortunate from a timing perspective that this missed 2.9.0, but I
still think it should wait for the next minor. Merging a feature into a
maintenance release sets the wrong precedent.

Best,
Andrew

On Tue, Nov 21, 2017 at 1:08 AM, Junping Du > wrote:

Thanks Kai for calling out this feature/improvement for attention and
Andrew for comments.


While I agree that maintenance release should focus on important bug fix
only, I doubt we have strict rules to disallow any features/improvements
to
land on maint release especially when those are small footprint or low
impact on existing code/features. In practice, we indeed has 77 new
features/improvements in latest 2.7.3 and 2.7.4 release.


Back to HADOOP-14964, I did a quick check and it looks like case here
belongs to self-contained improvement that has very low impact on
existing
code base, so I am OK with the improvement get landed on branch-2.8 in
case
it is well reviewed and tested.


However, as RM of branch-2.8, I have two concerns to accept it in our
2.8.3 release:

1. Timing - as I mentioned in beginning, the main purpose of 2.8.3 are
for
several critical bug fixes and we should target to release it very soon -
my current plan is to cut RC out within this week inline with 

[jira] [Created] (YARN-7556) Fair scheduler configuration should allow resource types in the minResources and maxResources properties

[Daniel Templeton (JIRA)]

Daniel Templeton created YARN-7556:
--

 Summary: Fair scheduler configuration should allow resource types 
in the minResources and maxResources properties
 Key: YARN-7556
 URL: https://issues.apache.org/jira/browse/YARN-7556
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: fairscheduler
Affects Versions: 3.0.0-beta1
Reporter: Daniel Templeton
Assignee: Daniel Templeton
Priority: Critical






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-7555) Support multiple resource types in YARN native services

[Wangda Tan (JIRA)]

Wangda Tan created YARN-7555:


 Summary: Support multiple resource types in YARN native services
 Key: YARN-7555
 URL: https://issues.apache.org/jira/browse/YARN-7555
 Project: Hadoop YARN
  Issue Type: Bug
  Components: yarn-native-services
Reporter: Wangda Tan
Assignee: Wangda Tan
Priority: Critical


We need to support specifying multiple resource type in addition to memory/cpu 
in YARN native services



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Apache YARN Committers & Contributors Meetup #5

[Daniel Templeton]

We're long past due for another contributors meetup.  Cloudera is 
excited to host this event at our new Palo Alto Galactic Headquarters 
(https://www.google.com/maps/place/Cloudera+Galactic+HQ/@37.4254615,-122.1413431,17z) 
on December 6th from 2pm-5pm PST.  We will have a Google Hangout set up 
so that remote participants can dial in.  We will also provide drinks 
and snacks.


Our agenda will be roughly:

* Celebrate Hadoop 3.0.0, identify any loose ends, and talk about 3.0.1
* Planning around Hadoop 3.1.0
* Planning around the branch-2 releases, Hadoop 2.10.0, etc.
* Key signing
* Plan for the next bug bash

Hope to see you there!
Daniel

PS: I wasn't able to figure out how to create an event in our meetup 
group for this.  Anyone want to give me a pointer or set it up?


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC3)

[Arun Suresh]

Hello Konstantin

We actually did discuss a bit about this:
https://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg28407.html
https://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg28403.html

We finally decided to add the following disclaimer on the release page (
https://hadoop.apache.org/releases.html):
"Please note: Although this release has been tested on fairly large
clusters, production users can wait for a subsequent point release which
will contain fixes from further stabilization and downstream adoption."

Hope this suffices.




On Wed, Nov 22, 2017 at 10:30 AM, Konstantin Shvachko 
wrote:

> Hey guys,
>
> I don't think this has been discussed, pardon if it was.
> As it stands today hadoop 2.9.0 is marked as stable release. Isn't that
> deceptive for users?
> Not to diminish the quality and not to understate the effort, which was
> huge and very much appreciated.
> But it is the first in the series, with quite a few experimental features,
> and big delta from the last stable, ...
>
> Thanks,
> --Konstantin
>
>
> On Fri, Nov 17, 2017 at 1:56 PM, Subru Krishnan  wrote:
>
> > Wrapping up the vote with my +1.
> >
> > Deployed RC3 on a federated YARN cluster with 6 sub-clusters:
> > - ran multiple sample jobs
> > - enabled opportunistic containers and submitted more samples
> > - configured HDFS federation and reran jobs.
> >
> >
> > With 13 binding +1s and 7 non-binding +1s and no +/-1s, pleased to
> announce
> > the vote is passed successfully.
> >
> > Thanks to the many of you who contributed to the release and made this
> > possible and to everyone in this thread who took the time/effort to
> > validate and vote!
> >
> > We’ll push the release bits and send out an announcement for 2.9.0 soon.
> >
> > Cheers,
> > Subru
> >
> > On Fri, Nov 17, 2017 at 12:41 PM Eric Payne 
> > wrote:
> >
> > > Thanks Arun and Subru for the hard work on this release.
> > >
> > > +1 (binding)
> > >
> > > Built from source and stood up a pseudo cluster with 4 NMs
> > >
> > > Tested the following:
> > >
> > > o User limits are honored during In-queue preemption
> > >
> > > o Priorities are honored during In-queue preemption
> > >
> > > o Can kill applications from the command line
> > >
> > > o Users with different weights are assigned resources proportional to
> > > their weights.
> > >
> > > Thanks,
> > > -Eric Payne
> > >
> > >
> > > --
> > > *From:* Arun Suresh 
> > > *To:* yarn-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org;
> > Hadoop
> > > Common ; Hdfs-dev <
> > > hdfs-...@hadoop.apache.org>
> > > *Cc:* Subramaniam Krishnan 
> > > *Sent:* Monday, November 13, 2017 6:10 PM
> > >
> > > *Subject:* [VOTE] Release Apache Hadoop 2.9.0 (RC3)
> > >
> > > Hi Folks,
> > >
> > > Apache Hadoop 2.9.0 is the first release of Hadoop 2.9 line and will be
> > the
> > > starting release for Apache Hadoop 2.9.x line - it includes 30 New
> > Features
> > > with 500+ subtasks, 407 Improvements, 790 Bug fixes new fixed issues
> > since
> > > 2.8.2.
> > >
> > > More information about the 2.9.0 release plan can be found here:
> > > *
> > > https://cwiki.apache.org/confluence/display/HADOOP/
> > Roadmap#Roadmap-Version2.9
> > > <
> > > https://cwiki.apache.org/confluence/display/HADOOP/
> > Roadmap#Roadmap-Version2.9
> > > >*
> > >
> > > New RC is available at: *
> > > https://home.apache.org/~asuresh/hadoop-2.9.0-RC3/
> > > *
> > >
> > > The RC tag in git is: release-2.9.0-RC3, and the latest commit id is:
> > > 756ebc8394e473ac25feac05fa493f6d612e6c50.
> > >
> > > The maven artifacts are available via repository.apache.org at:
> > > <
> > > https://www.google.com/url?q=https%3A%2F%2Frepository.
> > apache.org%2Fcontent%2Frepositories%2Forgapachehadoop-1066=D&
> > sntz=1=AFQjCNFcern4uingMV_sEreko_zeLlgdlg
> > > >*https://repository.apache.org/content/repositories/
> > orgapachehadoop-1068/
> > >  > orgapachehadoop-1068/
> > > >*
> > >
> > > We are carrying over the votes from the previous RC given that the
> delta
> > is
> > > the license fix.
> > >
> > > Given the above - we are also going to stick with the original deadline
> > for
> > > the vote : ending on Friday 17th November 2017 2pm PT time.
> > >
> > > Thanks,
> > > -Arun/Subru
> > >
> > >
> > >
> >
>

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC3)

[Konstantin Shvachko]

Hey guys,

I don't think this has been discussed, pardon if it was.
As it stands today hadoop 2.9.0 is marked as stable release. Isn't that
deceptive for users?
Not to diminish the quality and not to understate the effort, which was
huge and very much appreciated.
But it is the first in the series, with quite a few experimental features,
and big delta from the last stable, ...

Thanks,
--Konstantin


On Fri, Nov 17, 2017 at 1:56 PM, Subru Krishnan  wrote:

> Wrapping up the vote with my +1.
>
> Deployed RC3 on a federated YARN cluster with 6 sub-clusters:
> - ran multiple sample jobs
> - enabled opportunistic containers and submitted more samples
> - configured HDFS federation and reran jobs.
>
>
> With 13 binding +1s and 7 non-binding +1s and no +/-1s, pleased to announce
> the vote is passed successfully.
>
> Thanks to the many of you who contributed to the release and made this
> possible and to everyone in this thread who took the time/effort to
> validate and vote!
>
> We’ll push the release bits and send out an announcement for 2.9.0 soon.
>
> Cheers,
> Subru
>
> On Fri, Nov 17, 2017 at 12:41 PM Eric Payne 
> wrote:
>
> > Thanks Arun and Subru for the hard work on this release.
> >
> > +1 (binding)
> >
> > Built from source and stood up a pseudo cluster with 4 NMs
> >
> > Tested the following:
> >
> > o User limits are honored during In-queue preemption
> >
> > o Priorities are honored during In-queue preemption
> >
> > o Can kill applications from the command line
> >
> > o Users with different weights are assigned resources proportional to
> > their weights.
> >
> > Thanks,
> > -Eric Payne
> >
> >
> > --
> > *From:* Arun Suresh 
> > *To:* yarn-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org;
> Hadoop
> > Common ; Hdfs-dev <
> > hdfs-...@hadoop.apache.org>
> > *Cc:* Subramaniam Krishnan 
> > *Sent:* Monday, November 13, 2017 6:10 PM
> >
> > *Subject:* [VOTE] Release Apache Hadoop 2.9.0 (RC3)
> >
> > Hi Folks,
> >
> > Apache Hadoop 2.9.0 is the first release of Hadoop 2.9 line and will be
> the
> > starting release for Apache Hadoop 2.9.x line - it includes 30 New
> Features
> > with 500+ subtasks, 407 Improvements, 790 Bug fixes new fixed issues
> since
> > 2.8.2.
> >
> > More information about the 2.9.0 release plan can be found here:
> > *
> > https://cwiki.apache.org/confluence/display/HADOOP/
> Roadmap#Roadmap-Version2.9
> > <
> > https://cwiki.apache.org/confluence/display/HADOOP/
> Roadmap#Roadmap-Version2.9
> > >*
> >
> > New RC is available at: *
> > https://home.apache.org/~asuresh/hadoop-2.9.0-RC3/
> > *
> >
> > The RC tag in git is: release-2.9.0-RC3, and the latest commit id is:
> > 756ebc8394e473ac25feac05fa493f6d612e6c50.
> >
> > The maven artifacts are available via repository.apache.org at:
> > <
> > https://www.google.com/url?q=https%3A%2F%2Frepository.
> apache.org%2Fcontent%2Frepositories%2Forgapachehadoop-1066=D&
> sntz=1=AFQjCNFcern4uingMV_sEreko_zeLlgdlg
> > >*https://repository.apache.org/content/repositories/
> orgapachehadoop-1068/
> >  orgapachehadoop-1068/
> > >*
> >
> > We are carrying over the votes from the previous RC given that the delta
> is
> > the license fix.
> >
> > Given the above - we are also going to stick with the original deadline
> for
> > the vote : ending on Friday 17th November 2017 2pm PT time.
> >
> > Thanks,
> > -Arun/Subru
> >
> >
> >
>

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Brahma Reddy Battula]

bq. (a) The behavior of this command. Clearly, it will conflict with the
MapReduce JHS - only one of them can be started on the same node.

Yes,Since the PID  file will be same.

This problem will not present in branch-2.9 + as pid file will be suffix
with mapred(for JHS)
and
yarn (for AHS)
..i.e
PID is exported on each daemon script.

May be,
1) Change the name
2) Create PID based on the CLASS Name, here applicationhistoryserver and
jobhistoryserver
3) Use same as branch-2.9..i.e suffixing with mapred or yarn


@allen, any thoughts on this..?




On Wed, Nov 22, 2017 at 2:46 AM, Vinod Kumar Vavilapalli  wrote:

> >> - $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver doesn't
> even work. Not just deprecated in favor of timelineserver as was advertised.
> >
> >   This works for me in trunk and the bash code doesn’t appear to
> have changed in a very long time.  Probably something local to your
> install.  (I do notice that the deprecation message says “starting” which
> is awkward when the stop command is given though.)  Also: is the
> deprecation message even true at this point?
>
>
> Sorry, I mischaracterized the problem.
>
> The real issue is that I cannot use this command line when the MapReduce
> JobHistoryServer is already started on the same machine.
>
> ~/tmp/yarn$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver
> WARNING: Use of this script to start YARN daemons is deprecated.
> WARNING: Attempting to execute replacement "yarn --daemon start" instead.
> DEPRECATED: Use of this command to start the timeline server is deprecated.
> Instead use the timelineserver command for it.
> Starting the History Server anyway...
> historyserver is running as process 86156.  Stop it first.
>
> So, it looks like in shell-scripts, there can ever be only one daemon of a
> given name, irrespective of which daemon scripts are invoked.
>
> We need to figure out two things here
>  (a) The behavior of this command. Clearly, it will conflict with the
> MapReduce JHS - only one of them can be started on the same node.
>  (b) We need to figure out if this V1 TimelineService should even be
> support given ATSv2.
>
> @Vrushani / @Rohith / @Varun Saxena et.al, if you are watching, please
> comment on (b).
>
> Thanks
> +Vinod




-- 



--Brahma Reddy Battula

Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/47/

[Nov 20, 2017 4:01:50 PM] (bibinchundatt) YARN-7489. 
ConcurrentModificationException in RMAppImpl#getRMAppMetrics.
[Nov 20, 2017 7:01:15 PM] (bibinchundatt) YARN-7525. Incorrect query parameters 
in cluster nodes REST API
[Nov 20, 2017 10:21:58 PM] (subru) YARN-6128. Add support for AMRMProxy HA. 
(Botong Huang via Subru).
[Nov 21, 2017 1:11:18 AM] (szetszwo) HDFS-12813.  RequestHedgingProxyProvider 
can hide Exception thrown from
[Nov 21, 2017 4:50:49 AM] (aajisaka) HADOOP-15045. ISA-L bulid options are 
documented in branch-2.




-1 overall


The following subsystems voted -1:
asflicense unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Unreaped Processes :

   hadoop-hdfs:19 
   bkjournal:5 
   hadoop-yarn-common:1 
   hadoop-yarn-server-timelineservice:1 
   hadoop-yarn-client:6 
   hadoop-yarn-applications-distributedshell:1 
   hadoop-mapreduce-client-app:2 
   hadoop-mapreduce-client-jobclient:12 
   hadoop-distcp:2 
   hadoop-sls:1 

Failed junit tests :

   hadoop.hdfs.web.TestHttpsFileSystem 
   hadoop.hdfs.web.TestWebHdfsTokens 
   hadoop.hdfs.web.TestWebHdfsFileSystemContract 
   hadoop.hdfs.TestDatanodeReport 
   hadoop.hdfs.web.TestHftpFileSystem 
   hadoop.hdfs.web.TestWebHDFSForHA 
   hadoop.hdfs.web.TestWebHDFSXAttr 
   
hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy 
   hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 
   
hadoop.yarn.applications.distributedshell.TestDistributedShellWithNodeLabels 
   hadoop.tools.TestIntegration 
   hadoop.tools.TestDistCpViewFs 
   hadoop.yarn.sls.TestReservationSystemInvariants 
   hadoop.resourceestimator.solver.impl.TestLpSolver 
   hadoop.resourceestimator.service.TestResourceEstimatorService 

Timed out junit tests :

   org.apache.hadoop.hdfs.TestLeaseRecovery2 
   org.apache.hadoop.hdfs.TestRead 
   org.apache.hadoop.hdfs.TestDFSInotifyEventInputStream 
   org.apache.hadoop.hdfs.TestDatanodeLayoutUpgrade 
   org.apache.hadoop.hdfs.TestFileAppendRestart 
   org.apache.hadoop.hdfs.security.TestDelegationToken 
   org.apache.hadoop.hdfs.web.TestWebHdfsWithRestCsrfPreventionFilter 
   org.apache.hadoop.hdfs.TestDFSMkdirs 
   org.apache.hadoop.hdfs.TestDFSOutputStream 
   org.apache.hadoop.hdfs.web.TestWebHDFS 
   org.apache.hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs 
   org.apache.hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes 
   org.apache.hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs 
   org.apache.hadoop.hdfs.TestDistributedFileSystem 
   org.apache.hadoop.hdfs.TestReplaceDatanodeFailureReplication 
   org.apache.hadoop.hdfs.TestDFSShell 
   org.apache.hadoop.hdfs.web.TestWebHDFSAcl 
   org.apache.hadoop.contrib.bkjournal.TestBootstrapStandbyWithBKJM 
   org.apache.hadoop.contrib.bkjournal.TestBookKeeperJournalManager 
   org.apache.hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   org.apache.hadoop.contrib.bkjournal.TestBookKeeperAsHASharedDir 
   org.apache.hadoop.contrib.bkjournal.TestBookKeeperSpeculativeRead 
   org.apache.hadoop.yarn.webapp.TestWebApp 
   
org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServices
 
   org.apache.hadoop.yarn.client.TestRMFailover 
   org.apache.hadoop.yarn.client.cli.TestYarnCLI 
   org.apache.hadoop.yarn.client.TestApplicationMasterServiceProtocolOnHA 
   org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA 
   org.apache.hadoop.yarn.client.api.impl.TestYarnClient 
   org.apache.hadoop.yarn.client.api.impl.TestAMRMClient 
   
org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell 
   org.apache.hadoop.mapreduce.v2.app.TestStagingCleanup 
   org.apache.hadoop.mapreduce.v2.app.TestJobEndNotifier 
   org.apache.hadoop.mapred.TestClusterMRNotification 
   org.apache.hadoop.mapred.TestMiniMRClasspath 
   org.apache.hadoop.mapred.TestMRCJCFileInputFormat 
   org.apache.hadoop.mapred.TestClusterMapReduceTestCase 
   org.apache.hadoop.mapred.TestMRIntermediateDataEncryption 
   org.apache.hadoop.mapred.TestMRTimelineEventHandling 
   org.apache.hadoop.mapred.join.TestDatamerge 
   org.apache.hadoop.mapred.TestJobName 
   org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath 
   org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers 
   org.apache.hadoop.mapred.TestNetworkedJob 
   org.apache.hadoop.mapred.TestMROpportunisticMaps