Build failed in Jenkins: Hadoop-Yarn-trunk-Java8 #799

2015-12-15 Thread Apache Jenkins Server 
See 

Changes:

[zxu] Update CHANGES.txt to move MAPREDUCE-6436 from YARN to MAPREDUCE

--
[...truncated 5169 lines...]
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestSchedulingUpdate
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.49 sec - in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestSchedulingUpdate
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerEventLog
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.402 sec - in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerEventLog
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
Tests run: 89, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.93 sec - in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestComputeFairShares
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.464 sec - in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestComputeFairShares
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.154 sec - in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestAppRunnability
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.164 sec - in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestAppRunnability
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestSchedulingPolicy
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.631 sec - in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestSchedulingPolicy
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestQueueManager
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.047 sec - in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestQueueManager
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSParentQueue
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.936 sec - in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSParentQueue
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.policies.TestEmptyQueues
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.467 sec - in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.policies.TestEmptyQueues
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.policies.TestDominantResourceFairnessPolicy
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.463 sec - in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.policies.TestDominantResourceFairnessPolicy
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppAttempt
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.881 sec - in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppAttempt
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestQueuePlacementPolicy
Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.686 sec - in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestQueuePlacementPolicy
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option Ma

Hadoop-Yarn-trunk-Java8 - Build # 799 - Failure

2015-12-15 Thread Apache Jenkins Server 
See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/799/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 5362 lines...]
Running 
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.TestRMContainerImpl
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.073 sec - in 
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.TestRMContainerImpl
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running 
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationMasterService
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 49.992 sec - in 
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationMasterService
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
Tests run: 50, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 390.736 sec - 
in org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart

Results :

Tests run: 1255, Failures: 0, Errors: 0, Skipped: 2

[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop YARN  SUCCESS [  3.002 s]
[INFO] Apache Hadoop YARN API  SUCCESS [01:25 min]
[INFO] Apache Hadoop YARN Common . SUCCESS [02:19 min]
[INFO] Apache Hadoop YARN Server . SUCCESS [  0.051 s]
[INFO] Apache Hadoop YARN Server Common .. SUCCESS [ 36.125 s]
[INFO] Apache Hadoop YARN NodeManager  SUCCESS [09:35 min]
[INFO] Apache Hadoop YARN Web Proxy .. SUCCESS [ 15.962 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService .. SUCCESS [03:09 min]
[INFO] Apache Hadoop YARN ResourceManager  FAILURE [57:22 min]
[INFO] Apache Hadoop YARN Server Tests ... SKIPPED
[INFO] Apache Hadoop YARN Client . SKIPPED
[INFO] Apache Hadoop YARN SharedCacheManager . SKIPPED
[INFO] Apache Hadoop YARN Applications ... SKIPPED
[INFO] Apache Hadoop YARN DistributedShell ... SKIPPED
[INFO] Apache Hadoop YARN Unmanaged Am Launcher .. SKIPPED
[INFO] Apache Hadoop YARN Site ... SKIPPED
[INFO] Apache Hadoop YARN Registry ... SKIPPED
[INFO] Apache Hadoop YARN Project  SKIPPED
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 01:14 h
[INFO] Finished at: 2015-12-16T07:52:10+00:00
[INFO] Final Memory: 65M/1005M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on 
project hadoop-yarn-server-resourcemanager: ExecutionException: 
java.lang.RuntimeException: The forked VM terminated without properly saying 
goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd 
/home/jenkins/jenkins-slave/workspace/Hadoop-Yarn-trunk-Java8/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 && /home/jenkins/tools/java/jdk1.8.0/jre/bin/java -Xmx2048m 
-XX:MaxPermSize=768m -XX:+HeapDumpOnOutOfMemoryError -jar 
/home/jenkins/jenkins-slave/workspace/Hadoop-Yarn-trunk-Java8/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/target/surefire/surefirebooter9209041983489341959.jar
 
/home/jenkins/jenkins-slave/workspace/Hadoop-Yarn-trunk-Java8/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/target/surefire/surefire4266442782760814290tmp
 
/home/jenkins/jenkins-slave/workspace/Hadoop-Yarn-trunk-Java8/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/target/surefire/surefire_1849123413283859352667tmp
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :hadoop-yarn-server-resourcemanager
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
Sending e-mails to: yarn-dev@hadoop.apache.org
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

Jenkins build is back to normal : Hadoop-Yarn-trunk #1519

2015-12-15 Thread Apache Jenkins Server 
See

Re: [VOTE] Release Apache Hadoop 2.6.3 RC0

2015-12-15 Thread Akira AJISAKA 

Thanks Junping for starting release process.
I noticed there are duplicated tags:

* release-2.6.3-RC0
* release-2.6.3-RC0.1
* release-2.6.3-RC1

Could you remove RC0.1 and RC1?

Regards,
Akira

On 12/16/15 10:17, yliu wrote:

Thanks Junping, +1.
Download the tarball and deploy a small HDFS/YARN cluster, and verify few
basic functionalities.

Regards,
Yi Liu

On Wed, Dec 16, 2015 at 6:42 AM, Chang Li  wrote:


Thanks Junping, + 1(non binding). Downloaded the tarball, compiled and
built locally. Ran some MR jobs successfully.

Best,
Chang

On Tue, Dec 15, 2015 at 3:17 PM, Wangda Tan  wrote:


Thanks Junping,

+1 (binding). Deploy a cluster locally, run distributed shell and MR job,
both successfully finished.

Regards,
Wangda


On Tue, Dec 15, 2015 at 12:43 PM, Naganarasimha Garla <
naganarasimha...@gmail.com> wrote:


Hi Junping,

+0 (non binding)

Though everything else is working  fine (downloaded the tar ball and
installed single node cluster setup and  verified  few MR jobs),
submission of Unmanaged AM is getting the RM down. YARN-4452 has
already been raised and i am working on it. Will provide the patch for
the trunk and the 2.6.3 version asap

Regards,

+ Naga



Thanks for the work Junping! Downloaded the src tarball. Built locally
and successfully ran
in single node mode with a few map reduce jobs. LGTM.

Li Lu

On Dec 14, 2015, at 04:23, Junping Du
mailto:j...@hortonworks.com>>
wrote:

?Thanks Sarjeet and Tsuyoshi for reporting this. I just fix permission
issue and download
should work now. Please try to download it again. Thanks!


Thanks,


Junping



From: sarjeet singh mailto:sarje...@usc.edu>>
Sent: Sunday, December 13, 2015 6:44 PM
To: common-...@hadoop.apache.org
Cc: mapreduce-...@hadoop.apache.org
mapreduce-...@hadoop.apache.org

;

hdfs-...@hadoop.apache.org;
yarn-dev@hadoop.apache.org;
junping...@apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.6.3 RC0

I am also getting the same error when downloading tar.gz:

"You don't have permission to access
/~junping_du/hadoop-2.6.3-RC0/hadoop-2.6.3-RC0-src.tar.gz
on this server."

- Sarjeet Singh

On Sat, Dec 12, 2015 at 4:17 PM, Tsuyoshi Ozawa
mailto:oz...@apache.org>>
wrote:
Hi Junping,

Thank you for starting the voting.
I cannot access the tar.gz file because of permission error. Could you
check the permission to access the files?

Forbidden
You don't have permission to access
/~junping_du/hadoop-2.6.3-RC0/hadoop-2.6.3-RC0-src.tar.gz
on this server.

Thanks,
- Tsuyoshi

On Sat, Dec 12, 2015 at 9:16 AM, Junping Du
mailto:j...@hortonworks.com>>
wrote:

Hi all developers in hadoop community,
   I've created a release candidate RC0 for Apache Hadoop 2.6.3 (the
next maintenance release
to follow up 2.6.2.) according to email thread of release plan 2.6.3
[1]. Sorry for this RC
coming a bit late as several blocker issues were getting committed
until yesterday. Below
is the details:

The RC is available for validation at:
*http://people.apache.org/~junping_du/hadoop-2.6.3-RC0/
*

The RC tag in git is: release-2.6.3-RC0

The maven artifacts are staged via
repository.apache.org<
http://repository.apache.org>
at:
*

https://repository.apache.org/content/repositories/orgapachehadoop-1025/

?

<

https://repository.apache.org/content/repositories/orgapachehadoop-1025/

*


You can find my public key
at:http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS

Please try the release and vote. The vote will run for the usual 5

days.


Thanks and happy weekend!


Cheers,

Junping


[1]: 2.6.3 release plan: http://markmail.org/thread/nc2jogbgni37vu6y

[jira] [Created] (YARN-4462) Scheduler should prevent certain application from being preempted

2015-12-15 Thread Tao Jie (JIRA) 
Tao Jie created YARN-4462:
-

 Summary: Scheduler should prevent certain application from being 
preempted
 Key: YARN-4462
 URL: https://issues.apache.org/jira/browse/YARN-4462
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: scheduler
Affects Versions: 2.6.0
Reporter: Tao Jie


When scheduler preemption is enabled, applications could be preempted if they 
obtain resource over they should take. 
When a mapreduce application is preempted some resource, it just runs slower. 
However, when the preempted application is a long-run service, such as tomcat 
running in slider, the service would fail.
So we should have a flag for application to indicate the scheduler that those 
application should not be preempted. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : Hadoop-Yarn-trunk-Java8 #798

2015-12-15 Thread Apache Jenkins Server 
See

Re: [VOTE] Release Apache Hadoop 2.6.3 RC0

2015-12-15 Thread yliu 
Thanks Junping, +1.
Download the tarball and deploy a small HDFS/YARN cluster, and verify few
basic functionalities.

Regards,
Yi Liu

On Wed, Dec 16, 2015 at 6:42 AM, Chang Li  wrote:

> Thanks Junping, + 1(non binding). Downloaded the tarball, compiled and
> built locally. Ran some MR jobs successfully.
>
> Best,
> Chang
>
> On Tue, Dec 15, 2015 at 3:17 PM, Wangda Tan  wrote:
>
> > Thanks Junping,
> >
> > +1 (binding). Deploy a cluster locally, run distributed shell and MR job,
> > both successfully finished.
> >
> > Regards,
> > Wangda
> >
> >
> > On Tue, Dec 15, 2015 at 12:43 PM, Naganarasimha Garla <
> > naganarasimha...@gmail.com> wrote:
> >
> > > Hi Junping,
> > >
> > > +0 (non binding)
> > >
> > > Though everything else is working  fine (downloaded the tar ball and
> > > installed single node cluster setup and  verified  few MR jobs),
> > > submission of Unmanaged AM is getting the RM down. YARN-4452 has
> > > already been raised and i am working on it. Will provide the patch for
> > > the trunk and the 2.6.3 version asap
> > >
> > > Regards,
> > >
> > > + Naga
> > >
> > > 
> > >
> > > Thanks for the work Junping! Downloaded the src tarball. Built locally
> > > and successfully ran
> > > in single node mode with a few map reduce jobs. LGTM.
> > >
> > > Li Lu
> > >
> > > On Dec 14, 2015, at 04:23, Junping Du
> > > mailto:j...@hortonworks.com>>
> > > wrote:
> > >
> > > ?Thanks Sarjeet and Tsuyoshi for reporting this. I just fix permission
> > > issue and download
> > > should work now. Please try to download it again. Thanks!
> > >
> > >
> > > Thanks,
> > >
> > >
> > > Junping
> > >
> > >
> > > 
> > > From: sarjeet singh mailto:sarje...@usc.edu>>
> > > Sent: Sunday, December 13, 2015 6:44 PM
> > > To: common-...@hadoop.apache.org
> > > Cc: mapreduce-...@hadoop.apache.org > mapreduce-...@hadoop.apache.org
> > > >;
> > > hdfs-...@hadoop.apache.org;
> > > yarn-dev@hadoop.apache.org;
> > > junping...@apache.org
> > > Subject: Re: [VOTE] Release Apache Hadoop 2.6.3 RC0
> > >
> > > I am also getting the same error when downloading tar.gz:
> > >
> > > "You don't have permission to access
> > > /~junping_du/hadoop-2.6.3-RC0/hadoop-2.6.3-RC0-src.tar.gz
> > > on this server."
> > >
> > > - Sarjeet Singh
> > >
> > > On Sat, Dec 12, 2015 at 4:17 PM, Tsuyoshi Ozawa
> > > mailto:oz...@apache.org>>
> > > wrote:
> > > Hi Junping,
> > >
> > > Thank you for starting the voting.
> > > I cannot access the tar.gz file because of permission error. Could you
> > > check the permission to access the files?
> > >
> > > Forbidden
> > > You don't have permission to access
> > > /~junping_du/hadoop-2.6.3-RC0/hadoop-2.6.3-RC0-src.tar.gz
> > > on this server.
> > >
> > > Thanks,
> > > - Tsuyoshi
> > >
> > > On Sat, Dec 12, 2015 at 9:16 AM, Junping Du
> > > mailto:j...@hortonworks.com> > > j...@hortonworks.com>>
> > > wrote:
> > >
> > > Hi all developers in hadoop community,
> > >   I've created a release candidate RC0 for Apache Hadoop 2.6.3 (the
> > > next maintenance release
> > > to follow up 2.6.2.) according to email thread of release plan 2.6.3
> > > [1]. Sorry for this RC
> > > coming a bit late as several blocker issues were getting committed
> > > until yesterday. Below
> > > is the details:
> > >
> > > The RC is available for validation at:
> > > *http://people.apache.org/~junping_du/hadoop-2.6.3-RC0/
> > > *
> > >
> > > The RC tag in git is: release-2.6.3-RC0
> > >
> > > The maven artifacts are staged via
> > > repository.apache.org<
> > > http://repository.apache.org>
> > > at:
> > > *
> > https://repository.apache.org/content/repositories/orgapachehadoop-1025/
> ?
> > > <
> > https://repository.apache.org/content/repositories/orgapachehadoop-1025/
> > > >*
> > >
> > > You can find my public key
> > > at:http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
> > >
> > > Please try the release and vote. The vote will run for the usual 5
> days.
> > >
> > > Thanks and happy weekend!
> > >
> > >
> > > Cheers,
> > >
> > > Junping
> > >
> > >
> > > [1]: 2.6.3 release plan: http://markmail.org/thread/nc2jogbgni37vu6y
> > >
> >
>

Build failed in Jenkins: Hadoop-Yarn-trunk #1518

2015-12-15 Thread Apache Jenkins Server 
See 

Changes:

[wang] HDFS-8894. Set SO_KEEPALIVE on DN server sockets. Contributed by Kanaka

[umamahesh] HADOOP-12639. Improve JavaDoc for getTrimmedStrings. (BELUGA BEHR 
via

--
[...truncated 9208 lines...]
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.87 sec - in 
org.apache.hadoop.yarn.server.resourcemanager.nodelabels.TestRMNodeLabelsManager
Running 
org.apache.hadoop.yarn.server.resourcemanager.placement.TestUserGroupMappingPlacementRule
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.911 sec - in 
org.apache.hadoop.yarn.server.resourcemanager.placement.TestUserGroupMappingPlacementRule
Running org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.527 sec - in 
org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
Running 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerApplicationAttempt
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.183 sec - in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerApplicationAttempt
Running 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler
Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.302 sec - 
in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler
Running 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerHealth
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.68 sec - in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerHealth
Running 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerUtils
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.524 sec - in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerUtils
Running 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.policy.TestFairOrderingPolicy
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.604 sec - in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.policy.TestFairOrderingPolicy
Running 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.policy.TestFifoOrderingPolicy
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.565 sec - in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.policy.TestFifoOrderingPolicy
Running 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.TestResourceUsage
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.616 sec - in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.TestResourceUsage
Running 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.TestAbstractYarnScheduler
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.967 sec - 
in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.TestAbstractYarnScheduler
Running 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 288.569 sec - 
in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
Running 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestChildQueueOrder
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.193 sec - in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestChildQueueOrder
Running 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 67.374 sec - 
in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing
Running 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue
Tests run: 30, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.635 sec - 
in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue
Running 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueMappings
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.721 sec - in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueMappings
Running 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerQueueACLs
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.553 sec - in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerQueueACLs
Running 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 63.007 sec - in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation
Running 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationLimits
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elaps

Hadoop-Yarn-trunk - Build # 1518 - Still Failing

2015-12-15 Thread Apache Jenkins Server 
See https://builds.apache.org/job/Hadoop-Yarn-trunk/1518/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 9401 lines...]
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.699 sec - in 
org.apache.hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector
Running 
org.apache.hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 186.866 sec - 
in 
org.apache.hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter

Results :

Tests in error: 
  TestApplicationACLs.setup:152 » YarnRuntime 
java.lang.reflect.InvocationTarget...
  TestApplicationACLs.tearDown:164 » NoClassDefFound 
org/apache/hadoop/service/S...

Tests run: 1290, Failures: 0, Errors: 2, Skipped: 2

[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop YARN  SUCCESS [  4.629 s]
[INFO] Apache Hadoop YARN API  SUCCESS [04:21 min]
[INFO] Apache Hadoop YARN Common . SUCCESS [03:13 min]
[INFO] Apache Hadoop YARN Server . SUCCESS [  0.094 s]
[INFO] Apache Hadoop YARN Server Common .. SUCCESS [ 50.390 s]
[INFO] Apache Hadoop YARN NodeManager  SUCCESS [10:03 min]
[INFO] Apache Hadoop YARN Web Proxy .. SUCCESS [ 28.261 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService .. SUCCESS [04:15 min]
[INFO] Apache Hadoop YARN ResourceManager  FAILURE [  01:06 h]
[INFO] Apache Hadoop YARN Server Tests ... SKIPPED
[INFO] Apache Hadoop YARN Client . SKIPPED
[INFO] Apache Hadoop YARN SharedCacheManager . SKIPPED
[INFO] Apache Hadoop YARN Applications ... SKIPPED
[INFO] Apache Hadoop YARN DistributedShell ... SKIPPED
[INFO] Apache Hadoop YARN Unmanaged Am Launcher .. SKIPPED
[INFO] Apache Hadoop YARN Site ... SKIPPED
[INFO] Apache Hadoop YARN Registry ... SKIPPED
[INFO] Apache Hadoop YARN Project  SKIPPED
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 01:29 h
[INFO] Finished at: 2015-12-16T00:47:05+00:00
[INFO] Final Memory: 61M/522M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on 
project hadoop-yarn-server-resourcemanager: There are test failures.
[ERROR] 
[ERROR] Please refer to 
/home/jenkins/jenkins-slave/workspace/Hadoop-Yarn-trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/target/surefire-reports
 for the individual test results.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :hadoop-yarn-server-resourcemanager
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
Sending e-mails to: yarn-dev@hadoop.apache.org
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
2 tests failed.
FAILED:  
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs.org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs

Error Message:
java.lang.reflect.InvocationTargetException

Stack Trace:
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: 
java.lang.reflect.InvocationTargetException
at 
org.apache.hadoop.net.NetUtils.getSocketFactoryFromProperty(NetUtils.java:143)
at 
org.apache.hadoop.net.NetUtils.getDefaultSocketFactory(NetUtils.java:123)
at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:645)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:602)
at 
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.(ApplicationClientProtocolPBClientImpl.java:175)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.jav

Re: [VOTE] Release Apache Hadoop 2.6.3 RC0

2015-12-15 Thread Chang Li 
Thanks Junping, + 1(non binding). Downloaded the tarball, compiled and
built locally. Ran some MR jobs successfully.

Best,
Chang

On Tue, Dec 15, 2015 at 3:17 PM, Wangda Tan  wrote:

> Thanks Junping,
>
> +1 (binding). Deploy a cluster locally, run distributed shell and MR job,
> both successfully finished.
>
> Regards,
> Wangda
>
>
> On Tue, Dec 15, 2015 at 12:43 PM, Naganarasimha Garla <
> naganarasimha...@gmail.com> wrote:
>
> > Hi Junping,
> >
> > +0 (non binding)
> >
> > Though everything else is working  fine (downloaded the tar ball and
> > installed single node cluster setup and  verified  few MR jobs),
> > submission of Unmanaged AM is getting the RM down. YARN-4452 has
> > already been raised and i am working on it. Will provide the patch for
> > the trunk and the 2.6.3 version asap
> >
> > Regards,
> >
> > + Naga
> >
> > 
> >
> > Thanks for the work Junping! Downloaded the src tarball. Built locally
> > and successfully ran
> > in single node mode with a few map reduce jobs. LGTM.
> >
> > Li Lu
> >
> > On Dec 14, 2015, at 04:23, Junping Du
> > mailto:j...@hortonworks.com>>
> > wrote:
> >
> > ?Thanks Sarjeet and Tsuyoshi for reporting this. I just fix permission
> > issue and download
> > should work now. Please try to download it again. Thanks!
> >
> >
> > Thanks,
> >
> >
> > Junping
> >
> >
> > 
> > From: sarjeet singh mailto:sarje...@usc.edu>>
> > Sent: Sunday, December 13, 2015 6:44 PM
> > To: common-...@hadoop.apache.org
> > Cc: mapreduce-...@hadoop.apache.org mapreduce-...@hadoop.apache.org
> > >;
> > hdfs-...@hadoop.apache.org;
> > yarn-dev@hadoop.apache.org;
> > junping...@apache.org
> > Subject: Re: [VOTE] Release Apache Hadoop 2.6.3 RC0
> >
> > I am also getting the same error when downloading tar.gz:
> >
> > "You don't have permission to access
> > /~junping_du/hadoop-2.6.3-RC0/hadoop-2.6.3-RC0-src.tar.gz
> > on this server."
> >
> > - Sarjeet Singh
> >
> > On Sat, Dec 12, 2015 at 4:17 PM, Tsuyoshi Ozawa
> > mailto:oz...@apache.org>>
> > wrote:
> > Hi Junping,
> >
> > Thank you for starting the voting.
> > I cannot access the tar.gz file because of permission error. Could you
> > check the permission to access the files?
> >
> > Forbidden
> > You don't have permission to access
> > /~junping_du/hadoop-2.6.3-RC0/hadoop-2.6.3-RC0-src.tar.gz
> > on this server.
> >
> > Thanks,
> > - Tsuyoshi
> >
> > On Sat, Dec 12, 2015 at 9:16 AM, Junping Du
> > mailto:j...@hortonworks.com> > j...@hortonworks.com>>
> > wrote:
> >
> > Hi all developers in hadoop community,
> >   I've created a release candidate RC0 for Apache Hadoop 2.6.3 (the
> > next maintenance release
> > to follow up 2.6.2.) according to email thread of release plan 2.6.3
> > [1]. Sorry for this RC
> > coming a bit late as several blocker issues were getting committed
> > until yesterday. Below
> > is the details:
> >
> > The RC is available for validation at:
> > *http://people.apache.org/~junping_du/hadoop-2.6.3-RC0/
> > *
> >
> > The RC tag in git is: release-2.6.3-RC0
> >
> > The maven artifacts are staged via
> > repository.apache.org<
> > http://repository.apache.org>
> > at:
> > *
> https://repository.apache.org/content/repositories/orgapachehadoop-1025/?
> > <
> https://repository.apache.org/content/repositories/orgapachehadoop-1025/
> > >*
> >
> > You can find my public key
> > at:http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
> >
> > Please try the release and vote. The vote will run for the usual 5 days.
> >
> > Thanks and happy weekend!
> >
> >
> > Cheers,
> >
> > Junping
> >
> >
> > [1]: 2.6.3 release plan: http://markmail.org/thread/nc2jogbgni37vu6y
> >
>

Re: [VOTE] Release Apache Hadoop 2.6.3 RC0

2015-12-15 Thread Wangda Tan 
Thanks Junping,

+1 (binding). Deploy a cluster locally, run distributed shell and MR job,
both successfully finished.

Regards,
Wangda


On Tue, Dec 15, 2015 at 12:43 PM, Naganarasimha Garla <
naganarasimha...@gmail.com> wrote:

> Hi Junping,
>
> +0 (non binding)
>
> Though everything else is working  fine (downloaded the tar ball and
> installed single node cluster setup and  verified  few MR jobs),
> submission of Unmanaged AM is getting the RM down. YARN-4452 has
> already been raised and i am working on it. Will provide the patch for
> the trunk and the 2.6.3 version asap
>
> Regards,
>
> + Naga
>
> 
>
> Thanks for the work Junping! Downloaded the src tarball. Built locally
> and successfully ran
> in single node mode with a few map reduce jobs. LGTM.
>
> Li Lu
>
> On Dec 14, 2015, at 04:23, Junping Du
> mailto:j...@hortonworks.com>>
> wrote:
>
> ?Thanks Sarjeet and Tsuyoshi for reporting this. I just fix permission
> issue and download
> should work now. Please try to download it again. Thanks!
>
>
> Thanks,
>
>
> Junping
>
>
> 
> From: sarjeet singh mailto:sarje...@usc.edu>>
> Sent: Sunday, December 13, 2015 6:44 PM
> To: common-...@hadoop.apache.org
> Cc: mapreduce-...@hadoop.apache.org >;
> hdfs-...@hadoop.apache.org;
> yarn-dev@hadoop.apache.org;
> junping...@apache.org
> Subject: Re: [VOTE] Release Apache Hadoop 2.6.3 RC0
>
> I am also getting the same error when downloading tar.gz:
>
> "You don't have permission to access
> /~junping_du/hadoop-2.6.3-RC0/hadoop-2.6.3-RC0-src.tar.gz
> on this server."
>
> - Sarjeet Singh
>
> On Sat, Dec 12, 2015 at 4:17 PM, Tsuyoshi Ozawa
> mailto:oz...@apache.org>>
> wrote:
> Hi Junping,
>
> Thank you for starting the voting.
> I cannot access the tar.gz file because of permission error. Could you
> check the permission to access the files?
>
> Forbidden
> You don't have permission to access
> /~junping_du/hadoop-2.6.3-RC0/hadoop-2.6.3-RC0-src.tar.gz
> on this server.
>
> Thanks,
> - Tsuyoshi
>
> On Sat, Dec 12, 2015 at 9:16 AM, Junping Du
> mailto:j...@hortonworks.com> j...@hortonworks.com>>
> wrote:
>
> Hi all developers in hadoop community,
>   I've created a release candidate RC0 for Apache Hadoop 2.6.3 (the
> next maintenance release
> to follow up 2.6.2.) according to email thread of release plan 2.6.3
> [1]. Sorry for this RC
> coming a bit late as several blocker issues were getting committed
> until yesterday. Below
> is the details:
>
> The RC is available for validation at:
> *http://people.apache.org/~junping_du/hadoop-2.6.3-RC0/
> *
>
> The RC tag in git is: release-2.6.3-RC0
>
> The maven artifacts are staged via
> repository.apache.org<
> http://repository.apache.org>
> at:
> *https://repository.apache.org/content/repositories/orgapachehadoop-1025/?
>  >*
>
> You can find my public key
> at:http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
>
> Please try the release and vote. The vote will run for the usual 5 days.
>
> Thanks and happy weekend!
>
>
> Cheers,
>
> Junping
>
>
> [1]: 2.6.3 release plan: http://markmail.org/thread/nc2jogbgni37vu6y
>

Hadoop-Yarn-trunk-Java8 - Build # 797 - Failure

2015-12-15 Thread Apache Jenkins Server 
See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/797/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 5449 lines...]
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.85 sec - in 
org.apache.hadoop.yarn.server.TestDiskFailures
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.yarn.server.TestContainerManagerSecurity
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 51.131 sec - in 
org.apache.hadoop.yarn.server.TestContainerManagerSecurity

Results :

Tests in error: 
  
TestMiniYarnClusterNodeUtilization.testUpdateNodeUtilization:116->verifySimulatedUtilization:217
 NullPointer

Tests run: 11, Failures: 0, Errors: 1, Skipped: 0

[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop YARN  SUCCESS [  3.573 s]
[INFO] Apache Hadoop YARN API  SUCCESS [01:23 min]
[INFO] Apache Hadoop YARN Common . SUCCESS [02:44 min]
[INFO] Apache Hadoop YARN Server . SUCCESS [  0.064 s]
[INFO] Apache Hadoop YARN Server Common .. SUCCESS [ 35.769 s]
[INFO] Apache Hadoop YARN NodeManager  SUCCESS [09:30 min]
[INFO] Apache Hadoop YARN Web Proxy .. SUCCESS [ 17.297 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService .. SUCCESS [03:18 min]
[INFO] Apache Hadoop YARN ResourceManager  SUCCESS [  01:01 h]
[INFO] Apache Hadoop YARN Server Tests ... FAILURE [02:04 min]
[INFO] Apache Hadoop YARN Client . SKIPPED
[INFO] Apache Hadoop YARN SharedCacheManager . SKIPPED
[INFO] Apache Hadoop YARN Applications ... SKIPPED
[INFO] Apache Hadoop YARN DistributedShell ... SKIPPED
[INFO] Apache Hadoop YARN Unmanaged Am Launcher .. SKIPPED
[INFO] Apache Hadoop YARN Site ... SKIPPED
[INFO] Apache Hadoop YARN Registry ... SKIPPED
[INFO] Apache Hadoop YARN Project  SKIPPED
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 01:21 h
[INFO] Finished at: 2015-12-15T21:00:07+00:00
[INFO] Final Memory: 80M/1008M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on 
project hadoop-yarn-server-tests: There are test failures.
[ERROR] 
[ERROR] Please refer to 
/home/jenkins/jenkins-slave/workspace/Hadoop-Yarn-trunk-Java8/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/target/surefire-reports
 for the individual test results.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :hadoop-yarn-server-tests
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
Sending e-mails to: yarn-dev@hadoop.apache.org
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
1 tests failed.
FAILED:  
org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization.testUpdateNodeUtilization

Error Message:
null

Stack Trace:
java.lang.NullPointerException: null
at 
org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization.verifySimulatedUtilization(TestMiniYarnClusterNodeUtilization.java:217)
at 
org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization.testUpdateNodeUtilization(TestMiniYarnClusterNodeUtilization.java:116)

Build failed in Jenkins: Hadoop-Yarn-trunk-Java8 #797

2015-12-15 Thread Apache Jenkins Server 
See 

Changes:

[jing9] HDFS-9371. Code cleanup for DatanodeManager. Contributed by Jing Zhao.

--
[...truncated 5256 lines...]
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.905 sec - in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerApplicationAttempt
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerHealth
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.011 sec - in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerHealth
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.yarn.server.resourcemanager.TestRMDispatcher
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.773 sec - in 
org.apache.hadoop.yarn.server.resourcemanager.TestRMDispatcher
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.yarn.server.resourcemanager.resource.TestResources
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.425 sec - in 
org.apache.hadoop.yarn.server.resourcemanager.resource.TestResources
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running 
org.apache.hadoop.yarn.server.resourcemanager.resource.TestResourceWeights
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.067 sec - in 
org.apache.hadoop.yarn.server.resourcemanager.resource.TestResourceWeights
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.yarn.server.resourcemanager.TestTokenClientRMService
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.227 sec - in 
org.apache.hadoop.yarn.server.resourcemanager.TestTokenClientRMService
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running 
org.apache.hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 177.566 sec - 
in 
org.apache.hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.yarn.server.resourcemanager.webapp.TestAppPage
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.043 sec - in 
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestAppPage
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebApp
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.342 sec - in 
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebApp
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running 
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.962 sec - in 
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running 
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes
Tests run: 23, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.149 sec - 
in org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running 
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesReservation
Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 39.641 sec - 
in 
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesReservation
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.89 sec - in 
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running 
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesForCSWithPartitions
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.555 sec - in 
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesForCSWithPartitions
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running 
org.apache.hadoop

Re: [VOTE] Release Apache Hadoop 2.6.3 RC0

2015-12-15 Thread Naganarasimha Garla 
Hi Junping,

+0 (non binding)

Though everything else is working  fine (downloaded the tar ball and
installed single node cluster setup and  verified  few MR jobs),
submission of Unmanaged AM is getting the RM down. YARN-4452 has
already been raised and i am working on it. Will provide the patch for
the trunk and the 2.6.3 version asap

Regards,

+ Naga



Thanks for the work Junping! Downloaded the src tarball. Built locally
and successfully ran
in single node mode with a few map reduce jobs. LGTM.

Li Lu

On Dec 14, 2015, at 04:23, Junping Du
mailto:j...@hortonworks.com>>
wrote:

?Thanks Sarjeet and Tsuyoshi for reporting this. I just fix permission
issue and download
should work now. Please try to download it again. Thanks!


Thanks,


Junping



From: sarjeet singh mailto:sarje...@usc.edu>>
Sent: Sunday, December 13, 2015 6:44 PM
To: common-...@hadoop.apache.org
Cc: mapreduce-...@hadoop.apache.org;
hdfs-...@hadoop.apache.org;yarn-dev@hadoop.apache.org;
junping...@apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.6.3 RC0

I am also getting the same error when downloading tar.gz:

"You don't have permission to access
/~junping_du/hadoop-2.6.3-RC0/hadoop-2.6.3-RC0-src.tar.gz
on this server."

- Sarjeet Singh

On Sat, Dec 12, 2015 at 4:17 PM, Tsuyoshi Ozawa
mailto:oz...@apache.org>>
wrote:
Hi Junping,

Thank you for starting the voting.
I cannot access the tar.gz file because of permission error. Could you
check the permission to access the files?

Forbidden
You don't have permission to access
/~junping_du/hadoop-2.6.3-RC0/hadoop-2.6.3-RC0-src.tar.gz
on this server.

Thanks,
- Tsuyoshi

On Sat, Dec 12, 2015 at 9:16 AM, Junping Du
mailto:j...@hortonworks.com>>
wrote:

Hi all developers in hadoop community,
  I've created a release candidate RC0 for Apache Hadoop 2.6.3 (the
next maintenance release
to follow up 2.6.2.) according to email thread of release plan 2.6.3
[1]. Sorry for this RC
coming a bit late as several blocker issues were getting committed
until yesterday. Below
is the details:

The RC is available for validation at:
*http://people.apache.org/~junping_du/hadoop-2.6.3-RC0/
*

The RC tag in git is: release-2.6.3-RC0

The maven artifacts are staged via
repository.apache.org
at:
*https://repository.apache.org/content/repositories/orgapachehadoop-1025/?
*

You can find my public key
at:http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS

Please try the release and vote. The vote will run for the usual 5 days.

Thanks and happy weekend!


Cheers,

Junping


[1]: 2.6.3 release plan: http://markmail.org/thread/nc2jogbgni37vu6y

[jira] [Created] (YARN-4461) Redundant nodeLocalityDelay log in LeafQueue

2015-12-15 Thread Jason Lowe (JIRA) 
Jason Lowe created YARN-4461:


 Summary: Redundant nodeLocalityDelay log in LeafQueue
 Key: YARN-4461
 URL: https://issues.apache.org/jira/browse/YARN-4461
 Project: Hadoop YARN
  Issue Type: Bug
  Components: capacityscheduler
Affects Versions: 2.7.1
Reporter: Jason Lowe
Priority: Trivial


In LeafQueue#setupQueueConfigs there's a redundant log of nodeLocalityDelay:
{code}
"nodeLocalityDelay = " + nodeLocalityDelay + "\n" +
"labels=" + labelStrBuilder.toString() + "\n" +
"nodeLocalityDelay = " +  nodeLocalityDelay + "\n" +
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [Timeline V2 branch] Latest timeline v2 and SMP problem

2015-12-15 Thread Varun Saxena 
Thanks Li Lu.
Checked the patch. Changes LGTM.

Regards,
Varun Saxena.

On Wed, Dec 16, 2015 at 1:07 AM, Li Lu  wrote:

> OK I fixed this and verified locally. Patch is posted to YARN-4460. Patch
> is simple so a quick review is much appreciated. Anyone hitting this
> problem is more than welcome to verify. Thanks!
>
> Li Lu
>
> On Dec 15, 2015, at 11:00, Li Lu  wrote:
>
> Thanks Varun and Naga! I verified locally that the V2 publisher introduced
> in YARN-4129 caused this problem. I’ll open a JIRA and post a quick fix
> right away. Thanks for the information!
>
> Li Lu
>
> On Dec 14, 2015, at 21:38, Naganarasimha G R (Naga) <
> garlanaganarasi...@huawei.com >> wrote:
>
> Hi Varun & Li,
>
> Yes Varun most possible reason would be what you mentioned and it has to
> be done in serviceInit which is taken care in V1 Publisher but missed in V2
> Publisher.
> Entire logic present in serviceStart of V2Publisher should be moved to
> serviceInit.
>
> But was wondering for which event/entity ? was it in RM Recover mode ?
>
> Regards,
> + Naga
>
> 
> From: Varun Saxena [vsaxena.va...@gmail.com >]
> Sent: Tuesday, December 15, 2015 10:48
> To: Li Lu
> Cc: yarn-dev@hadoop.apache.org >; Sangjin Lee; Junping Du; Vrushali
> Channapattan; Joep Rottinghuis; Naganarasimha G R (Naga)
> Subject: Re: [Timeline V2 branch] Latest timeline v2 and SMP problem
>
> Hi Li,
>
> This is because we are registering the event in serviceStart() instead of
> serviceInit().
> As SMP is the last service in the list, its started right in the end i.e.
> even after all the RPCs', UI related stuff.
>
> This can cause an app flow to start before the SMP/V2Publisher service has
> even started. This is what causes the issue.
>
> You want to raise JIRA for this issue or should I ? I can handle it.
>
> Regards,
> Varun Saxena.
>
> On Tue, Dec 15, 2015 at 8:35 AM, Li Lu  mailto:l...@hortonworks.com >> wrote:
> Thanks Sangjin. I’ll keep tracing this. Meanwhile, if anybody has
> reproduced the problem, please feel free to let me know. Thanks!
>
> Li Lu
>
> On Dec 14, 2015, at 18:16, Sangjin Lee  mailto:sjl...@gmail.com >> wrote:
>
> Can you bisect the commits to see if you can isolate which commit
> introduced the issue?
>
> On Mon, Dec 14, 2015 at 5:39 PM, Li Lu  mailto:l...@hortonworks.com >> wrote:
>
> Hi YARN developers working on Timeline v2 (YARN-2928) branch,
>
> I just realized I’ve accidentally turned off SMP for my local Timeline v2
> build. After I turned yarn.system-metrics-publisher.enabled back on, the RM
> fails to start with the following FATAL message:
>
> 2015-12-14 17:27:54,125 INFO  ipc.Server (Server.java:run(797)) - IPC
> Server listener on 8033: starting
> 2015-12-14 17:27:54,127 FATAL event.AsyncDispatcher
> (AsyncDispatcher.java:dispatch(189)) - Error in dispatcher thread true
> java.lang.Exception: No handler for registered for class
>
> org.apache.hadoop.yarn.server.resourcemanager.metrics.AbstractSystemMetricsPubli
> sher$SystemMetricsEventType
>   at
>
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:185)
>   at
>
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:109)
>   at java.lang.Thread.run(Thread.java:745)
> 2015-12-14 17:27:54,127 INFO  event.AsyncDispatcher
> (AsyncDispatcher.java:register(208)) - Registering class
> org.apache.hadoop.yarn.serve
>
> r.resourcemanager.metrics.AbstractSystemMetricsPublisher$SystemMetricsEventType
> for class org.apache.hadoop.yarn.server.resourcemanager.m
> etrics.TimelineServiceV2Publisher$TimelineV2EventHandler
>
> Interestingly, we’re registering this class to timeline v2 handler in the
> next line of log. I’m wondering if this is caused by some of my missing
> configs, or a newly introduced issue? Has anybody on feature-YARN-2928
> branch noticed this issue? Thanks!
>
> Li Lu
>
>
>

Re: [Timeline V2 branch] Latest timeline v2 and SMP problem

2015-12-15 Thread Li Lu 
OK I fixed this and verified locally. Patch is posted to YARN-4460. Patch is 
simple so a quick review is much appreciated. Anyone hitting this problem is 
more than welcome to verify. Thanks!

Li Lu

On Dec 15, 2015, at 11:00, Li Lu 
mailto:l...@hortonworks.com>> wrote:

Thanks Varun and Naga! I verified locally that the V2 publisher introduced in 
YARN-4129 caused this problem. I’ll open a JIRA and post a quick fix right 
away. Thanks for the information!

Li Lu

On Dec 14, 2015, at 21:38, Naganarasimha G R (Naga) 
mailto:garlanaganarasi...@huawei.com>>
 wrote:

Hi Varun & Li,

Yes Varun most possible reason would be what you mentioned and it has to be 
done in serviceInit which is taken care in V1 Publisher but missed in V2 
Publisher.
Entire logic present in serviceStart of V2Publisher should be moved to 
serviceInit.

But was wondering for which event/entity ? was it in RM Recover mode ?

Regards,
+ Naga


From: Varun Saxena 
[vsaxena.va...@gmail.com]
Sent: Tuesday, December 15, 2015 10:48
To: Li Lu
Cc: 
yarn-dev@hadoop.apache.org;
 Sangjin Lee; Junping Du; Vrushali Channapattan; Joep Rottinghuis; 
Naganarasimha G R (Naga)
Subject: Re: [Timeline V2 branch] Latest timeline v2 and SMP problem

Hi Li,

This is because we are registering the event in serviceStart() instead of 
serviceInit().
As SMP is the last service in the list, its started right in the end i.e. even 
after all the RPCs', UI related stuff.

This can cause an app flow to start before the SMP/V2Publisher service has even 
started. This is what causes the issue.

You want to raise JIRA for this issue or should I ? I can handle it.

Regards,
Varun Saxena.

On Tue, Dec 15, 2015 at 8:35 AM, Li Lu 
mailto:l...@hortonworks.com>>
 wrote:
Thanks Sangjin. I’ll keep tracing this. Meanwhile, if anybody has reproduced 
the problem, please feel free to let me know. Thanks!

Li Lu

On Dec 14, 2015, at 18:16, Sangjin Lee 
mailto:sjl...@gmail.com>> wrote:

Can you bisect the commits to see if you can isolate which commit
introduced the issue?

On Mon, Dec 14, 2015 at 5:39 PM, Li Lu 
mailto:l...@hortonworks.com>>
 wrote:

Hi YARN developers working on Timeline v2 (YARN-2928) branch,

I just realized I’ve accidentally turned off SMP for my local Timeline v2
build. After I turned yarn.system-metrics-publisher.enabled back on, the RM
fails to start with the following FATAL message:

2015-12-14 17:27:54,125 INFO  ipc.Server (Server.java:run(797)) - IPC
Server listener on 8033: starting
2015-12-14 17:27:54,127 FATAL event.AsyncDispatcher
(AsyncDispatcher.java:dispatch(189)) - Error in dispatcher thread true
java.lang.Exception: No handler for registered for class
org.apache.hadoop.yarn.server.resourcemanager.metrics.AbstractSystemMetricsPubli
sher$SystemMetricsEventType
  at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:185)
  at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:109)
  at java.lang.Thread.run(Thread.java:745)
2015-12-14 17:27:54,127 INFO  event.AsyncDispatcher
(AsyncDispatcher.java:register(208)) - Registering class
org.apache.hadoop.yarn.serve
r.resourcemanager.metrics.AbstractSystemMetricsPublisher$SystemMetricsEventType
for class org.apache.hadoop.yarn.server.resourcemanager.m
etrics.TimelineServiceV2Publisher$TimelineV2EventHandler

Interestingly, we’re registering this class to timeline v2 handler in the
next line of log. I’m wondering if this is caused by some of my missing
configs, or a newly introduced issue? Has anybody on feature-YARN-2928
branch noticed this issue? Thanks!

Li Lu

Build failed in Jenkins: Hadoop-Yarn-trunk #1517

2015-12-15 Thread Apache Jenkins Server 
See 

Changes:

[jing9] HDFS-9371. Code cleanup for DatanodeManager. Contributed by Jing Zhao.

--
[...truncated 8613 lines...]
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0

[INFO] 
[INFO] --- maven-jar-plugin:2.5:jar (default-jar) @ 
hadoop-yarn-server-web-proxy ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-source-plugin:2.3:jar-no-fork (hadoop-java-sources) @ 
hadoop-yarn-server-web-proxy ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-source-plugin:2.3:test-jar-no-fork (hadoop-java-sources) @ 
hadoop-yarn-server-web-proxy ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (dist-enforce) @ 
hadoop-yarn-server-web-proxy ---
[INFO] 
[INFO] --- maven-site-plugin:3.4:attach-descriptor (attach-descriptor) @ 
hadoop-yarn-server-web-proxy ---
[INFO] 
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ 
hadoop-yarn-server-web-proxy ---
[INFO] 
Loading source files for package org.apache.hadoop.yarn.server.webproxy...
Loading source files for package 
org.apache.hadoop.yarn.server.webproxy.amfilter...
Constructing Javadoc information...
Standard Doclet version 1.7.0_55
Building tree for all the packages and classes...
Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating

Hadoop-Yarn-trunk - Build # 1517 - Still Failing

2015-12-15 Thread Apache Jenkins Server 
See https://builds.apache.org/job/Hadoop-Yarn-trunk/1517/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 8806 lines...]
Running org.apache.hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore
Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.67 sec - in 
org.apache.hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore
Running org.apache.hadoop.yarn.server.timeline.TestMemoryTimelineStore
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.415 sec - in 
org.apache.hadoop.yarn.server.timeline.TestMemoryTimelineStore

Results :

Tests in error: 
  TestTimelineAuthenticationFilter.testDelegationTokenOperations:235 » IO 
org.ap...

Tests run: 197, Failures: 0, Errors: 1, Skipped: 0

[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop YARN  SUCCESS [  4.431 s]
[INFO] Apache Hadoop YARN API  SUCCESS [05:21 min]
[INFO] Apache Hadoop YARN Common . SUCCESS [03:52 min]
[INFO] Apache Hadoop YARN Server . SUCCESS [  0.130 s]
[INFO] Apache Hadoop YARN Server Common .. SUCCESS [01:02 min]
[INFO] Apache Hadoop YARN NodeManager  SUCCESS [11:15 min]
[INFO] Apache Hadoop YARN Web Proxy .. SUCCESS [ 44.442 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService .. FAILURE [04:00 min]
[INFO] Apache Hadoop YARN ResourceManager  SKIPPED
[INFO] Apache Hadoop YARN Server Tests ... SKIPPED
[INFO] Apache Hadoop YARN Client . SKIPPED
[INFO] Apache Hadoop YARN SharedCacheManager . SKIPPED
[INFO] Apache Hadoop YARN Applications ... SKIPPED
[INFO] Apache Hadoop YARN DistributedShell ... SKIPPED
[INFO] Apache Hadoop YARN Unmanaged Am Launcher .. SKIPPED
[INFO] Apache Hadoop YARN Site ... SKIPPED
[INFO] Apache Hadoop YARN Registry ... SKIPPED
[INFO] Apache Hadoop YARN Project  SKIPPED
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 26:24 min
[INFO] Finished at: 2015-12-15T19:25:21+00:00
[INFO] Final Memory: 89M/1008M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on 
project hadoop-yarn-server-applicationhistoryservice: There are test failures.
[ERROR] 
[ERROR] Please refer to 
/home/jenkins/jenkins-slave/workspace/Hadoop-Yarn-trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/target/surefire-reports
 for the individual test results.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :hadoop-yarn-server-applicationhistoryservice
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
Sending e-mails to: yarn-dev@hadoop.apache.org
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
1 tests failed.
FAILED:  
org.apache.hadoop.yarn.server.timeline.security.TestTimelineAuthenticationFilter.testDelegationTokenOperations[1]

Error Message:
org.apache.hadoop.security.authentication.client.AuthenticationException: 
GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC 
reply)

Stack Trace:
java.io.IOException: 
org.apache.hadoop.security.authentication.client.AuthenticationException: 
GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC 
reply)
at sun.security.krb5.KdcComm.send(KdcComm.java:250)
at sun.security.krb5.KdcComm.send(KdcComm.java:191)
at sun.security.krb5.KrbTgsReq.send(KrbTgsReq.java:187)
at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:202)
at 
sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:311)
at 
sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:115)
at

[jira] [Created] (YARN-4460) [Bug fix] RM fails to start when SMP is enabled

2015-12-15 Thread Li Lu (JIRA) 
Li Lu created YARN-4460:
---

 Summary: [Bug fix] RM fails to start when SMP is enabled
 Key: YARN-4460
 URL: https://issues.apache.org/jira/browse/YARN-4460
 Project: Hadoop YARN
  Issue Type: Sub-task
Affects Versions: YARN-2928
Reporter: Li Lu
Assignee: Li Lu


When SMP is enabled, the RM starts with the following fatal message: 
{code}
FATAL event.AsyncDispatcher (AsyncDispatcher.java:dispatch(189)) - Error in 
dispatcher thread true
java.lang.Exception: No handler for registered for class 
org.apache.hadoop.yarn.server.resourcemanager.metrics.AbstractSystemMetricsPubli
sher$SystemMetricsEventType
   at 
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:185)
   at 
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:109)
   at java.lang.Thread.run(Thread.java:745)
{code}

We should register event handlers in service init stage in 
TimelineServiceV2Publisher to fix this problem. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [Timeline V2 branch] Latest timeline v2 and SMP problem

2015-12-15 Thread Li Lu 
Thanks Varun and Naga! I verified locally that the V2 publisher introduced in 
YARN-4129 caused this problem. I’ll open a JIRA and post a quick fix right 
away. Thanks for the information!

Li Lu

On Dec 14, 2015, at 21:38, Naganarasimha G R (Naga) 
mailto:garlanaganarasi...@huawei.com>> wrote:

Hi Varun & Li,

Yes Varun most possible reason would be what you mentioned and it has to be 
done in serviceInit which is taken care in V1 Publisher but missed in V2 
Publisher.
Entire logic present in serviceStart of V2Publisher should be moved to 
serviceInit.

But was wondering for which event/entity ? was it in RM Recover mode ?

Regards,
+ Naga


From: Varun Saxena [vsaxena.va...@gmail.com]
Sent: Tuesday, December 15, 2015 10:48
To: Li Lu
Cc: yarn-dev@hadoop.apache.org; Sangjin Lee; 
Junping Du; Vrushali Channapattan; Joep Rottinghuis; Naganarasimha G R (Naga)
Subject: Re: [Timeline V2 branch] Latest timeline v2 and SMP problem

Hi Li,

This is because we are registering the event in serviceStart() instead of 
serviceInit().
As SMP is the last service in the list, its started right in the end i.e. even 
after all the RPCs', UI related stuff.

This can cause an app flow to start before the SMP/V2Publisher service has even 
started. This is what causes the issue.

You want to raise JIRA for this issue or should I ? I can handle it.

Regards,
Varun Saxena.

On Tue, Dec 15, 2015 at 8:35 AM, Li Lu 
mailto:l...@hortonworks.com>> wrote:
Thanks Sangjin. I’ll keep tracing this. Meanwhile, if anybody has reproduced 
the problem, please feel free to let me know. Thanks!

Li Lu

On Dec 14, 2015, at 18:16, Sangjin Lee 
mailto:sjl...@gmail.com>> wrote:

Can you bisect the commits to see if you can isolate which commit
introduced the issue?

On Mon, Dec 14, 2015 at 5:39 PM, Li Lu 
mailto:l...@hortonworks.com>> wrote:

Hi YARN developers working on Timeline v2 (YARN-2928) branch,

I just realized I’ve accidentally turned off SMP for my local Timeline v2
build. After I turned yarn.system-metrics-publisher.enabled back on, the RM
fails to start with the following FATAL message:

2015-12-14 17:27:54,125 INFO  ipc.Server (Server.java:run(797)) - IPC
Server listener on 8033: starting
2015-12-14 17:27:54,127 FATAL event.AsyncDispatcher
(AsyncDispatcher.java:dispatch(189)) - Error in dispatcher thread true
java.lang.Exception: No handler for registered for class
org.apache.hadoop.yarn.server.resourcemanager.metrics.AbstractSystemMetricsPubli
sher$SystemMetricsEventType
   at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:185)
   at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:109)
   at java.lang.Thread.run(Thread.java:745)
2015-12-14 17:27:54,127 INFO  event.AsyncDispatcher
(AsyncDispatcher.java:register(208)) - Registering class
org.apache.hadoop.yarn.serve
r.resourcemanager.metrics.AbstractSystemMetricsPublisher$SystemMetricsEventType
for class org.apache.hadoop.yarn.server.resourcemanager.m
etrics.TimelineServiceV2Publisher$TimelineV2EventHandler

Interestingly, we’re registering this class to timeline v2 handler in the
next line of log. I’m wondering if this is caused by some of my missing
configs, or a newly introduced issue? Has anybody on feature-YARN-2928
branch noticed this issue? Thanks!

Li Lu

[jira] [Created] (YARN-4459) container-executor might kill process wrongly

2015-12-15 Thread Jun Gong (JIRA) 
Jun Gong created YARN-4459:
--

 Summary: container-executor might kill process wrongly
 Key: YARN-4459
 URL: https://issues.apache.org/jira/browse/YARN-4459
 Project: Hadoop YARN
  Issue Type: Bug
  Components: nodemanager
Reporter: Jun Gong
Assignee: Jun Gong


When calling 'signal_container_as_user' in container-executor, it first checks 
whether process group exists, if not, it will kill the process itself(if it the 
process exists).  It is not reasonable because that the process group does not 
exist means corresponding container has finished, if we kill the process 
itself, we just kill wrong process.

We found it happened in our cluster many times. We used same account for 
starting NM and submitted app, and container-executor sometimes killed NM(the 
wrongly killed process might just be a newly started thread and was NM's child 
process).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Release Apache Hadoop 2.6.3 RC0

2015-12-15 Thread Lei Chang 
Nice. +1.

On Sat, Dec 12, 2015 at 9:16 AM, Junping Du > wrote:

> Hi all developers in hadoop community,
>I've created a release candidate RC0 for Apache Hadoop 2.6.3 (the next 
> maintenance release
> to follow up 2.6.2.) according to email thread of release plan 2.6.3 [1]. 
> Sorry for this RC
> coming a bit late as several blocker issues were getting committed until 
> yesterday. Below
> is the details:
>
> The RC is available for validation at:
> *http://people.apache.org/~junping_du/hadoop-2.6.3-RC0/
> *
>
> The RC tag in git is: release-2.6.3-RC0
>
> The maven artifacts are staged via repository.apache.org at:
> *https://repository.apache.org/content/repositories/orgapachehadoop-1025/?
> *
>
> You can find my public key 
> at:http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
>
> Please try the release and vote. The vote will run for the usual 5 days.
>
> Thanks and happy weekend!
>
>
> Cheers,
>
> Junping
>
>
>

Reviewers needed YARN-4430

2015-12-15 Thread Steve Loughran 

Hi

can someone look @ https://issues.apache.org/jira/browse/YARN-4430

thx

-steve

Jenkins build is back to normal : Hadoop-Yarn-trunk-Java8 #796

2015-12-15 Thread Apache Jenkins Server 
See

Build failed in Jenkins: Hadoop-Yarn-trunk #1516

2015-12-15 Thread Apache Jenkins Server 
See 

Changes:

[zxu] YARN-4440. FSAppAttempt#getAllowedLocalityLevelByTime should init the

[shv] HDFS-9516. Truncate file fails with data dirs on multiple disks.

[zxu] MAPREDUCE-6436. JobHistory cache issue. Contributed by Kai Sasaki

[zxu] Update CHANGES.txt to add YARN-3857 and YARN-3535 to branch-2.6

--
[...truncated 7595 lines...]
 [exec] Scanning dependencies of target test-container-executor
 [exec] make[2]: Leaving directory 
`
 [exec] make -f CMakeFiles/test-container-executor.dir/build.make 
CMakeFiles/test-container-executor.dir/build
 [exec] make[2]: Entering directory 
`
 [exec] /usr/bin/cmake -E cmake_progress_report 

 4
 [exec] [100%] Building C object 
CMakeFiles/test-container-executor.dir/main/native/container-executor/test/test-container-executor.c.o
 [exec] /usr/bin/cc   -g -O2 -Wall -pthread  -D_GNU_SOURCE 
-I
 
-I
 
-I
 
-I
-o 
CMakeFiles/test-container-executor.dir/main/native/container-executor/test/test-container-executor.c.o
   -c 

 [exec] 
:
 In function 'main':
 [exec] 
:839:10:
 warning: ignoring return value of 'seteuid', declared with attribute 
warn_unused_result [-Wunused-result]
 [exec]seteuid(0);
 [exec]   ^
 [exec] Linking C executable target/usr/local/bin/test-container-executor
 [exec] /usr/bin/cmake -E cmake_link_script 
CMakeFiles/test-container-executor.dir/link.txt --verbose=1
 [exec] /usr/bin/cc   -g -O2 -Wall -pthread  -D_GNU_SOURCE 
CMakeFiles/test-container-executor.dir/main/native/container-executor/test/test-container-executor.c.o
  -o target/usr/local/bin/test-container-executor -rdynamic libcontainer.a 
 [exec] make[2]: Leaving directory 
`
 [exec] /usr/bin/cmake -E cmake_progress_report 

  4
 [exec] [100%] Built target test-container-executor
 [exec] make[1]: Leaving directory 
`
 [exec] /usr/bin/cmake -E cmake_progress_start 

 0
 [exec] [ 50%] Built target container
 [exec] [ 75%] Built target container-executor
 [exec] [100%] Built target test-container-executor
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ 
hadoop-yarn-server-nodemanager ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 5 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ 
hadoop-yarn-server-nodemanager ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 85 source files to

Hadoop-Yarn-trunk - Build # 1516 - Failure

2015-12-15 Thread Apache Jenkins Server 
See https://builds.apache.org/job/Hadoop-Yarn-trunk/1516/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 7788 lines...]
Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 35.025 sec - 
in org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServicesApps
Running org.apache.hadoop.yarn.server.nodemanager.webapp.TestContainerLogsPage
Tests run: 5, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 5.168 sec - in 
org.apache.hadoop.yarn.server.nodemanager.webapp.TestContainerLogsPage

Results :

Failed tests: 
  
TestLogAggregationService.testLocalFileDeletionOnDiskFull:285->verifyLocalFileDeletion:229
 null
  
TestLogAggregationService.testLocalFileDeletionAfterUpload:265->verifyLocalFileDeletion:229
 null

Tests run: 398, Failures: 2, Errors: 0, Skipped: 17

[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop YARN  SUCCESS [  6.666 s]
[INFO] Apache Hadoop YARN API  SUCCESS [07:17 min]
[INFO] Apache Hadoop YARN Common . SUCCESS [05:48 min]
[INFO] Apache Hadoop YARN Server . SUCCESS [  0.131 s]
[INFO] Apache Hadoop YARN Server Common .. SUCCESS [01:38 min]
[INFO] Apache Hadoop YARN NodeManager  FAILURE [13:10 min]
[INFO] Apache Hadoop YARN Web Proxy .. SKIPPED
[INFO] Apache Hadoop YARN ApplicationHistoryService .. SKIPPED
[INFO] Apache Hadoop YARN ResourceManager  SKIPPED
[INFO] Apache Hadoop YARN Server Tests ... SKIPPED
[INFO] Apache Hadoop YARN Client . SKIPPED
[INFO] Apache Hadoop YARN SharedCacheManager . SKIPPED
[INFO] Apache Hadoop YARN Applications ... SKIPPED
[INFO] Apache Hadoop YARN DistributedShell ... SKIPPED
[INFO] Apache Hadoop YARN Unmanaged Am Launcher .. SKIPPED
[INFO] Apache Hadoop YARN Site ... SKIPPED
[INFO] Apache Hadoop YARN Registry ... SKIPPED
[INFO] Apache Hadoop YARN Project  SKIPPED
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 28:05 min
[INFO] Finished at: 2015-12-15T10:33:55+00:00
[INFO] Final Memory: 76M/888M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on 
project hadoop-yarn-server-nodemanager: There are test failures.
[ERROR] 
[ERROR] Please refer to 
/home/jenkins/jenkins-slave/workspace/Hadoop-Yarn-trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/surefire-reports
 for the individual test results.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :hadoop-yarn-server-nodemanager
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
Sending e-mails to: yarn-dev@hadoop.apache.org
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
2 tests failed.
FAILED:  
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService.testLocalFileDeletionOnDiskFull

Error Message:
null

Stack Trace:
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertFalse(Assert.java:64)
at org.junit.Assert.assertFalse(Assert.java:74)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService.verifyLocalFileDeletion(TestLogAggregationService.java:229)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService.testLocalFileDeletionOnDiskFull(TestLogAggregationService.java:285)


FAILED:  
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService.testLocalFileDeletionAfterUpload

Error Message:
null

Stack Trace:
java.lang.AssertionErro

[jira] [Created] (YARN-4458) Compilation error at branch-2.7 due to getNodeLabelExpression not defined in NMContainerStatusPBImpl.

2015-12-15 Thread zhihai xu (JIRA) 
zhihai xu created YARN-4458:
---

 Summary: Compilation error at branch-2.7 due to 
getNodeLabelExpression not defined in NMContainerStatusPBImpl.
 Key: YARN-4458
 URL: https://issues.apache.org/jira/browse/YARN-4458
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: zhihai xu
Assignee: zhihai xu






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Release Apache Hadoop 2.7.2 RC0

2015-12-15 Thread Konstantin Shvachko 
Sorry for bringing this up late.
I think we should pick up HDFS-9516 for this release.
Rather critical bug fix, but up to you, Vinod.

Thanks,
--Konst

On Wed, Nov 11, 2015 at 8:31 PM, Vinod Kumar Vavilapalli  wrote:

> Hi all,
>
>
> I've created a release candidate RC0 for Apache Hadoop 2.7.2.
>
>
> As discussed before, this is the next maintenance release to follow up
> 2.7.1.
>
>
> The RC is available for validation at:
>
> *http://people.apache.org/~vinodkv/hadoop-2.7.2-RC0/
>
> *
>
>
> The RC tag in git is: release-2.7.2-RC0
>
>
> The maven artifacts are available via repository.apache.org at
>
> *https://repository.apache.org/content/repositories/orgapachehadoop-1023/
>
>  >*
>
>
> As you may have noted, an unusually long 2.6.3 release caused 2.7.2 to slip
> by quite a bit. This release's related discussion threads are linked below:
> [1] and [2].
>
>
> Please try the release and vote; the vote will run for the usual 5 days.
>
>
> Thanks,
>
> Vinod
>
>
> [1]: 2.7.2 release plan: http://markmail.org/message/oozq3gvd4nhzsaes
>
> [2]: Planning Apache Hadoop 2.7.2
> http://markmail.org/message/iktqss2qdeykgpqk
>