Success: YARN-1452 PreCommit Build #3452
Jira: https://issues.apache.org/jira/browse/YARN-1452
Build: https://builds.apache.org/job/PreCommit-YARN-Build/3452/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 3042 lines...]
/bin/kill -9 6326
kill: No such process
NOP
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12636524/YARN-1452.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3452//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3452//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
177dd5657406014d35ec98d720a1d1172ebbe506 logged out
==
==
Finished build.
==
==
Archiving artifacts
Description set: YARN-1452
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
Build failed in Jenkins: Hadoop-Yarn-trunk #520
See https://builds.apache.org/job/Hadoop-Yarn-trunk/520/changes Changes: [jing9] HDFS-5840. Follow-up to HDFS-5138 to improve error handling during partial upgrade failures. Contributed by Aaron T. Myers, Suresh Srinivas, and Jing Zhao. [suresh] HDFS-6125. Cleanup unnecessary cast in HDFS code base. Contributed by Suresh Srinivas. [vinodkv] YARN-1850. Introduced the ability to optionally disable sending out timeline-events in the TimelineClient. Contributed by Zhijie Shen. [szetszwo] HADOOP-10425. LocalFileSystem.getContentSummary should not count crc files. [vinodkv] MAPREDUCE-5795. Fixed MRAppMaster to record the correct job-state after it recovers from a commit during a previous attempt. Contributed by Xuan Gong. [arp] HDFS-6124. Add final modifier to class members. (Contributed by Suresh Srinivas) [kasha] HADOOP-10423. Clarify compatibility policy document for combination of new client and old server. (Chris Nauroth via kasha) [cnauroth] HADOOP-10422. Remove redundant logging of RPC retry attempts. Contributed by Chris Nauroth. [cnauroth] HDFS-5846. Shuffle phase is slow in Windows - FadviseFileRegion::transferTo does not read disks efficiently. Contributed by Nikola Vujic. [jing9] Move HDFS-5138 to 2.4.0 section in CHANGES.txt [jing9] HDFS-6135. In HDFS upgrade with HA setup, JournalNode cannot handle layout version bump when rolling back. Contributed by Jing Zhao. [brandonli] HDFS-6050. NFS does not handle exceptions correctly in a few places. Contributed by Brandon Li [jianhe] YARN-1852. Fixed RMAppAttempt to not resend AttemptFailed/AttemptKilled events to already recovered Failed/Killed RMApps. Contributed by Rohith Sharmaks [cnauroth] MAPREDUCE-5791. Shuffle phase is slow in Windows - FadviseFileRegion::transferTo does not read disks efficiently. Contributed by Nikola Vujic. [szetszwo] HADOOP-10015. UserGroupInformation prints out excessive warnings. Contributed by Nicolas Liochon [zjshen] YARN-1838. Enhanced timeline service getEntities API to get entities from a given entity ID or insertion timestamp. Contributed by Billie Rinaldi. [jeagles] YARN-1670. aggregated log writer can write more log data then it says is the log length (Mit Desai via jeagles) [kihwal] HDFS-3087. Decomissioning on NN restart can complete without blocks being replicated. Contributed by Rushabh S Shah. -- [...truncated 18060 lines...] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2065) IPC Server handler 4 on 59853 daemon prio=5 tid=238 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2065) IPC Server handler 7 on 45253 daemon prio=5 tid=174 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2065) IPC Server handler 8 on 59853 daemon prio=5 tid=242 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2065) IPC Server Responder daemon prio=5 tid=233 runnable java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) at
[jira] [Created] (YARN-1871) We should eliminate writing *PBImpl code in YARN
Wangda Tan created YARN-1871: Summary: We should eliminate writing *PBImpl code in YARN Key: YARN-1871 URL: https://issues.apache.org/jira/browse/YARN-1871 Project: Hadoop YARN Issue Type: Improvement Components: api Affects Versions: 2.4.0 Reporter: Wangda Tan Assignee: Wangda Tan Currently, We need write PBImpl classes one by one. After running find . -name *PBImpl*.java | xargs wc -l under hadoop source code directory, we can see, there're more than 25,000 LOC. I think we should improve this, which will be very helpful for YARN developers to make changes for YARN protocols. There're only some limited patterns in current *PBImpl, * Simple types, like string, int32, float. * List? types * Map? types * Enum types Code generation should be enough to generate such PBImpl classes. Some other requirements are, * Leave other related code alone, like service implemention (e.g. ContainerManagerImpl). * (If possible) Forward compatibility, developpers can write their own PBImpl or genereate them. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (YARN-1872) TestDistributedShell occasionally fails in trunk
Ted Yu created YARN-1872: Summary: TestDistributedShell occasionally fails in trunk Key: YARN-1872 URL: https://issues.apache.org/jira/browse/YARN-1872 Project: Hadoop YARN Issue Type: Test Reporter: Ted Yu From https://builds.apache.org/job/Hadoop-Yarn-trunk/520/console : TestDistributedShell#testDSShellWithCustomLogPropertyFile failed and TestDistributedShell#testDSShell timed out. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: About Map 100% reduce %0 issue
You are missing the following in your yarn site:
property
nameyarn.nodemanager.aux-services.mapreduce_shuffle.class/name
valueorg.apache.hadoop.mapred.ShuffleHandler/value
/property
You will need to restart the nodemanager for this property to take effect.
-- Hitesh
On Mar 24, 2014, at 9:34 PM, Vincent,Wei wrote:
All
I am a new comer for Hadoop, I have run
the hadoop-mapreduce-examples-2.2.0.jar wordcount, but the result is that
it always pending at map 100% and reduce %0.
14/03/25 20:19:20 INFO client.RMProxy: Connecting to ResourceManager at
master/159.99.249.63:8032
14/03/25 20:19:20 INFO input.FileInputFormat: Total input paths to process
: 1
14/03/25 20:19:20 INFO mapreduce.JobSubmitter: number of splits:1
14/03/25 20:19:20 INFO Configuration.deprecation: user.name is deprecated.
Instead, use mapreduce.job.user.name
14/03/25 20:19:20 INFO Configuration.deprecation: mapred.jar is deprecated.
Instead, use mapreduce.job.jar
14/03/25 20:19:20 INFO Configuration.deprecation: mapred.output.value.class
is deprecated. Instead, use mapreduce.job.output.value.class
14/03/25 20:19:20 INFO Configuration.deprecation: mapreduce.combine.class
is deprecated. Instead, use mapreduce.job.combine.class
14/03/25 20:19:20 INFO Configuration.deprecation: mapreduce.map.class is
deprecated. Instead, use mapreduce.job.map.class
14/03/25 20:19:20 INFO Configuration.deprecation: mapred.job.name is
deprecated. Instead, use mapreduce.job.name
14/03/25 20:19:20 INFO Configuration.deprecation: mapreduce.reduce.class is
deprecated. Instead, use mapreduce.job.reduce.class
14/03/25 20:19:20 INFO Configuration.deprecation: mapred.input.dir is
deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
14/03/25 20:19:20 INFO Configuration.deprecation: mapred.output.dir is
deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
14/03/25 20:19:20 INFO Configuration.deprecation: mapred.map.tasks is
deprecated. Instead, use mapreduce.job.maps
14/03/25 20:19:20 INFO Configuration.deprecation: mapred.output.key.class
is deprecated. Instead, use mapreduce.job.output.key.class
14/03/25 20:19:20 INFO Configuration.deprecation: mapred.working.dir is
deprecated. Instead, use mapreduce.job.working.dir
14/03/25 20:19:20 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_1395747600383_0002
14/03/25 20:19:20 INFO impl.YarnClientImpl: Submitted application
application_1395747600383_0002 to ResourceManager at master/
159.99.249.63:8032
14/03/25 20:19:20 INFO mapreduce.Job: The url to track the job:
http://master:8088/proxy/application_1395747600383_0002/
14/03/25 20:19:20 INFO mapreduce.Job: Running job: job_1395747600383_0002
14/03/25 20:19:24 INFO mapreduce.Job: Job job_1395747600383_0002 running in
uber mode : false
14/03/25 20:19:24 INFO mapreduce.Job: map 0% reduce 0%
14/03/25 20:19:28 INFO mapreduce.Job: map 100% reduce 0%
14/03/25 20:19:31 INFO mapreduce.Job: Task Id :
attempt_1395747600383_0002_r_00_0, Status : FAILED
Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error
in shuffle in fetcher#5
at
org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:121)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:380)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
Caused by: java.io.IOException: Exceeded MAX_FAILED_UNIQUE_FETCHES;
bailing-out.
at
org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.checkReducerHealth(ShuffleSchedulerImpl.java:323)
at
org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.copyFailed(ShuffleSchedulerImpl.java:245)
at
org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:347)
at
org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:165)
someone says that this is caused by hosts configure .I have checked my
/etc/hosts on all Mater slaves:
127.0.0.1 localhost.localdomain localhost
159.99.249.63 master
159.99.249.203 slave1
159.99.249.99 slave2
159.99.249.88 slave3
Would you please help me to fix the issue, many thanks .
my yarn-site.xml
?xml version=1.0?
?xml-stylesheet type=text/xsl href=configuration.xsl?
configuration
property
descriptionThe hostname of the RM./description
nameyarn.resourcemanager.hostname/name
valuemaster/value
/property
property
nameyarn.nodemanager.aux-services/name
valuemapreduce_shuffle/value
/property
property
descriptionThe address of the container manager in the NM./description
nameyarn.nodemanager.address/name
value${yarn.nodemanager.hostname}:8041/value
/property
[jira] [Created] (YARN-1873) TestDistributedShell#testDSShell fails
Mit Desai created YARN-1873:
---
Summary: TestDistributedShell#testDSShell fails
Key: YARN-1873
URL: https://issues.apache.org/jira/browse/YARN-1873
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Mit Desai
Assignee: Mit Desai
testDSShell fails when the tests are run in random order. I see a cleanup issue
here.
{noformat}
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 72.222 sec
FAILURE! - in
org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell
testOrder(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
Time elapsed: 44.127 sec FAILURE!
java.lang.AssertionError: expected:1 but was:6
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at
org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShell(TestDistributedShell.java:204)
at
org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testOrder(TestDistributedShell.java:134)
Results :
Failed tests:
TestDistributedShell.testOrder:134-testDSShell:204 expected:1 but was:6
{noformat}
The Line numbers will be little deviated because I was trying to reproduce the
error by running the tests in specific order. But the Line that causes the
assert fail is {{Assert.assertEquals(1,
entitiesAttempts.getEntities().size());}}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
Failed: YARN-1867 PreCommit Build #3454
Jira: https://issues.apache.org/jira/browse/YARN-1867
Build: https://builds.apache.org/job/PreCommit-YARN-Build/3454/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 123 lines...]
no change for http://svn.apache.org/repos/asf/hadoop/nightly since the previous
build
No emails were triggered.
[PreCommit-YARN-Build] $ /bin/bash /tmp/hudson2505182573111833833.sh
32768
Running in Jenkins mode
==
==
Testing patch for YARN-1867.
==
==
At revision 1581491.
YARN-1867 patch is being downloaded at Tue Mar 25 20:16:10 UTC 2014 from
http://issues.apache.org/jira/secure/attachment/12636769/YARN-1867-20140325.txt
cp: cannot stat `/home/jenkins/buildSupport/lib/*': No such file or directory
The patch does not appear to apply with p0 to p2
PATCH APPLICATION FAILED
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12636769/YARN-1867-20140325.txt
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3454//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
a13a58a4953899dce318ffa54beb82e02f4858d2 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
No tests ran.
[jira] [Created] (YARN-1874) Cleanup: Move RMActiveServices out of ResourceManager into its own file
Karthik Kambatla created YARN-1874: -- Summary: Cleanup: Move RMActiveServices out of ResourceManager into its own file Key: YARN-1874 URL: https://issues.apache.org/jira/browse/YARN-1874 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Karthik Kambatla As [~vinodkv] noticed on YARN-1867, ResourceManager is hard to maintain. We should move RMActiveServices out to make it more manageable. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (YARN-1875) TestRMHA#testStartAndTransitions is failing
Mit Desai created YARN-1875:
---
Summary: TestRMHA#testStartAndTransitions is failing
Key: YARN-1875
URL: https://issues.apache.org/jira/browse/YARN-1875
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Mit Desai
Attachments: Log.rtf
{noformat}
testStartAndTransitions(org.apache.hadoop.yarn.server.resourcemanager.TestRMHA)
Time elapsed: 5.883 sec FAILURE!
java.lang.AssertionError: Incorrect value for metric availableMB
expected:2048 but was:4096
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMHA.assertMetric(TestRMHA.java:396)
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMHA.verifyClusterMetrics(TestRMHA.java:387)
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMHA.testStartAndTransitions(TestRMHA.java:160)
Results :
Failed tests:
TestRMHA.testStartAndTransitions:160-verifyClusterMetrics:387-assertMetric:396
Incorrect value for metric availableMB expected:2048 but was:4096
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
Success: YARN-1521 PreCommit Build #3453
Jira: https://issues.apache.org/jira/browse/YARN-1521
Build: https://builds.apache.org/job/PreCommit-YARN-Build/3453/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 3480 lines...]
/bin/kill -9 32335
kill: No such process
NOP
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12636766/YARN-1521.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3453//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3453//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
ffb0999be98ef3777ae37f6bbe56d4063467d725 logged out
==
==
Finished build.
==
==
Archiving artifacts
Description set: YARN-1521
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
Success: YARN-1873 PreCommit Build #3455
Jira: https://issues.apache.org/jira/browse/YARN-1873
Build: https://builds.apache.org/job/PreCommit-YARN-Build/3455/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 2930 lines...]
/bin/kill -9 24014
kill: No such process
NOP
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12636768/YARN-1873.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3455//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3455//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
a1eb818b527f31d694d66bacc54cc12dfc264e2a logged out
==
==
Finished build.
==
==
Archiving artifacts
Description set: YARN-1873
Recording test results
Publish JUnit test result report is waiting for a checkpoint on
PreCommit-YARN-Build #3453
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
Success: YARN-1867 PreCommit Build #3458
Jira: https://issues.apache.org/jira/browse/YARN-1867
Build: https://builds.apache.org/job/PreCommit-YARN-Build/3458/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 3036 lines...]
/bin/kill -9 31907
kill: No such process
NOP
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12636787/YARN-1867-20140325-trunk.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3458//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3458//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
6b3bdb58d8504b6e6d941ab60bd64f75b7417fe5 logged out
==
==
Finished build.
==
==
Archiving artifacts
Description set: YARN-1867
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
Success: YARN-1521 PreCommit Build #3459
Jira: https://issues.apache.org/jira/browse/YARN-1521
Build: https://builds.apache.org/job/PreCommit-YARN-Build/3459/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 3367 lines...]
/bin/kill -9 10434
kill: No such process
NOP
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12636836/YARN-1521.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3459//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3459//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
a69787a04bd77fd8b25bafd250153ccb2c6f2527 logged out
==
==
Finished build.
==
==
Archiving artifacts
Description set: YARN-1521
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
Success: YARN-1452 PreCommit Build #3460
Jira: https://issues.apache.org/jira/browse/YARN-1452
Build: https://builds.apache.org/job/PreCommit-YARN-Build/3460/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 2989 lines...]
/bin/kill -9 12563
kill: No such process
NOP
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12636842/YARN-1452.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3460//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3460//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
45cb610b14d20bf66ef6dfacb93ad3a60fd1cdb3 logged out
==
==
Finished build.
==
==
Archiving artifacts
Description set: YARN-1452
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Created] (YARN-1876) Document the REST APIs of timeline and generic history services
Zhijie Shen created YARN-1876: - Summary: Document the REST APIs of timeline and generic history services Key: YARN-1876 URL: https://issues.apache.org/jira/browse/YARN-1876 Project: Hadoop YARN Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (YARN-1877) ZK store: Add yarn.resourcemanager.zk-state-store.root-node.auth for root node auth
Karthik Kambatla created YARN-1877: -- Summary: ZK store: Add yarn.resourcemanager.zk-state-store.root-node.auth for root node auth Key: YARN-1877 URL: https://issues.apache.org/jira/browse/YARN-1877 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Affects Versions: 2.3.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Priority: Critical -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (YARN-1878) Yarn standby RM taking long to transition to active
Arpit Gupta created YARN-1878: - Summary: Yarn standby RM taking long to transition to active Key: YARN-1878 URL: https://issues.apache.org/jira/browse/YARN-1878 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.4.0 Reporter: Arpit Gupta Assignee: Xuan Gong In our HA tests we are noticing that some times it can take upto 10s for the standby RM to transition to active. -- This message was sent by Atlassian JIRA (v6.2#6252)
Failed: YARN-1878 PreCommit Build #3461
Jira: https://issues.apache.org/jira/browse/YARN-1878
Build: https://builds.apache.org/job/PreCommit-YARN-Build/3461/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 3076 lines...]
NOP
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12636856/YARN-1878.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3461//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3461//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
7f5d27cea31c7828e476b12f68f3ea3a2a486be5 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
All tests passed
RE: Capacity scheduler puts all containers on one box
As I knew,,I never seen RM assigning containers entirely one NM until that NM is fullI might be wrong... But I used to configure following three properties,which will restirict the number of containers for one NM .. Hope following configurations will help you to solve your disk saturation and following formula is obtained based on some tests. # of containers = min (2*CORES, 1.8*DISKS, (Total available RAM) / MIN_CONTAINER_SIZE) property nameyarn.nodemanager.resource.memory-mb/name value40960/value /property property namemapreduce.map.memory.mb/name value4096/value /property property namemapreduce.reduce.memory.mb/name value8192/value /property Thanks Regards Brahma Reddy Battula From: Sandy Ryza [sandy.r...@cloudera.com] Sent: Saturday, March 22, 2014 12:18 PM To: yarn-dev@hadoop.apache.org Subject: Re: Capacity scheduler puts all containers on one box For a work-around, if you turn on DRF / multi-resource scheduling, you could use vcore capacities to limit the number of containers per node? On Fri, Mar 21, 2014 at 11:35 PM, Chris Riccomini criccom...@linkedin.comwrote: Hey Guys, @Vinod: We aren't overriding the default, so we must be using -1 as the setting. @Sandy: We aren't specifying any racks/hosts when sending the resource requests. +1 regarding introducing a similar limit in capacity scheduler. Any recommended work-arounds in the mean time? Our utilization of the grid is very low because we're having to force high memory requests for the containers in order to guarantee a maximum number of containers on a single node (e.g. Set container memory MB set to 17GB to disallow more than 2 containers from being assigned to any one 48GB node). Cheers, Chris On 3/21/14 11:30 PM, Sandy Ryza sandy.r...@cloudera.com wrote: yarn.scheduler.capacity.node-locality-delay will help if the app is requesting containers at particular locations, but won't help spread things out evenly otherwise. The Fair Scheduler attempts an even spread. By default, it only schedules a single container each time it considers a node. Decoupling scheduling from node heartbeats (YARN-1010) makes it so that a high node heartbeat interval doesn't result in this being slow. Now that the Capacity Scheduler has similar capabilities (YARN-1512), it might make sense to introduce a similar limit? -Sandy On Fri, Mar 21, 2014 at 4:42 PM, Vinod Kumar Vavilapalli vino...@apache.org wrote: What's the value for yarn.scheduler.capacity.node-locality-delay? It is -1 by default in 2.2. We fixed the default to be a reasonable 40 (nodes in a rack) in 2.3.0 that should spread containers a bit. Thanks, +Vinod On Mar 21, 2014, at 12:48 PM, Chris Riccomini criccom...@linkedin.com wrote: Hey Guys, We're running YARN 2.2 with the capacity scheduler. Each NM is running with 40G of memory capacity. When we request a series containers with 2G of memory from a single AM, we see the RM assigning them entirely to one NM until that NM is full, and then moving on to the next, and so on. Essentially, we have a grid with 20 nodes, and two are completely full, and the rest are completely empty. This is problematic because our containers use disk heavily, and are completely saturating the disks on the two nodes, which slows all of the containers down on these NMs. 1. Is this expected behavior of the capacity scheduler? What about the fifo scheduler? 2. Is the recommended work around just to increase memory allocation per-container as a proxy for the disk capacity that's required? Given that there's no disk-level isolation, and no disk-level resource, I don't see another way around this. Cheers, Chris -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Capacity scheduler puts all containers on one box
Hi Chris, I agree with Sandy, if you want to distribute container evenly, fair scheduler is a much better choice. If you insist to use capacity scheduler, as Vinod said, if setting yarn.scheduler.capacity.node-locality-delay to the # node in each rack, this will spread the containers to different nodes. One more thing you have to check is the data distribution. If your data is distributed evenly in the cluster, setting yarn.scheduler.capacity.node-locality-delay is useful. But if your data is skewed, the containers will be assigned to part of the nodes due to data locality awareness. On Sat, Mar 22, 2014 at 2:35 PM, Chris Riccomini criccom...@linkedin.comwrote: Hey Guys, @Vinod: We aren't overriding the default, so we must be using -1 as the setting. @Sandy: We aren't specifying any racks/hosts when sending the resource requests. +1 regarding introducing a similar limit in capacity scheduler. Any recommended work-arounds in the mean time? Our utilization of the grid is very low because we're having to force high memory requests for the containers in order to guarantee a maximum number of containers on a single node (e.g. Set container memory MB set to 17GB to disallow more than 2 containers from being assigned to any one 48GB node). Cheers, Chris On 3/21/14 11:30 PM, Sandy Ryza sandy.r...@cloudera.com wrote: yarn.scheduler.capacity.node-locality-delay will help if the app is requesting containers at particular locations, but won't help spread things out evenly otherwise. The Fair Scheduler attempts an even spread. By default, it only schedules a single container each time it considers a node. Decoupling scheduling from node heartbeats (YARN-1010) makes it so that a high node heartbeat interval doesn't result in this being slow. Now that the Capacity Scheduler has similar capabilities (YARN-1512), it might make sense to introduce a similar limit? -Sandy On Fri, Mar 21, 2014 at 4:42 PM, Vinod Kumar Vavilapalli vino...@apache.org wrote: What's the value for yarn.scheduler.capacity.node-locality-delay? It is -1 by default in 2.2. We fixed the default to be a reasonable 40 (nodes in a rack) in 2.3.0 that should spread containers a bit. Thanks, +Vinod On Mar 21, 2014, at 12:48 PM, Chris Riccomini criccom...@linkedin.com wrote: Hey Guys, We're running YARN 2.2 with the capacity scheduler. Each NM is running with 40G of memory capacity. When we request a series containers with 2G of memory from a single AM, we see the RM assigning them entirely to one NM until that NM is full, and then moving on to the next, and so on. Essentially, we have a grid with 20 nodes, and two are completely full, and the rest are completely empty. This is problematic because our containers use disk heavily, and are completely saturating the disks on the two nodes, which slows all of the containers down on these NMs. 1. Is this expected behavior of the capacity scheduler? What about the fifo scheduler? 2. Is the recommended work around just to increase memory allocation per-container as a proxy for the disk capacity that's required? Given that there's no disk-level isolation, and no disk-level resource, I don't see another way around this. Cheers, Chris -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Regards Gordon Wang
[jira] [Created] (YARN-1879) Mark Idempotent/AtMostOnce annotations to ApplicationMasterProtocol and add tests for it
Jian He created YARN-1879: - Summary: Mark Idempotent/AtMostOnce annotations to ApplicationMasterProtocol and add tests for it Key: YARN-1879 URL: https://issues.apache.org/jira/browse/YARN-1879 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He -- This message was sent by Atlassian JIRA (v6.2#6252)
