[jira] [Created] (YARN-3286) RMNode#totalCapability has stale capability after NM is reconnected.
Rohith created YARN-3286: Summary: RMNode#totalCapability has stale capability after NM is reconnected. Key: YARN-3286 URL: https://issues.apache.org/jira/browse/YARN-3286 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Rohith Assignee: Rohith This is found while fixing YARN-3222 mentioned in the comment [link1|https://issues.apache.org/jira/browse/YARN-3222?focusedCommentId=14339799page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14339799] and [link2|https://issues.apache.org/jira/browse/YARN-3222?focusedCommentId=14344739page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14344739] And RMNode#ReconnectNodeTransition clean up : It always remove an old node and add a new node. This need to be examined whether this is really required. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Looking to a Hadoop 3 release
Hi Junping, thanks for your response, I view branch-3 as essentially the same size as our recent 2.x releases, with the exception of incompatible changes like classpath isolation and JDK8 target version. These, while perhaps not revolutionary, are still incompatible, and require a major version bump. I don't see a forking of the community effort, since backports should flow pretty easily from branch-3 to branch-2 the same way they currently can flow from branch-2 to branch-2.6. It's just an extra git commit, not like what we had to deal with in the branch-1 days with a custom backport. Hopefully that addresses your concerns. Thanks, Andrew On Tue, Mar 3, 2015 at 6:12 AM, Junping Du j...@hortonworks.com wrote: Thanks all for good discussions here. +1 on supporting Java 8 ASAP. In addition, I agree that we should separating this effort with cutting down Hadoop 3. IMO, Hadoop is still very cool today, and we should only consider Hadoop 3 until we have revolutionary feature (like YARN for 2.0) which deserve to break fundamental compatibilities. Or it may just cause more distractions for community effort. Just 2 cents. Thanks, Junping From: Akira AJISAKA ajisa...@oss.nttdata.co.jp Sent: Tuesday, March 03, 2015 12:04 PM To: common-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; yarn-dev@hadoop.apache.org Subject: Re: Looking to a Hadoop 3 release Thanks Andrew for bringing this up. +1 mostly looks fine but I'm thinking it's not now to cut branch-3. classpath isolation IMHO, classpath isolation is a good thing to do. We should pay down the technical dept ASAP. I'm willing to help. I'm thinking we can cut branch-3 and release 3.0 alpha after HADOOP-11656 is fixed. That is, I'd like to mark this issue as a blocker for 3.0. I wonder that even if we cut branch-3 now, trunk and branch-3 would be the same for a while. That seems useless. JDK8 As Steve suggested, JDK8 can be in both trunk and branch-2. +1 for moving to JDK8 ASAP. maintaining 2.x For user side, now there is little merit to upgrade to 3.x. More important thing is how long 2.x will be maintained. Therefore we should consider when to stop backporting new features to 2.x, and when to stop maintaining 2.x. I'd like to maintain 2.x as long as possible, at least one year after 3.x GA release. * Other issue What's the current status of HDFS symlink? If HADOOP-10019 requires some incompatible changes, I'd like to include in 3.x. Regards, Akira On 3/2/15 15:19, Andrew Wang wrote: Hi devs, It's been a year and a half since 2.x went GA, and I think we're about due for a 3.x release. Notably, there are two incompatible changes I'd like to call out, that will have a tremendous positive impact for our users. First, classpath isolation being done at HADOOP-11656, which has been a long-standing request from many downstreams and Hadoop users. Second, bumping the source and target JDK version to JDK8 (related to HADOOP-11090), which is important since JDK7 is EOL in April 2015 (two months from now). In the past, we've had issues with our dependencies discontinuing support for old JDKs, so this will future-proof us. Between the two, we'll also have quite an opportunity to clean up and upgrade our dependencies, another common user and developer request. I'd like to propose that we start rolling a series of monthly-ish series of 3.0 alpha releases ASAP, with myself volunteering to take on the RM and other cat herding responsibilities. There are already quite a few changes slated for 3.0 besides the above (for instance the shell script rewrite) so there's already value in a 3.0 alpha, and the more time we give downstreams to integrate, the better. This opens up discussion about inclusion of other changes, but I'm hoping to freeze incompatible changes after maybe two alphas, do a beta (with no further incompat changes allowed), and then finally a 3.x GA. For those keeping track, that means a 3.x GA in about four months. I would also like to stress though that this is not intended to be a big bang release. For instance, it would be great if we could maintain wire compatibility between 2.x and 3.x, so rolling upgrades work. Keeping branch-2 and branch-3 similar also makes backports easier, since we're likely maintaining 2.x for a while yet. Please let me know any comments / concerns related to the above. If people are friendly to the idea, I'd like to cut a branch-3 and start working on the first alpha. Best, Andrew
Re: Looking to a Hadoop 3 release
I want to understand a lot more about the classpath isolation (HADOOP-11656) proposal, specifically, what is proposed and does it have to be tagged as incompatible? That's a bigger change than must setting javac.version=8 in the POM —though given what a fundamental problem it addresses, I'm in favour of doing something there. On 3 March 2015 at 08:05:46, Andrew Wang (andrew.w...@cloudera.commailto:andrew.w...@cloudera.com) wrote: I view branch-3 as essentially the same size as our recent 2.x releases, with the exception of incompatible changes like classpath isolation and JDK8 target version. These, while perhaps not revolutionary, are still incompatible, and require a major version bump.
[jira] [Created] (YARN-3289) Docker images should be downloaded during localization
Ravi Prakash created YARN-3289: -- Summary: Docker images should be downloaded during localization Key: YARN-3289 URL: https://issues.apache.org/jira/browse/YARN-3289 Project: Hadoop YARN Issue Type: Sub-task Reporter: Ravi Prakash We currently call docker run on images while launching containers. If the image size if sufficiently big, the task will timeout. We should download the image we want to run during localization (if possible) to prevent this -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Looking to a Hadoop 3 release
On Mar 3, 2015, at 9:36 AM, Karthik Kambatla ka...@cloudera.com wrote: If we preserve API compat and try to preserve wire compat, I don't see the harm in bumping the major release. If we preserve compatibility, then there is no need to bump major number. It allows us to include several fixes/features in trunk in a release. If we are not actively thinking of a way to release items in trunk, why even have it? What are the fixes and features in trunk that you would like to see get out quickly? Can these be back ported easily to branch 2? sanjay
[jira] [Created] (YARN-3287) TimelineClient kerberos authentication failure uses wrong login context.
Jonathan Eagles created YARN-3287: - Summary: TimelineClient kerberos authentication failure uses wrong login context. Key: YARN-3287 URL: https://issues.apache.org/jira/browse/YARN-3287 Project: Hadoop YARN Issue Type: Bug Reporter: Jonathan Eagles Assignee: Daryn Sharp TimelineClientImpl:doPosting is not wrapped in a doAs, which can cause failure for yarn clients to create timeline domains during job submission. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-3288) Document and fix indentation in the DockerContainerExecutor code
Ravi Prakash created YARN-3288: -- Summary: Document and fix indentation in the DockerContainerExecutor code Key: YARN-3288 URL: https://issues.apache.org/jira/browse/YARN-3288 Project: Hadoop YARN Issue Type: Sub-task Reporter: Ravi Prakash Assignee: Ravi Prakash Priority: Trivial The DockerContainerExecutor has several lines over 80 chars and could use some more documentation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Looking to a Hadoop 3 release
Totally agreed. I just left a comment there on the current state and what is needed. As of now, I think the big (and only?) changes are flipping the default classloader for tasks and splitting the HDFS jar. Thanks, +Vinod On Mar 3, 2015, at 9:02 AM, Steve Loughran ste...@hortonworks.commailto:ste...@hortonworks.com wrote: I want to understand a lot more about the classpath isolation (HADOOP-11656) proposal, specifically, what is proposed and does it have to be tagged as incompatible? That's a bigger change than must setting javac.version=8 in the POM —though given what a fundamental problem it addresses, I'm in favour of doing something there. On 3 March 2015 at 08:05:46, Andrew Wang (andrew.w...@cloudera.commailto:andrew.w...@cloudera.com) wrote: I view branch-3 as essentially the same size as our recent 2.x releases, with the exception of incompatible changes like classpath isolation and JDK8 target version. These, while perhaps not revolutionary, are still incompatible, and require a major version bump.
Re: Looking to a Hadoop 3 release
Between: * removing -finalize * breaking HDFS browsing * changing du’s output (in the 2.7 branch) * changing various names of metrics (either intentionally or otherwise) * changing the JDK release … and probably lots of other stuff in branch-2 I haven’t seen/know about, our best course of action is to: $ git rm hadoop-common-project/hadoop-common/src/site/markdown/Compatibility.md At least this way we as caretakers don’t come across as hypocrits. It’s pretty clear the direction has shown we only care about API compatibility and the rest is ignored when it isn’t “convenient”. [The next time someone tells you that Hadoop is hard to operate, I want you think about this email.] (1) Making 2.7 build with JDK7 led to the *exact* situation I figured it would: now we have a precedent where we just say to the community “You know those guarantees? Yeah, you might as well ignore them because we’re going to change the core component any damn time we feel like it.” We haven’t made a release branch off of trunk since branch-0.23. If anyone thinks that’s healthy, there is some beach property in Alberta you might be interested in as well. Our release cycle came to a screeching halt after 0.20 and we’ve never recovered. However, I offer an alternative. This same circular argument comes up all the time: (2) * There aren’t enough changes in trunk to make a new branch. * We can’t upgrade/change component X because there is no plan to make a new major release. To quote Frozen: Let It Go We’re probably at the point where there aren’t likely to be very many more earth shattering changes to the Hadoop code base. The community has decided instead to push these types of changes as separate projects via incubator to avoid the committer paralysis that this community suffers. Because of this, I don’t think the “enough changes” argument works anymore. Instead, we need to pick a new metric to build a cadence to force regular updates. I’d offer that the “every two years” JDK EOL sets the perfect cadence, matched by many other enterprise and OSS software, and gives us an opportunity to reflect in the version number that the critical component of our software has changed. This cadence allows for people to plan appropriately and know what our roadmap and direction actually is. Folks are more likely to build “real” solutions rather than make compromises that suffer in quality in the name of compatibility simply because they don’t know when their work will actually show up. We’ll have a normal, regular opportunity to update dependencies (regardless of the state of HADOOP-11656). Now, if you’ll excuse me, I have more contributor's patches to go through. (1) FWIW, I made the decision not to worry about backward compatibility in the shell code rewrite when I made the realization that the jsvc log and pid file names were poorly chosen to allow for certain capabilities. Did anyone actually touch them from outside the software? Probably not. But it is still effectively an interface, so off to trunk it went. (2) … and that’s before we even get to the “Version numbers are cheap” arguments that were made during the Great Renames of 0.20 and 0.23.
Failed: YARN-3267 PreCommit Build #6821
Jira: https://issues.apache.org/jira/browse/YARN-3267
Build: https://builds.apache.org/job/PreCommit-YARN-Build/6821/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 3955 lines...]
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice:
org.apache.hadoop.mapred.TestReduceFetch
org.apache.hadoop.yarn.server.timeline.TestTimelineDataManager
org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices
The following test timeouts occurred in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice:
org.apache.hadoop.mapred.TestMRIntermediateDataEncryption
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6821//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6821//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
dc273e394384f9cb157ae25ebedf7d4543702d48 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-YARN-Build #6776
Archived 30 artifacts
Archive block size is 32768
Received 35 blocks and 7975083 bytes
Compression is 12.6%
Took 1 min 24 sec
Description set: MAPREDUCE-5583
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
3 tests failed.
FAILED:
org.apache.hadoop.mapred.TestReduceFetchFromPartialMem.testReduceFromPartialMem
Error Message:
Job failed!
Stack Trace:
java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:838)
at
org.apache.hadoop.mapred.TestReduceFetchFromPartialMem.runJob(TestReduceFetchFromPartialMem.java:300)
at
org.apache.hadoop.mapred.TestReduceFetchFromPartialMem.testReduceFromPartialMem(TestReduceFetchFromPartialMem.java:93)
FAILED:
org.apache.hadoop.yarn.server.timeline.TestTimelineDataManager.testGetOldEntitiesWithOutDomainId
Error Message:
expected:DEFAULT but was:null
Stack Trace:
java.lang.AssertionError: expected:DEFAULT but was:null
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at
org.apache.hadoop.yarn.server.timeline.TestTimelineDataManager.testGetOldEntitiesWithOutDomainId(TestTimelineDataManager.java:97)
FAILED:
org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices.testGetEntitiesWithYarnACLsEnabled
Error Message:
expected:1 but was:2
Stack Trace:
java.lang.AssertionError: expected:1 but was:2
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at
Failed: YARN-2981 PreCommit Build #6822
Jira: https://issues.apache.org/jira/browse/YARN-2981
Build: https://builds.apache.org/job/PreCommit-YARN-Build/6822/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 3464 lines...]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12702254/YARN-2981.patch
against trunk revision e17e5ba.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
org.apache.hadoop.yarn.server.nodemanager.TestDockerContainerExecutorWithMocks
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6822//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6822//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
99435a3c6ed4a3e672223282d341f27b07b36fb9 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-YARN-Build #6776
Archived 15 artifacts
Archive block size is 32768
Received 179 blocks and 3092095 bytes
Compression is 65.5%
Took 2.3 sec
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
1 tests failed.
FAILED:
org.apache.hadoop.yarn.server.nodemanager.TestDockerContainerExecutorWithMocks.testContainerLaunch
Error Message:
expected:[/bin/true, run, --rm, --net=host, --name, CONTAINER_ID, --memory,
0m, --cpu-shares, 0, --user, nobody, -v,
/tmp/nm-local-dir142541929:/tmp/nm-local-dir142541929, -v,
/tmp/userlogs142541929:/tmp/userlogs142541929, -v,
/tmp/temp-1425419244525:/tmp/temp-1425419244525,
sequenceiq/hadoop-docker:2.4.1, bash,
/tmp/temp-1425419244525/launch_container.sh] but was:[/bin/true, run, --rm,
--net=host, --name, CONTAINER_ID, -v,
/tmp/nm-local-dir142541929:/tmp/nm-local-dir142541929, -v,
/tmp/userlogs142541929:/tmp/userlogs142541929, -v,
/tmp/temp-1425419244525:/tmp/temp-1425419244525,
sequenceiq/hadoop-docker:2.4.1, bash,
/tmp/temp-1425419244525/launch_container.sh]
Stack Trace:
java.lang.AssertionError: expected:[/bin/true, run, --rm, --net=host, --name,
CONTAINER_ID, --memory, 0m, --cpu-shares, 0, --user, nobody, -v,
/tmp/nm-local-dir142541929:/tmp/nm-local-dir142541929, -v,
/tmp/userlogs142541929:/tmp/userlogs142541929, -v,
/tmp/temp-1425419244525:/tmp/temp-1425419244525,
sequenceiq/hadoop-docker:2.4.1, bash,
/tmp/temp-1425419244525/launch_container.sh] but was:[/bin/true, run, --rm,
--net=host, --name, CONTAINER_ID, -v,
/tmp/nm-local-dir142541929:/tmp/nm-local-dir142541929, -v,
/tmp/userlogs142541929:/tmp/userlogs142541929, -v,
/tmp/temp-1425419244525:/tmp/temp-1425419244525,
sequenceiq/hadoop-docker:2.4.1, bash,
/tmp/temp-1425419244525/launch_container.sh]
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at
org.apache.hadoop.yarn.server.nodemanager.TestDockerContainerExecutorWithMocks.testContainerLaunch(TestDockerContainerExecutorWithMocks.java:241)
Failed: YARN-2190 PreCommit Build #6826
Jira: https://issues.apache.org/jira/browse/YARN-2190
Build: https://builds.apache.org/job/PreCommit-YARN-Build/6826/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 20 lines...]
[PreCommit-YARN-Build] $ /bin/bash /tmp/hudson7099632601958480760.sh
Running in Jenkins mode
==
==
Testing patch for YARN-2190.
==
==
HEAD is now at e17e5ba YARN-3272. Surface container locality info in RM web UI
(Jian He via wangda)
Switched to branch 'trunk'
Your branch is up-to-date with 'origin/trunk'.
Current branch trunk is up to date.
YARN-2190 patch is being downloaded at Tue Mar 3 22:10:29 UTC 2015 from
http://issues.apache.org/jira/secure/attachment/12702264/YARN-2190.10.patch
cp: cannot stat '/home/jenkins/buildSupport/lib/*': No such file or directory
The patch does not appear to apply with p0 to p2
PATCH APPLICATION FAILED
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12702264/YARN-2190.10.patch
against trunk revision e17e5ba.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6826//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
7abb1ca31dadb5e7d13e04716b49b791163e3a6d logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
No tests ran.
Failed: YARN-2190 PreCommit Build #6823
Jira: https://issues.apache.org/jira/browse/YARN-2190
Build: https://builds.apache.org/job/PreCommit-YARN-Build/6823/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 23 lines...]
==
==
Testing patch for YARN-2190.
==
==
HEAD is now at e17e5ba YARN-3272. Surface container locality info in RM web UI
(Jian He via wangda)
Previous HEAD position was e17e5ba... YARN-3272. Surface container locality
info in RM web UI (Jian He via wangda)
Switched to branch 'trunk'
Your branch is behind 'origin/trunk' by 2 commits, and can be fast-forwarded.
(use git pull to update your local branch)
First, rewinding head to replay your work on top of it...
Fast-forwarded trunk to e17e5ba9d7e2bd45ba6884f59f8045817594b284.
YARN-2190 patch is being downloaded at Tue Mar 3 21:07:53 UTC 2015 from
http://issues.apache.org/jira/secure/attachment/12702229/YARN-2190.9.patch
cp: cannot stat '/home/jenkins/buildSupport/lib/*': No such file or directory
The patch does not appear to apply with p0 to p2
PATCH APPLICATION FAILED
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12702229/YARN-2190.9.patch
against trunk revision e17e5ba.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6823//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
2202e88fe387bcf050b5e314c572c46db56ee68c logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
No tests ran.
Failed: YARN-3242 PreCommit Build #6824
Jira: https://issues.apache.org/jira/browse/YARN-3242
Build: https://builds.apache.org/job/PreCommit-YARN-Build/6824/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 4400 lines...]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12702245/YARN-3242.004.patch
against trunk revision e17e5ba.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1236 javac
compiler warnings (more than the trunk's current 1199 warnings).
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
34 warning messages.
See
https://builds.apache.org/job/PreCommit-YARN-Build/6824//artifact/patchprocess/diffJavadocWarnings.txt
for details.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws:
org.apache.hadoop.ha.TestZKFailoverController
org.apache.hadoop.ha.TestZKFailoverControllerStress
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6824//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6824//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6824//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
a53f62c9288b02fb3290acf5a34b6b48367b28d9 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
3 tests failed.
FAILED: org.apache.hadoop.ha.TestZKFailoverController.testZooKeeperFailure
Error Message:
Address already in use
Stack Trace:
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:444)
at sun.nio.ch.Net.bind(Net.java:436)
at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
at
org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:95)
at
org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:126)
at
org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:119)
at
org.apache.hadoop.ha.ClientBaseWithFixes.createNewServerInstance(ClientBaseWithFixes.java:341)
at
org.apache.hadoop.ha.ClientBaseWithFixes.startServer(ClientBaseWithFixes.java:438)
at
org.apache.hadoop.ha.ClientBaseWithFixes.setUp(ClientBaseWithFixes.java:402)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at
[jira] [Created] (YARN-3291) DockerContainerExecutor should run as a non-root user
Abin Shahab created YARN-3291: - Summary: DockerContainerExecutor should run as a non-root user Key: YARN-3291 URL: https://issues.apache.org/jira/browse/YARN-3291 Project: Hadoop YARN Issue Type: Improvement Reporter: Abin Shahab Currently DockerContainerExecutor runs container as root. This can be run as the user which is not root. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-3290) DockerContainerExecutor should optionally limit memory and cpu
Abin Shahab created YARN-3290: - Summary: DockerContainerExecutor should optionally limit memory and cpu Key: YARN-3290 URL: https://issues.apache.org/jira/browse/YARN-3290 Project: Hadoop YARN Issue Type: Improvement Reporter: Abin Shahab Currently, DockerContainerExecutor does not set cgroup limits on memory and cpu. It should follow LCE's example to set cgroup limits. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Looking to a Hadoop 3 release
I started pitching in more on that JIRA. To add, I think we can and should strive for doing this in a compatible manner, whatever the approach. Marking and calling it incompatible before we see proposal/patch seems premature to me. Commented the same on JIRA: https://issues.apache.org/jira/browse/HADOOP-11656?focusedCommentId=14345875page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14345875. Thanks +Vinod On Mar 2, 2015, at 8:08 PM, Andrew Wang andrew.w...@cloudera.commailto:andrew.w...@cloudera.com wrote: Regarding classpath isolation, based on what I hear from our customers, it's still a big problem (even after the MR classloader work). The latest Jackson version bump was quite painful for our downstream projects, and the HDFS client still leaks a lot of dependencies. Would welcome more discussion of this on HADOOP-11656, Steve, Colin, and Haohui have already chimed in.
[jira] [Created] (YARN-3292) [Umbrella] Tests and/or tools for YARN backwards compatibility verification
Li Lu created YARN-3292: --- Summary: [Umbrella] Tests and/or tools for YARN backwards compatibility verification Key: YARN-3292 URL: https://issues.apache.org/jira/browse/YARN-3292 Project: Hadoop YARN Issue Type: Improvement Reporter: Li Lu Assignee: Li Lu YARN-666 added the support to YARN rolling upgrade. In order to support this feature, we made changes from many perspectives. There were many assumptions made together with these existing changes. Future code changes may break these assumptions by accident, and hence break the YARN rolling upgrades feature. To simplify YARN RU regression tests, maybe we would like to create a set of tools/tests that can verify YARN RU backward compatibility. On the very first step, we may want to have a compatibility checker for important protocols and APIs. We may also want to incorporate these tools into our test Jenkins runs, if necessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Failed: YARN-3222 PreCommit Build #6828
Jira: https://issues.apache.org/jira/browse/YARN-3222
Build: https://builds.apache.org/job/PreCommit-YARN-Build/6828/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 3355 lines...]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12702276/0005-YARN-3222.patch
against trunk revision e17e5ba.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1151 javac
compiler warnings (more than the trunk's current 185 warnings).
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
43 warning messages.
See
https://builds.apache.org/job/PreCommit-YARN-Build/6828//artifact/patchprocess/diffJavadocWarnings.txt
for details.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-tools/hadoop-distcp.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6828//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6828//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6828//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
01e2ac90981715123fb1a8a1a358f32a82700041 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-YARN-Build #6825
Archived 16 artifacts
Archive block size is 32768
Received 19 blocks and 7974784 bytes
Compression is 7.2%
Took 1.7 sec
[description-setter] Could not determine description.
Recording test results
Publish JUnit test result report is waiting for a checkpoint on
PreCommit-YARN-Build #6827
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
All tests passed
Failed: YARN-3242 PreCommit Build #6827
Jira: https://issues.apache.org/jira/browse/YARN-3242
Build: https://builds.apache.org/job/PreCommit-YARN-Build/6827/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 4784 lines...]
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.security.TestAMRMTokens
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStoreZKClientConnections
The following test timeouts occurred in
hadoop-common-project/hadoop-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6827//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6827//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6827//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
4500e1d46cb67edf12a19eb0a7c1478a5b104d82 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-YARN-Build #6825
ERROR: Failed to archive
{patchprocess/patchJavacWarnings.txt=patchprocess/patchJavacWarnings.txt,
patchprocess/patchJavadocWarnings.txt=patchprocess/patchJavadocWarnings.txt,
patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.xml=patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.xml,
patchprocess/trunkJavadocWarnings.txt=patchprocess/trunkJavadocWarnings.txt,
patchprocess/patchFindBugsOutputhadoop-common.txt=patchprocess/patchFindBugsOutputhadoop-common.txt,
patchprocess/patchFindbugsWarningshadoop-hdfs.xml=patchprocess/patchFindbugsWarningshadoop-hdfs.xml,
patchprocess/newPatchFindbugsWarningshadoop-hdfs.html=patchprocess/newPatchFindbugsWarningshadoop-hdfs.html,
patchprocess/testrun_hadoop-common.txt=patchprocess/testrun_hadoop-common.txt,
patchprocess/diffJavadocWarnings.txt=patchprocess/diffJavadocWarnings.txt,
patchprocess/newPatchFindbugsWarningshadoop-hdfs.xml=patchprocess/newPatchFindbugsWarningshadoop-hdfs.xml,
patchprocess/patchFindBugsOutputhadoop-yarn-server-resourcemanager.txt=patchprocess/patchFindBugsOutputhadoop-yarn-server-resourcemanager.txt,
patchprocess/patchFindBugsOutputhadoop-hdfs.txt=patchprocess/patchFindBugsOutputhadoop-hdfs.txt,
patchprocess/trunkJavacWarnings.txt=patchprocess/trunkJavacWarnings.txt,
patchprocess/testrun_hadoop-hdfs.txt=patchprocess/testrun_hadoop-hdfs.txt,
patchprocess/newPatchFindbugsWarningshadoop-common.xml=patchprocess/newPatchFindbugsWarningshadoop-common.xml,
patchprocess/newPatchFindbugsWarningshadoop-common.html=patchprocess/newPatchFindbugsWarningshadoop-common.html,
patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html=patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html,
patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt=patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt,
patchprocess/patchReleaseAuditOutput.txt=patchprocess/patchReleaseAuditOutput.txt,
patchprocess/patchReleaseAuditWarnings.txt=patchprocess/patchReleaseAuditWarnings.txt,
patchprocess/patchFindbugsWarningshadoop-common.xml=patchprocess/patchFindbugsWarningshadoop-common.xml,
patchprocess/filteredPatchJavacWarnings.txt=patchprocess/filteredPatchJavacWarnings.txt,
patchprocess/patchFindbugsWarningshadoop-yarn-server-resourcemanager.xml=patchprocess/patchFindbugsWarningshadoop-yarn-server-resourcemanager.xml,
patchprocess/patchEclipseOutput.txt=patchprocess/patchEclipseOutput.txt,
patchprocess/filteredTrunkJavacWarnings.txt=patchprocess/filteredTrunkJavacWarnings.txt}
due to internal error; falling back to full archiving
java.lang.IllegalStateException:
Success: YARN-2786 PreCommit Build #6830
Jira: https://issues.apache.org/jira/browse/YARN-2786
Build: https://builds.apache.org/job/PreCommit-YARN-Build/6830/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 3258 lines...]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12702312/YARN-2786-20150303-1-trunk.patch
against trunk revision 5af693f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6830//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6830//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
7da5d10cc9f8a251e8741120319d8070b3e0fad8 logged out
==
==
Finished build.
==
==
Archiving artifacts
Sending artifact delta relative to PreCommit-YARN-Build #6825
Archived 15 artifacts
Archive block size is 32768
Received 52 blocks and 7094458 bytes
Compression is 19.4%
Took 1.6 sec
Description set: YARN-2786
Recording test results
Publish JUnit test result report is waiting for a checkpoint on
PreCommit-YARN-Build #6829
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
Failed: YARN-3267 PreCommit Build #6829
Jira: https://issues.apache.org/jira/browse/YARN-3267
Build: https://builds.apache.org/job/PreCommit-YARN-Build/6829/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 5604 lines...]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12702284/YARN_3267_WIP3.patch
against trunk revision e17e5ba.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The test build failed in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6829//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6829//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
8de9acaf223a6b938d496faa66896afccf653e89 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-YARN-Build #6825
Archived 30 artifacts
Archive block size is 32768
Received 234 blocks and 1512667 bytes
Compression is 83.5%
Took 0.81 sec
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Resolved] (YARN-3285) Convert branch-2 .apt.vm files of YARN to markdown
[ https://issues.apache.org/jira/browse/YARN-3285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He resolved YARN-3285. --- Resolution: Fixed Fix Version/s: 2.7.0 Hadoop Flags: Reviewed Committed to branch-2, thanks [~iwasakims] ! Convert branch-2 .apt.vm files of YARN to markdown -- Key: YARN-3285 URL: https://issues.apache.org/jira/browse/YARN-3285 Project: Hadoop YARN Issue Type: Improvement Components: documentation Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki Fix For: 2.7.0 Attachments: YARN-3285.001.patch Backport the conversion to markdown done in YARN-3168. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Success: YARN-2981 PreCommit Build #6825
Jira: https://issues.apache.org/jira/browse/YARN-2981
Build: https://builds.apache.org/job/PreCommit-YARN-Build/6825/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 3448 lines...]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12702266/YARN-2981.patch
against trunk revision e17e5ba.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6825//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6825//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
1b9907169cbe3b4988228d1d261d843f6e5c0814 logged out
==
==
Finished build.
==
==
Archiving artifacts
Sending artifact delta relative to PreCommit-YARN-Build #6776
Archived 15 artifacts
Archive block size is 32768
Received 180 blocks and 3057829 bytes
Compression is 65.9%
Took 0.93 sec
Description set: YARN-2981
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
Re: Looking to a Hadoop 3 release
Awesome, looks like we can just do this in a compatible manner - nothing else on the list seems like it warrants a (premature) major release. Thanks Vinod. Arun From: Vinod Kumar Vavilapalli vino...@hortonworks.com Sent: Tuesday, March 03, 2015 2:30 PM To: common-...@hadoop.apache.org Cc: hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org Subject: Re: Looking to a Hadoop 3 release I started pitching in more on that JIRA. To add, I think we can and should strive for doing this in a compatible manner, whatever the approach. Marking and calling it incompatible before we see proposal/patch seems premature to me. Commented the same on JIRA: https://issues.apache.org/jira/browse/HADOOP-11656?focusedCommentId=14345875page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14345875. Thanks +Vinod On Mar 2, 2015, at 8:08 PM, Andrew Wang andrew.w...@cloudera.commailto:andrew.w...@cloudera.com wrote: Regarding classpath isolation, based on what I hear from our customers, it's still a big problem (even after the MR classloader work). The latest Jackson version bump was quite painful for our downstream projects, and the HDFS client still leaks a lot of dependencies. Would welcome more discussion of this on HADOOP-11656, Steve, Colin, and Haohui have already chimed in.
Failed: YARN-3264 PreCommit Build #6832
Jira: https://issues.apache.org/jira/browse/YARN-3264
Build: https://builds.apache.org/job/PreCommit-YARN-Build/6832/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 23 lines...]
==
==
Testing patch for YARN-3264.
==
==
HEAD is now at 5af693f MAPREDUCE-6248. Exposed the internal MapReduce job's
information as a public API in DistCp. Contributed by Jing Zhao.
Previous HEAD position was 5af693f... MAPREDUCE-6248. Exposed the internal
MapReduce job's information as a public API in DistCp. Contributed by Jing Zhao.
Switched to branch 'trunk'
Your branch is behind 'origin/trunk' by 3 commits, and can be fast-forwarded.
(use git pull to update your local branch)
First, rewinding head to replay your work on top of it...
Fast-forwarded trunk to 5af693fde26755b6f175bd65f93cf4a80de0d1e0.
YARN-3264 patch is being downloaded at Wed Mar 4 01:50:25 UTC 2015 from
http://issues.apache.org/jira/secure/attachment/12702338/YARN-3264.001.patch
cp: cannot stat '/home/jenkins/buildSupport/lib/*': No such file or directory
The patch does not appear to apply with p0 to p2
PATCH APPLICATION FAILED
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12702338/YARN-3264.001.patch
against trunk revision 5af693f.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6832//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
9e3e2321ef0e50b83f0960ff37f3a47946b9df9e logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Description set: MAPREDUCE-6248
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
No tests ran.
Failed: YARN-1809 PreCommit Build #6831
Jira: https://issues.apache.org/jira/browse/YARN-1809
Build: https://builds.apache.org/job/PreCommit-YARN-Build/6831/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 3954 lines...]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12702329/YARN-1809.15-rebase.patch
against trunk revision 5af693f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 7 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6831//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6831//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6831//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
c71228647e9a26089dc95544a80c20fd4046e108 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-YARN-Build #6830
Archived 35 artifacts
Archive block size is 32768
Received 49 blocks and 8354970 bytes
Compression is 16.1%
Took 1.8 sec
Description set: MAPREDUCE-6248
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
All tests passed
Failed: YARN-3264 PreCommit Build #6834
Jira: https://issues.apache.org/jira/browse/YARN-3264
Build: https://builds.apache.org/job/PreCommit-YARN-Build/6834/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 23 lines...]
==
==
Testing patch for YARN-3264.
==
==
HEAD is now at f2d7a67 HDFS-7682. {{DistributedFileSystem#getFileChecksum}} of
a snapshotted file includes non-snapshotted content. Contributed by Charles
Lamb.
Previous HEAD position was f2d7a67... HDFS-7682.
{{DistributedFileSystem#getFileChecksum}} of a snapshotted file includes
non-snapshotted content. Contributed by Charles Lamb.
Switched to branch 'trunk'
Your branch is behind 'origin/trunk' by 5 commits, and can be fast-forwarded.
(use git pull to update your local branch)
First, rewinding head to replay your work on top of it...
Fast-forwarded trunk to f2d7a67a2c1d9dde10ed3171fdec65dff885afcc.
YARN-3264 patch is being downloaded at Wed Mar 4 04:10:51 UTC 2015 from
http://issues.apache.org/jira/secure/attachment/12702362/YARN-3264.002.patch
cp: cannot stat '/home/jenkins/buildSupport/lib/*': No such file or directory
The patch does not appear to apply with p0 to p2
PATCH APPLICATION FAILED
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12702362/YARN-3264.002.patch
against trunk revision f2d7a67.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6834//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
6b0258aeafc9659f5b849f9ffe935e1a2e73f2f3 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
No tests ran.
Failed: YARN-1809 PreCommit Build #6833
Jira: https://issues.apache.org/jira/browse/YARN-1809
Build: https://builds.apache.org/job/PreCommit-YARN-Build/6833/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 4001 lines...]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12702345/YARN-1809.16.patch
against trunk revision f2d7a67.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 7 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.timeline.security.TestTimelineAuthenticationFilter
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6833//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6833//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6833//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
765398a4b9117d050eb2296a556a720479e7f0f3 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-YARN-Build #6830
Archived 35 artifacts
Archive block size is 32768
Received 38 blocks and 8719211 bytes
Compression is 12.5%
Took 1.8 sec
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
1 tests failed.
REGRESSION:
org.apache.hadoop.yarn.server.timeline.security.TestTimelineAuthenticationFilter.testPutTimelineEntities[1]
Error Message:
java.net.SocketTimeoutException: Read timed out
Stack Trace:
com.sun.jersey.api.client.ClientHandlerException:
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
at sun.security.ssl.InputRecord.read(InputRecord.java:480)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
at
sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312)
at
sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339)
at
sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1323)
at
sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:563)
at
sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
at
sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153)
at
org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:190)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:128)
at
Re: Looking to a Hadoop 3 release
I am surprised classpath-isolation is being called a minor issue. We have been hearing users complain about Hadoop leaking its dependencies into the classpath for a while now, Guava being the culprit often. Not being able to upgrade our dependencies without affecting users has started to hamper our development too; e.g. Guava conflict with upgrading Curator version. If we preserve API compat and try to preserve wire compat, I don't see the harm in bumping the major release. It allows us to include several fixes/features in trunk in a release. If we are not actively thinking of a way to release items in trunk, why even have it? If there are any disadvantages to doing a major release, I would like to know. May be, we could arrive at a plan to accomplish it without those problems. Thanks Karthik On Tue, Mar 3, 2015 at 9:02 AM, Steve Loughran ste...@hortonworks.com wrote: I want to understand a lot more about the classpath isolation (HADOOP-11656) proposal, specifically, what is proposed and does it have to be tagged as incompatible? That's a bigger change than must setting javac.version=8 in the POM —though given what a fundamental problem it addresses, I'm in favour of doing something there. On 3 March 2015 at 08:05:46, Andrew Wang (andrew.w...@cloudera.com) wrote: I view branch-3 as essentially the same size as our recent 2.x releases, with the exception of incompatible changes like classpath isolation and JDK8 target version. These, while perhaps not revolutionary, are still incompatible, and require a major version bump. -- Karthik Kambatla Software Engineer, Cloudera Inc. http://five.sentenc.es
Re: Looking to a Hadoop 3 release
Thanks all for good discussions here. +1 on supporting Java 8 ASAP. In addition, I agree that we should separating this effort with cutting down Hadoop 3. IMO, Hadoop is still very cool today, and we should only consider Hadoop 3 until we have revolutionary feature (like YARN for 2.0) which deserve to break fundamental compatibilities. Or it may just cause more distractions for community effort. Just 2 cents. Thanks, Junping From: Akira AJISAKA ajisa...@oss.nttdata.co.jp Sent: Tuesday, March 03, 2015 12:04 PM To: common-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; yarn-dev@hadoop.apache.org Subject: Re: Looking to a Hadoop 3 release Thanks Andrew for bringing this up. +1 mostly looks fine but I'm thinking it's not now to cut branch-3. classpath isolation IMHO, classpath isolation is a good thing to do. We should pay down the technical dept ASAP. I'm willing to help. I'm thinking we can cut branch-3 and release 3.0 alpha after HADOOP-11656 is fixed. That is, I'd like to mark this issue as a blocker for 3.0. I wonder that even if we cut branch-3 now, trunk and branch-3 would be the same for a while. That seems useless. JDK8 As Steve suggested, JDK8 can be in both trunk and branch-2. +1 for moving to JDK8 ASAP. maintaining 2.x For user side, now there is little merit to upgrade to 3.x. More important thing is how long 2.x will be maintained. Therefore we should consider when to stop backporting new features to 2.x, and when to stop maintaining 2.x. I'd like to maintain 2.x as long as possible, at least one year after 3.x GA release. * Other issue What's the current status of HDFS symlink? If HADOOP-10019 requires some incompatible changes, I'd like to include in 3.x. Regards, Akira On 3/2/15 15:19, Andrew Wang wrote: Hi devs, It's been a year and a half since 2.x went GA, and I think we're about due for a 3.x release. Notably, there are two incompatible changes I'd like to call out, that will have a tremendous positive impact for our users. First, classpath isolation being done at HADOOP-11656, which has been a long-standing request from many downstreams and Hadoop users. Second, bumping the source and target JDK version to JDK8 (related to HADOOP-11090), which is important since JDK7 is EOL in April 2015 (two months from now). In the past, we've had issues with our dependencies discontinuing support for old JDKs, so this will future-proof us. Between the two, we'll also have quite an opportunity to clean up and upgrade our dependencies, another common user and developer request. I'd like to propose that we start rolling a series of monthly-ish series of 3.0 alpha releases ASAP, with myself volunteering to take on the RM and other cat herding responsibilities. There are already quite a few changes slated for 3.0 besides the above (for instance the shell script rewrite) so there's already value in a 3.0 alpha, and the more time we give downstreams to integrate, the better. This opens up discussion about inclusion of other changes, but I'm hoping to freeze incompatible changes after maybe two alphas, do a beta (with no further incompat changes allowed), and then finally a 3.x GA. For those keeping track, that means a 3.x GA in about four months. I would also like to stress though that this is not intended to be a big bang release. For instance, it would be great if we could maintain wire compatibility between 2.x and 3.x, so rolling upgrades work. Keeping branch-2 and branch-3 similar also makes backports easier, since we're likely maintaining 2.x for a while yet. Please let me know any comments / concerns related to the above. If people are friendly to the idea, I'd like to cut a branch-3 and start working on the first alpha. Best, Andrew
Build failed in Jenkins: Hadoop-Yarn-trunk-Java8 #121
See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/121/changes Changes: [jianhe] YARN-3281. Added RMStateStore to StateMachine visualization list. Contributed by Chengbing Liu [wheat9] HDFS-7785. Improve diagnostics information for HttpPutFailedException. Contributed by Chengbing Liu. [wangda] YARN-3270. Fix node label expression not getting set in ApplicationSubmissionContext (Rohit Agarwal via wangda) [vinodkv] YARN-3265. Fixed a deadlock in CapacityScheduler by always passing a queue's available resource-limit from the parent queue. Contributed by Wangda Tan. [szetszwo] HDFS-7302. Remove downgrade from namenode -rollingUpgrade startup option since it may incorrectly finalize an ongoing rolling upgrade. [gera] HADOOP-11605. FilterFileSystem#create with ChecksumOpt should propagate it to wrapped FS. (gera) [jing9] HDFS-7871. NameNodeEditLogRoller can keep printing 'Swallowing exception' message. Contributed by Jing Zhao. [ozawa] HADOOP-11602. Fix toUpperCase/toLowerCase to use Locale.ENGLISH. (ozawa) [ozawa] MAPREDUCE-6268. Fix typo in Task Attempt API's URL. Contributed by Ryu Kobayashi. [ozawa] MAPREDUCE-5657. Fix Javadoc errors caused by incorrect or illegal tags in doc comments. Contributed by Akira AJISAKA. [junping_du] MAPREDUCE-5583. Ability to limit running map and reduce tasks. Contributed by Jason Lowe. -- [...truncated 4273 lines...] at org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC.getServer(HadoopYarnProtoRPC.java:65) at org.apache.hadoop.yarn.ipc.YarnRPC.getServer(YarnRPC.java:54) at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceStart(ContainerManagerImpl.java:421) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:287) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerShutdown.testKillContainersOnShutdown(TestNodeManagerShutdown.java:146) Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.yarn.server.nodemanager.TestDockerContainerExecutorWithMocks Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.958 sec - in org.apache.hadoop.yarn.server.nodemanager.TestDockerContainerExecutorWithMocks Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.yarn.server.nodemanager.metrics.TestNodeManagerMetrics Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.716 sec - in org.apache.hadoop.yarn.server.nodemanager.metrics.TestNodeManagerMetrics Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.yarn.server.nodemanager.TestLocalDirsHandlerService Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.668 sec - in org.apache.hadoop.yarn.server.nodemanager.TestLocalDirsHandlerService Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerResync Tests run: 5, Failures: 0, Errors: 5, Skipped: 0, Time elapsed: 1.705 sec FAILURE! - in org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerResync testNMshutdownWhenResyncThrowException(org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerResync) Time elapsed: 0.913 sec ERROR! org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.net.BindException: Problem binding to [localhost:12345] java.net.BindException: Address already in use; For more details see: http://wiki.apache.org/hadoop/BindException at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:414) at sun.nio.ch.Net.bind(Net.java:406) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) at org.apache.hadoop.ipc.Server.bind(Server.java:411) at org.apache.hadoop.ipc.Server$Listener.init(Server.java:577) at org.apache.hadoop.ipc.Server.init(Server.java:2301) at org.apache.hadoop.ipc.RPC$Server.init(RPC.java:942) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server.init(ProtobufRpcEngine.java:534) at org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:509) at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:787) at org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.createServer(RpcServerFactoryPBImpl.java:173) at
Hadoop-Yarn-trunk-Java8 - Build # 121 - Still Failing
See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/121/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 4466 lines...] [INFO] [INFO] Reactor Summary: [INFO] [INFO] hadoop-yarn ... SUCCESS [ 3.520 s] [INFO] hadoop-yarn-api ... SUCCESS [01:00 min] [INFO] hadoop-yarn-common SUCCESS [02:09 min] [INFO] hadoop-yarn-server SUCCESS [ 0.071 s] [INFO] hadoop-yarn-server-common . SUCCESS [ 34.996 s] [INFO] hadoop-yarn-server-nodemanager FAILURE [05:02 min] [INFO] hadoop-yarn-server-web-proxy .. SKIPPED [INFO] hadoop-yarn-server-applicationhistoryservice .. SKIPPED [INFO] hadoop-yarn-server-resourcemanager SKIPPED [INFO] hadoop-yarn-server-tests .. SKIPPED [INFO] hadoop-yarn-client SKIPPED [INFO] hadoop-yarn-server-sharedcachemanager . SKIPPED [INFO] hadoop-yarn-applications .. SKIPPED [INFO] hadoop-yarn-applications-distributedshell . SKIPPED [INFO] hadoop-yarn-applications-unmanaged-am-launcher SKIPPED [INFO] hadoop-yarn-site .. SKIPPED [INFO] hadoop-yarn-registry .. SKIPPED [INFO] hadoop-yarn-project ... SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 08:51 min [INFO] Finished at: 2015-03-03T10:41:22+00:00 [INFO] Final Memory: 81M/903M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on project hadoop-yarn-server-nodemanager: There are test failures. [ERROR] [ERROR] Please refer to /home/jenkins/jenkins-slave/workspace/Hadoop-Yarn-trunk-Java8/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/surefire-reports for the individual test results. [ERROR] - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn goals -rf :hadoop-yarn-server-nodemanager Build step 'Execute shell' marked build as failure Archiving artifacts Recording test results Updating HDFS-7302 Updating YARN-3270 Updating MAPREDUCE-5657 Updating MAPREDUCE-6268 Updating HDFS-7871 Updating YARN-3265 Updating HADOOP-11605 Updating YARN-3281 Updating HDFS-7785 Updating MAPREDUCE-5583 Updating HADOOP-11602 Sending e-mails to: yarn-dev@hadoop.apache.org Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## 8 tests failed. REGRESSION: org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerReboot.testClearLocalDirWhenNodeReboot Error Message: java.net.BindException: Problem binding to [localhost:12345] java.net.BindException: Address already in use; For more details see: http://wiki.apache.org/hadoop/BindException Stack Trace: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.net.BindException: Problem binding to [localhost:12345] java.net.BindException: Address already in use; For more details see: http://wiki.apache.org/hadoop/BindException at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:414) at sun.nio.ch.Net.bind(Net.java:406) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) at org.apache.hadoop.ipc.Server.bind(Server.java:411) at org.apache.hadoop.ipc.Server$Listener.init(Server.java:577) at org.apache.hadoop.ipc.Server.init(Server.java:2301) at org.apache.hadoop.ipc.RPC$Server.init(RPC.java:942) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server.init(ProtobufRpcEngine.java:534) at org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:509) at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:787) at
Failed: YARN-3222 PreCommit Build #6818
Jira: https://issues.apache.org/jira/browse/YARN-3222
Build: https://builds.apache.org/job/PreCommit-YARN-Build/6818/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 3446 lines...]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12702122/0004-YARN-3222.patch
against trunk revision 9ae7f9e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 5 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6818//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6818//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6818//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
35ea4b946e8d33455e1557402e28ebb4b4177084 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-YARN-Build #6776
Archived 15 artifacts
Archive block size is 32768
Received 181 blocks and 3142398 bytes
Compression is 65.4%
Took 1.2 sec
Description set: MAPREDUCE-5657
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
All tests passed
Build failed in Jenkins: Hadoop-Yarn-trunk #855
See https://builds.apache.org/job/Hadoop-Yarn-trunk/855/changes Changes: [jianhe] YARN-3281. Added RMStateStore to StateMachine visualization list. Contributed by Chengbing Liu [wheat9] HDFS-7785. Improve diagnostics information for HttpPutFailedException. Contributed by Chengbing Liu. [wangda] YARN-3270. Fix node label expression not getting set in ApplicationSubmissionContext (Rohit Agarwal via wangda) [vinodkv] YARN-3265. Fixed a deadlock in CapacityScheduler by always passing a queue's available resource-limit from the parent queue. Contributed by Wangda Tan. [szetszwo] HDFS-7302. Remove downgrade from namenode -rollingUpgrade startup option since it may incorrectly finalize an ongoing rolling upgrade. [gera] HADOOP-11605. FilterFileSystem#create with ChecksumOpt should propagate it to wrapped FS. (gera) [jing9] HDFS-7871. NameNodeEditLogRoller can keep printing 'Swallowing exception' message. Contributed by Jing Zhao. [ozawa] HADOOP-11602. Fix toUpperCase/toLowerCase to use Locale.ENGLISH. (ozawa) [ozawa] MAPREDUCE-6268. Fix typo in Task Attempt API's URL. Contributed by Ryu Kobayashi. [ozawa] MAPREDUCE-5657. Fix Javadoc errors caused by incorrect or illegal tags in doc comments. Contributed by Akira AJISAKA. [junping_du] MAPREDUCE-5583. Ability to limit running map and reduce tasks. Contributed by Jason Lowe. -- [...truncated 8373 lines...] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.706 sec - in org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStoreZKClientConnections Running org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.679 sec - in org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore Running org.apache.hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.936 sec - in org.apache.hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore Running org.apache.hadoop.yarn.server.resourcemanager.rmapp.TestRMAppTransitions Tests run: 44, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.31 sec - in org.apache.hadoop.yarn.server.resourcemanager.rmapp.TestRMAppTransitions Running org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.TestRMAppAttemptTransitions Tests run: 70, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 5.535 sec - in org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.TestRMAppAttemptTransitions Running org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.TestAMLivelinessMonitor Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.83 sec - in org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.TestAMLivelinessMonitor Running org.apache.hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 221.864 sec - in org.apache.hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart Running org.apache.hadoop.yarn.server.resourcemanager.TestApplicationCleanup Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 51.778 sec - in org.apache.hadoop.yarn.server.resourcemanager.TestApplicationCleanup Running org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.958 sec - in org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA Running org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.367 sec - in org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService Running org.apache.hadoop.yarn.server.resourcemanager.nodelabels.TestRMNodeLabelsManager Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.013 sec - in org.apache.hadoop.yarn.server.resourcemanager.nodelabels.TestRMNodeLabelsManager Running org.apache.hadoop.yarn.server.resourcemanager.TestMoveApplication Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.807 sec - in org.apache.hadoop.yarn.server.resourcemanager.TestMoveApplication Running org.apache.hadoop.yarn.server.resourcemanager.TestApplicationMasterLauncher Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.317 sec - in org.apache.hadoop.yarn.server.resourcemanager.TestApplicationMasterLauncher Running org.apache.hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.575 sec - in org.apache.hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy Running org.apache.hadoop.yarn.server.resourcemanager.reservation.TestNoOverCommitPolicy Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.531 sec - in
Hadoop-Yarn-trunk - Build # 855 - Failure
See https://builds.apache.org/job/Hadoop-Yarn-trunk/855/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 8566 lines...] Tests run: 1011, Failures: 1, Errors: 0, Skipped: 1 [INFO] [INFO] Reactor Summary: [INFO] [INFO] hadoop-yarn ... SUCCESS [ 3.696 s] [INFO] hadoop-yarn-api ... SUCCESS [03:17 min] [INFO] hadoop-yarn-common SUCCESS [02:44 min] [INFO] hadoop-yarn-server SUCCESS [ 0.095 s] [INFO] hadoop-yarn-server-common . SUCCESS [ 40.433 s] [INFO] hadoop-yarn-server-nodemanager SUCCESS [06:01 min] [INFO] hadoop-yarn-server-web-proxy .. SUCCESS [ 22.672 s] [INFO] hadoop-yarn-server-applicationhistoryservice .. SUCCESS [03:01 min] [INFO] hadoop-yarn-server-resourcemanager FAILURE [44:21 min] [INFO] hadoop-yarn-server-tests .. SKIPPED [INFO] hadoop-yarn-client SKIPPED [INFO] hadoop-yarn-server-sharedcachemanager . SKIPPED [INFO] hadoop-yarn-applications .. SKIPPED [INFO] hadoop-yarn-applications-distributedshell . SKIPPED [INFO] hadoop-yarn-applications-unmanaged-am-launcher SKIPPED [INFO] hadoop-yarn-site .. SKIPPED [INFO] hadoop-yarn-registry .. SKIPPED [INFO] hadoop-yarn-project ... SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 01:00 h [INFO] Finished at: 2015-03-03T11:33:29+00:00 [INFO] Final Memory: 61M/724M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on project hadoop-yarn-server-resourcemanager: ExecutionException: java.lang.RuntimeException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called? [ERROR] Command was /bin/sh -c cd /home/jenkins/jenkins-slave/workspace/Hadoop-Yarn-trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager /home/jenkins/tools/java/jdk1.7.0_55/jre/bin/java -Xmx4096m -XX:MaxPermSize=768m -XX:+HeapDumpOnOutOfMemoryError -jar /home/jenkins/jenkins-slave/workspace/Hadoop-Yarn-trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/target/surefire/surefirebooter7323524529220951737.jar /home/jenkins/jenkins-slave/workspace/Hadoop-Yarn-trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/target/surefire/surefire8319032661484443186tmp /home/jenkins/jenkins-slave/workspace/Hadoop-Yarn-trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/target/surefire/surefire_2452537876203083844248tmp [ERROR] - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn goals -rf :hadoop-yarn-server-resourcemanager Build step 'Execute shell' marked build as failure Archiving artifacts Recording test results Updating HDFS-7302 Updating YARN-3270 Updating MAPREDUCE-5657 Updating MAPREDUCE-6268 Updating HDFS-7871 Updating YARN-3265 Updating HADOOP-11605 Updating YARN-3281 Updating HDFS-7785 Updating MAPREDUCE-5583 Updating HADOOP-11602 Sending e-mails to: yarn-dev@hadoop.apache.org Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## 1 tests failed. REGRESSION: org.apache.hadoop.yarn.server.resourcemanager.security.TestRMDelegationTokens.testRMDTMasterKeyStateOnRollingMasterKey Error Message: null Stack Trace: java.lang.AssertionError: null at org.junit.Assert.fail(Assert.java:86) at org.junit.Assert.assertTrue(Assert.java:41) at org.junit.Assert.assertTrue(Assert.java:52) at org.apache.hadoop.yarn.server.resourcemanager.security.TestRMDelegationTokens$TestRMDelegationTokenSecretManager.checkCurrentKeyInStateStore(TestRMDelegationTokens.java:208) at
Re: Looking to a Hadoop 3 release
Thanks Andrew for bringing this up. +1 mostly looks fine but I'm thinking it's not now to cut branch-3. classpath isolation IMHO, classpath isolation is a good thing to do. We should pay down the technical dept ASAP. I'm willing to help. I'm thinking we can cut branch-3 and release 3.0 alpha after HADOOP-11656 is fixed. That is, I'd like to mark this issue as a blocker for 3.0. I wonder that even if we cut branch-3 now, trunk and branch-3 would be the same for a while. That seems useless. JDK8 As Steve suggested, JDK8 can be in both trunk and branch-2. +1 for moving to JDK8 ASAP. maintaining 2.x For user side, now there is little merit to upgrade to 3.x. More important thing is how long 2.x will be maintained. Therefore we should consider when to stop backporting new features to 2.x, and when to stop maintaining 2.x. I'd like to maintain 2.x as long as possible, at least one year after 3.x GA release. * Other issue What's the current status of HDFS symlink? If HADOOP-10019 requires some incompatible changes, I'd like to include in 3.x. Regards, Akira On 3/2/15 15:19, Andrew Wang wrote: Hi devs, It's been a year and a half since 2.x went GA, and I think we're about due for a 3.x release. Notably, there are two incompatible changes I'd like to call out, that will have a tremendous positive impact for our users. First, classpath isolation being done at HADOOP-11656, which has been a long-standing request from many downstreams and Hadoop users. Second, bumping the source and target JDK version to JDK8 (related to HADOOP-11090), which is important since JDK7 is EOL in April 2015 (two months from now). In the past, we've had issues with our dependencies discontinuing support for old JDKs, so this will future-proof us. Between the two, we'll also have quite an opportunity to clean up and upgrade our dependencies, another common user and developer request. I'd like to propose that we start rolling a series of monthly-ish series of 3.0 alpha releases ASAP, with myself volunteering to take on the RM and other cat herding responsibilities. There are already quite a few changes slated for 3.0 besides the above (for instance the shell script rewrite) so there's already value in a 3.0 alpha, and the more time we give downstreams to integrate, the better. This opens up discussion about inclusion of other changes, but I'm hoping to freeze incompatible changes after maybe two alphas, do a beta (with no further incompat changes allowed), and then finally a 3.x GA. For those keeping track, that means a 3.x GA in about four months. I would also like to stress though that this is not intended to be a big bang release. For instance, it would be great if we could maintain wire compatibility between 2.x and 3.x, so rolling upgrades work. Keeping branch-2 and branch-3 similar also makes backports easier, since we're likely maintaining 2.x for a while yet. Please let me know any comments / concerns related to the above. If people are friendly to the idea, I'd like to cut a branch-3 and start working on the first alpha. Best, Andrew
Re: The Activities of Apache Hadoop Community
Hi all, One year after the previous post, we collected and analyzed JIRA tickets again to investigate the activities of Apache Hadoop community in 2014. http://ajisakaa.blogspot.com/2015/02/the-activities-of-apache-hadoop.html As we expected in the previous post, the activities of Apache Hadoop community was continued to expand also in 2014. We hope it will be the same in 2015. Thanks, Akira On 2/13/14 11:20, Akira AJISAKA wrote: Hi all, We collected and analyzed JIRA tickets to investigate the activities of Apache Hadoop Community in 2013. http://ajisakaa.blogspot.com/2014/02/the-activities-of-apache-hadoop.html We counted the number of the organizations, the lines of code, and the number of the issues. As a result, we confirmed all of them are increasing and Hadoop community is getting more active. We appreciate continuous contributions of developers and we hope the activities will expand also in 2014. Thanks, Akira
Failed: YARN-3248 PreCommit Build #6820
Jira: https://issues.apache.org/jira/browse/YARN-3248
Build: https://builds.apache.org/job/PreCommit-YARN-Build/6820/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 3435 lines...]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12702135/apache-yarn-3248.2.patch
against trunk revision 4228de9.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 5 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6820//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6820//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6820//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
670e2e734b3b1c6f87589ee89e370af90698a46b logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-YARN-Build #6776
Archived 15 artifacts
Archive block size is 32768
Received 36 blocks and 7893771 bytes
Compression is 13.0%
Took 2.9 sec
Description set: MAPREDUCE-5583
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
All tests passed
Failed: YARN-3249 PreCommit Build #6817
Jira: https://issues.apache.org/jira/browse/YARN-3249
Build: https://builds.apache.org/job/PreCommit-YARN-Build/6817/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 3520 lines...]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12702071/YARN-3249.5.patch
against trunk revision 742f9d9.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 5 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6817//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6817//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6817//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
5be38379aa0ec387c8b0977b333a0dc1d271ad5e logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-YARN-Build #6776
Archived 20 artifacts
Archive block size is 32768
Received 35 blocks and 8290664 bytes
Compression is 12.2%
Took 2 sec
Description set: MAPREDUCE-6268
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
All tests passed
Failed: YARN-3248 PreCommit Build #6819
Jira: https://issues.apache.org/jira/browse/YARN-3248
Build: https://builds.apache.org/job/PreCommit-YARN-Build/6819/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 49 lines...]
Hunk #7 succeeded at 204 (offset 3 lines).
Hunk #8 succeeded at 228 (offset 3 lines).
Checking patch
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java...
error: while searching for:
class AppsBlock extends HtmlBlock {
final ConcurrentMapApplicationId, RMApp apps;
private final Configuration conf;
@Inject
AppsBlock(ResourceManager rm, ViewContext ctx, Configuration conf) {
super(ctx);
apps = rm.getRMContext().getRMApps();
this.conf = conf;
}
@Override public void render(Block html) {
error: patch failed:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java:46
error:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java:
patch does not apply
PATCH APPLICATION FAILED
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12702133/apache-yarn-3248.1.patch
against trunk revision 4228de9.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6819//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
7e7a66453eb892add8e69c331b35ff200a322c70 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Description set: MAPREDUCE-5583
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
No tests ran.
