Build failed in Jenkins: Hadoop-Common-trunk #1094

[Apache Jenkins Server]

See 

Changes:

[kasha] YARN-1784. TestContainerAllocation assumes CapacityScheduler. (Robert 
Kanter via kasha)

[jing9] HADOOP-10475. ConcurrentModificationException in 
AbstractDelegationTokenSelector.selectToken(). Contributed by Jing Zhao.

[wheat9] HADOOP-10474. Move o.a.h.record to hadoop-streaming. Contributed by 
Haohui Mai.

[vinodkv] YARN-1908. Fixed DistributedShell to not fail in secure clusters. 
Contributed by Vinod Kumar Vavilapalli and Jian He.

[wheat9] HDFS-6169. Move the address in WebImageViewer. Contributed by Akira 
Ajisaka.

[kasha] YARN-1757. NM Recovery. Auxiliary service support. (Jason Lowe via 
kasha)

--
[...truncated 60591 lines...]
Adding reference: maven.local.repository
[DEBUG] Initialize Maven Ant Tasks
parsing buildfile 
jar:file:/home/jenkins/.m2/repository/org/apache/maven/plugins/maven-antrun-plugin/1.7/maven-antrun-plugin-1.7.jar!/org/apache/maven/ant/tasks/antlib.xml
 with URI = 
jar:file:/home/jenkins/.m2/repository/org/apache/maven/plugins/maven-antrun-plugin/1.7/maven-antrun-plugin-1.7.jar!/org/apache/maven/ant/tasks/antlib.xml
 from a zip file
parsing buildfile 
jar:file:/home/jenkins/.m2/repository/org/apache/ant/ant/1.8.2/ant-1.8.2.jar!/org/apache/tools/ant/antlib.xml
 with URI = 
jar:file:/home/jenkins/.m2/repository/org/apache/ant/ant/1.8.2/ant-1.8.2.jar!/org/apache/tools/ant/antlib.xml
 from a zip file
Class org.apache.maven.ant.tasks.AttachArtifactTask loaded from parent loader 
(parentFirst)
 +Datatype attachartifact org.apache.maven.ant.tasks.AttachArtifactTask
Class org.apache.maven.ant.tasks.DependencyFilesetsTask loaded from parent 
loader (parentFirst)
 +Datatype dependencyfilesets org.apache.maven.ant.tasks.DependencyFilesetsTask
Setting project property: test.build.dir -> 

Setting project property: test.exclude.pattern -> _
Setting project property: hadoop.assemblies.version -> 3.0.0-SNAPSHOT
Setting project property: test.exclude -> _
Setting project property: distMgmtSnapshotsId -> apache.snapshots.https
Setting project property: project.build.sourceEncoding -> UTF-8
Setting project property: java.security.egd -> file:///dev/urandom
Setting project property: distMgmtSnapshotsUrl -> 
https://repository.apache.org/content/repositories/snapshots
Setting project property: distMgmtStagingUrl -> 
https://repository.apache.org/service/local/staging/deploy/maven2
Setting project property: avro.version -> 1.7.4
Setting project property: test.build.data -> 

Setting project property: commons-daemon.version -> 1.0.13
Setting project property: hadoop.common.build.dir -> 

Setting project property: testsThreadCount -> 4
Setting project property: maven.test.redirectTestOutputToFile -> true
Setting project property: jdiff.version -> 1.0.9
Setting project property: build.platform -> Linux-i386-32
Setting project property: project.reporting.outputEncoding -> UTF-8
Setting project property: distMgmtStagingName -> Apache Release Distribution 
Repository
Setting project property: protobuf.version -> 2.5.0
Setting project property: failIfNoTests -> false
Setting project property: protoc.path -> ${env.HADOOP_PROTOC_PATH}
Setting project property: jersey.version -> 1.9
Setting project property: distMgmtStagingId -> apache.staging.https
Setting project property: distMgmtSnapshotsName -> Apache Development Snapshot 
Repository
Setting project property: ant.file -> 

[DEBUG] Setting properties with prefix: 
Setting project property: project.groupId -> org.apache.hadoop
Setting project property: project.artifactId -> hadoop-common-project
Setting project property: project.name -> Apache Hadoop Common Project
Setting project property: project.description -> Apache Hadoop Common Project
Setting project property: project.version -> 3.0.0-SNAPSHOT
Setting project property: project.packaging -> pom
Setting project property: project.build.directory -> 

Setting project property: project.build.outputDirectory -> 

Setting project property: project.build.testOutputDirectory -> 

Setting project property: project.build.sourceDirectory -> 

Setting project property: project.build.testSourceDire

[jira] [Resolved] (HADOOP-9096) Improve performance of Windows install scripts

[Ivan Mitic (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ivan Mitic resolved HADOOP-9096.


Resolution: Won't Fix

Install scripts are no longer distributed with hadoop.

> Improve performance of Windows install scripts
> --
>
> Key: HADOOP-9096
> URL: https://issues.apache.org/jira/browse/HADOOP-9096
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 1-win
>Reporter: Ivan Mitic
>
> Things to improve for better performance:
> The whole install is taking 4-5 mins on a single box because of.
>  1) Inclusion of src & other temp folders (IVY etc..) in the winpkg so it is 
> taking longer time to decompress the zip folder
>  2) Nested zip files in the winpkg. If we remove the nested zips then after 
> manual unpack everything will be a xcopy install and reduces install time.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Plans of moving towards JDK7 in trunk

[Andrew Purtell]

A Java 8 runtime would also offer transparent performance improvements like
a reimplementation of ConcurrentSkipListMap, C2 support for AES cipher
acceleration with native CPU instructions, perf improvements for going from
String to byte[] or vice versa, and IIRC after 8u20 monitor lock elision
using restricted transactional memory with hardware support (if available).
Getting away from fully transparent changes but tractable to deal with
using reflection, removal of the permanent generation, support for AEAD
cipher modes like AES-GCM, stronger cipher and key exchange algorithms, TLS
1.2, support for some krb 5 features not handled previously.



On Tue, Apr 8, 2014 at 7:44 PM, Raymie Stata  wrote:

> > It might make sense to try to enumerate the benefits of switching to
> > Java7 APIs and dependencies.
>
>   - Java7 introduced a huge number of language, byte-code, API, and
> tooling enhancements!  Just to name a few: try-with-resources, newer
> and stronger encyrption methods, more scalable concurrency primitives.
>  See http://www.slideshare.net/boulderjug/55-things-in-java-7
>
>   - We can't update current dependencies, and we can't add cool new ones.
>
>   - Putting language/APIs aside, don't forget that a huge amount of effort
> goes into qualifying for Java6 (at least, I hope the folks claiming to
> support Java6 are putting in such an effort :-).  Wouldn't Hadoop
> users/customers be better served if qualification effort went into
> Java7/8 versus Java6/7?
>
> Getting to Java7 as a development env (and Java8 as a runtime env)
> seems like a no-brainer.  Question is: How?
>
> On Tue, Apr 8, 2014 at 10:21 AM, Sandy Ryza 
> wrote:
> > It might make sense to try to enumerate the benefits of switching to
> Java7
> > APIs and dependencies.  IMO, the ones listed so far on this thread don't
> > make a compelling enough case to drop Java6 in branch-2 on any time
> frame,
> > even if this means supporting Java6 through 2015.  For example, the
> change
> > in RawLocalFileSystem semantics might be an incompatible change for
> > branch-2 any way.
> >
> >
> > On Tue, Apr 8, 2014 at 10:05 AM, Karthik Kambatla  >wrote:
> >
> >> +1 to NOT breaking compatibility in branch-2.
> >>
> >> I think it is reasonable to require JDK7 for trunk, if we limit use of
> >> JDK7-only API to security fixes etc. If we make other optimizations
> (like
> >> IO), it would be a pain to backport things to branch-2. I guess this all
> >> depends on when we see ourselves shipping Hadoop-3. Any ideas on that?
> >>
> >>
> >> On Tue, Apr 8, 2014 at 9:19 AM, Eli Collins  wrote:
> >>
> >> > On Tue, Apr 8, 2014 at 2:00 AM, Ottenheimer, Davi
> >> >  wrote:
> >> > >> From: Eli Collins [mailto:e...@cloudera.com]
> >> > >> Sent: Monday, April 07, 2014 11:54 AM
> >> > >>
> >> > >>
> >> > >> IMO we should not drop support for Java 6 in a minor update of a
> >> stable
> >> > >> release (v2).  I don't think the larger Hadoop user base would
> find it
> >> > >> acceptable that upgrading to a minor update caused their systems to
> >> stop
> >> > >> working because they didn't upgrade Java. There are people still
> >> getting
> >> > >> support for Java 6. ...
> >> > >>
> >> > >> Thanks,
> >> > >> Eli
> >> > >
> >> > > Hi Eli,
> >> > >
> >> > > Technically you are correct those with extended support get critical
> >> > security fixes for 6 until the end of 2016. I am curious whether many
> of
> >> > those are in the Hadoop user base. Do you know? My guess is the vast
> >> > majority are within Oracle's official public end of life, which was
> over
> >> 12
> >> > months ago. Even Premier support ended Dec 2013:
> >> > >
> >> > > http://www.oracle.com/technetwork/java/eol-135779.html
> >> > >
> >> > > The end of Java 6 support carries much risk. It has to be
> considered in
> >> > terms of serious security vulnerabilities such as CVE-2013-2465 with
> CVSS
> >> > score 10.0.
> >> > >
> >> > > http://www.cvedetails.com/cve/CVE-2013-2465/
> >> > >
> >> > > Since you mentioned "caused systems to stop" as an example of what
> >> would
> >> > be a concern to Hadoop users, please note the CVE-2013-2465
> availability
> >> > impact:
> >> > >
> >> > > "Complete (There is a total shutdown of the affected resource. The
> >> > attacker can render the resource completely unavailable.)"
> >> > >
> >> > > This vulnerability was patched in Java 6 Update 51, but post end of
> >> > life. Apple pushed out the update specifically because of this
> >> > vulnerability (http://support.apple.com/kb/HT5717) as did some other
> >> > vendors privately, but for the majority of people using Java 6 means
> they
> >> > have a ticking time bomb.
> >> > >
> >> > > Allowing it to stay should be considered in terms of accepting the
> >> whole
> >> > risk posture.
> >> > >
> >> >
> >> > There are some who get extended support, but I suspect many just have
> >> > a if-it's-not-broke mentality when it comes to production deployments.
> >> > The current code supports both java6 and java7 and so all

Re: Plans of moving towards JDK7 in trunk

[Eli Collins]

I think this thread isn't so much about whether java7, 8 etc features
are valuable, they are useful of course, and we'll want to adopt them,
it's a question of how we adopt them and in which releases.

For the sake of this discussion we should separate the runtime from
the programming APIs. Users are already migrating to the java7 runtime
for most of the reasons listed below (support, performance, bugs,
etc), and the various distributions cert their Hadoop 2 based
distributions on java7.  This gives users many of the benefits of
java7, without forcing users off java6. Ie Hadoop does not need to
switch to the java7 programming APIs to make sure everyone has a
supported runtime.

The question here is really about when Hadoop, and the Hadoop
ecosystem (since adjacent projects often end up in the same classpath)
start using the java7 programming APIs and therefore break
compatibility with java6 runtimes. I think our java6 runtime users
would consider dropping support for their java runtime in an update of
a major release to be an incompatible change (the binaries stop
working on their current jvm). That may be worth it if we can
articulate sufficient value to offset the cost (they have to upgrade
their environment, might make rolling upgrades stop working, etc), but
I've not yet heard an argument that articulates the value relative to
the cost.  Eg upgrading to the java7 APIs allows us to pull in
dependencies with new major versions, but only if those dependencies
don't break compatibility (which is likely given that our classpaths
aren't so isolated), and, realistically, only if the entire Hadoop
stack moves to java7 as well (eg we have to recompile HBase to
generate v1.7 binaries even if they stick on API v1.6). I'm not aware
of a feature, bug etc that really motivates this.

An alternate approach is to keep the current stable release series
(v2.x) as is, and start using new APIs in trunk (for v3). This will be
a major upgrade for Hadoop and therefore an incompatible change like
this is to be expected (it would be great if this came with additional
changes to better isolate classpaths and dependencies from each
other). It allows us to continue to support multiple types of users
with different branches, vs forcing all users onto a new version. It
of course means that 2.x users will not get the benefits of the new
API, but its unclear what those benefits are given they can already
get the benefits of adopting the newer java runtimes today.

Thanks,
Eli


On Wed, Apr 9, 2014 at 9:38 AM, Andrew Purtell  wrote:
> A Java 8 runtime would also offer transparent performance improvements like
> a reimplementation of ConcurrentSkipListMap, C2 support for AES cipher
> acceleration with native CPU instructions, perf improvements for going from
> String to byte[] or vice versa, and IIRC after 8u20 monitor lock elision
> using restricted transactional memory with hardware support (if available).
> Getting away from fully transparent changes but tractable to deal with
> using reflection, removal of the permanent generation, support for AEAD
> cipher modes like AES-GCM, stronger cipher and key exchange algorithms, TLS
> 1.2, support for some krb 5 features not handled previously.
>
>
>
> On Tue, Apr 8, 2014 at 7:44 PM, Raymie Stata  wrote:
>
>> > It might make sense to try to enumerate the benefits of switching to
>> > Java7 APIs and dependencies.
>>
>>   - Java7 introduced a huge number of language, byte-code, API, and
>> tooling enhancements!  Just to name a few: try-with-resources, newer
>> and stronger encyrption methods, more scalable concurrency primitives.
>>  See http://www.slideshare.net/boulderjug/55-things-in-java-7
>>
>>   - We can't update current dependencies, and we can't add cool new ones.
>>
>>   - Putting language/APIs aside, don't forget that a huge amount of effort
>> goes into qualifying for Java6 (at least, I hope the folks claiming to
>> support Java6 are putting in such an effort :-).  Wouldn't Hadoop
>> users/customers be better served if qualification effort went into
>> Java7/8 versus Java6/7?
>>
>> Getting to Java7 as a development env (and Java8 as a runtime env)
>> seems like a no-brainer.  Question is: How?
>>
>> On Tue, Apr 8, 2014 at 10:21 AM, Sandy Ryza 
>> wrote:
>> > It might make sense to try to enumerate the benefits of switching to
>> Java7
>> > APIs and dependencies.  IMO, the ones listed so far on this thread don't
>> > make a compelling enough case to drop Java6 in branch-2 on any time
>> frame,
>> > even if this means supporting Java6 through 2015.  For example, the
>> change
>> > in RawLocalFileSystem semantics might be an incompatible change for
>> > branch-2 any way.
>> >
>> >
>> > On Tue, Apr 8, 2014 at 10:05 AM, Karthik Kambatla > >wrote:
>> >
>> >> +1 to NOT breaking compatibility in branch-2.
>> >>
>> >> I think it is reasonable to require JDK7 for trunk, if we limit use of
>> >> JDK7-only API to security fixes etc. If we make other optimizations
>> (like
>> >> IO), it would 

[jira] [Created] (HADOOP-10487) Racy code in UserGroupInformation#ensureInitialized()

[Haohui Mai (JIRA)]

Haohui Mai created HADOOP-10487:
---

 Summary: Racy code in UserGroupInformation#ensureInitialized()
 Key: HADOOP-10487
 URL: https://issues.apache.org/jira/browse/HADOOP-10487
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Haohui Mai
Assignee: Haohui Mai


UserGroupInformation#ensureInitialized() uses the double-check-locking pattern 
to reduce the synchronization cost:

{code}
  private static void ensureInitialized() {
if (conf == null) {
  synchronized(UserGroupInformation.class) {
if (conf == null) { // someone might have beat us
  initialize(new Configuration(), false);
}
  }
}
  }
{code}

As [~tlipcon] pointed out in the original jira (HADOOP-9748). This pattern is 
incorrect. Please see more details in 
http://en.wikipedia.org/wiki/Double-checked_locking and 
http://www.cs.umd.edu/~pugh/java/memoryModel/DoubleCheckedLocking.html

This jira proposes to use the static class holder pattern to do it correctly.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Plans of moving towards JDK7 in trunk

[Vinayakumar B]

+1 for keeping jdk 6 suppprt in branch-2 and start using jdk 7 in trunk.

I agree that this approach makes patch generation difficult for branch-2
and trunk.

Also the actual benefit and real issues after start using jdk7 will be
known only if atleast one of the release is out in trunk version.

Regards,
Vinay
I think this thread isn't so much about whether java7, 8 etc features
are valuable, they are useful of course, and we'll want to adopt them,
it's a question of how we adopt them and in which releases.

For the sake of this discussion we should separate the runtime from
the programming APIs. Users are already migrating to the java7 runtime
for most of the reasons listed below (support, performance, bugs,
etc), and the various distributions cert their Hadoop 2 based
distributions on java7.  This gives users many of the benefits of
java7, without forcing users off java6. Ie Hadoop does not need to
switch to the java7 programming APIs to make sure everyone has a
supported runtime.

The question here is really about when Hadoop, and the Hadoop
ecosystem (since adjacent projects often end up in the same classpath)
start using the java7 programming APIs and therefore break
compatibility with java6 runtimes. I think our java6 runtime users
would consider dropping support for their java runtime in an update of
a major release to be an incompatible change (the binaries stop
working on their current jvm). That may be worth it if we can
articulate sufficient value to offset the cost (they have to upgrade
their environment, might make rolling upgrades stop working, etc), but
I've not yet heard an argument that articulates the value relative to
the cost.  Eg upgrading to the java7 APIs allows us to pull in
dependencies with new major versions, but only if those dependencies
don't break compatibility (which is likely given that our classpaths
aren't so isolated), and, realistically, only if the entire Hadoop
stack moves to java7 as well (eg we have to recompile HBase to
generate v1.7 binaries even if they stick on API v1.6). I'm not aware
of a feature, bug etc that really motivates this.

An alternate approach is to keep the current stable release series
(v2.x) as is, and start using new APIs in trunk (for v3). This will be
a major upgrade for Hadoop and therefore an incompatible change like
this is to be expected (it would be great if this came with additional
changes to better isolate classpaths and dependencies from each
other). It allows us to continue to support multiple types of users
with different branches, vs forcing all users onto a new version. It
of course means that 2.x users will not get the benefits of the new
API, but its unclear what those benefits are given they can already
get the benefits of adopting the newer java runtimes today.

Thanks,
Eli


On Wed, Apr 9, 2014 at 9:38 AM, Andrew Purtell  wrote:
> A Java 8 runtime would also offer transparent performance improvements
like
> a reimplementation of ConcurrentSkipListMap, C2 support for AES cipher
> acceleration with native CPU instructions, perf improvements for going
from
> String to byte[] or vice versa, and IIRC after 8u20 monitor lock elision
> using restricted transactional memory with hardware support (if
available).
> Getting away from fully transparent changes but tractable to deal with
> using reflection, removal of the permanent generation, support for AEAD
> cipher modes like AES-GCM, stronger cipher and key exchange algorithms,
TLS
> 1.2, support for some krb 5 features not handled previously.
>
>
>
> On Tue, Apr 8, 2014 at 7:44 PM, Raymie Stata  wrote:
>
>> > It might make sense to try to enumerate the benefits of switching to
>> > Java7 APIs and dependencies.
>>
>>   - Java7 introduced a huge number of language, byte-code, API, and
>> tooling enhancements!  Just to name a few: try-with-resources, newer
>> and stronger encyrption methods, more scalable concurrency primitives.
>>  See http://www.slideshare.net/boulderjug/55-things-in-java-7
>>
>>   - We can't update current dependencies, and we can't add cool new ones.
>>
>>   - Putting language/APIs aside, don't forget that a huge amount of
effort
>> goes into qualifying for Java6 (at least, I hope the folks claiming to
>> support Java6 are putting in such an effort :-).  Wouldn't Hadoop
>> users/customers be better served if qualification effort went into
>> Java7/8 versus Java6/7?
>>
>> Getting to Java7 as a development env (and Java8 as a runtime env)
>> seems like a no-brainer.  Question is: How?
>>
>> On Tue, Apr 8, 2014 at 10:21 AM, Sandy Ryza 
>> wrote:
>> > It might make sense to try to enumerate the benefits of switching to
>> Java7
>> > APIs and dependencies.  IMO, the ones listed so far on this thread
don't
>> > make a compelling enough case to drop Java6 in branch-2 on any time
>> frame,
>> > even if this means supporting Java6 through 2015.  For example, the
>> change
>> > in RawLocalFileSystem semantics might be an incompatible change for
>> > branch-2 any way.
>> >
>

[jira] [Created] (HADOOP-10488) TestKeyProviderFactory fails randomly

[Alejandro Abdelnur (JIRA)]

Alejandro Abdelnur created HADOOP-10488:
---

 Summary: TestKeyProviderFactory fails randomly
 Key: HADOOP-10488
 URL: https://issues.apache.org/jira/browse/HADOOP-10488
 Project: Hadoop Common
  Issue Type: Bug
  Components: test
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur


This test is fail randomly depending on the order of execution of the test 
methods, the reason is that the keystore used by the different testmethods is 
the same. We should either delete it before/after each test, or we should use a 
diff diff or each run.



--
This message was sent by Atlassian JIRA
(v6.2#6252)