Build failed in Jenkins: Hadoop-Common-trunk #868

2013-08-22 Thread Apache Jenkins Server 
See https://builds.apache.org/job/Hadoop-Common-trunk/868/changes

Changes:

[acmurthy] MAPREDUCE-5475. Ensure MRClientService verifies ACLs for users. 
Contributed by Jason Lowe.

[acmurthy] MAPREDUCE-5468. Fix MR AM recovery for map-only jobs. Contributed by 
Vinod K. V.

[acmurthy] YARN-1082. Addendum patch.

[jing9] HDFS-5045. Add more unit tests for retry cache to cover all AtMostOnce 
methods. Contributed by Jing Zhao.

[acmurthy] YARN-1082. Create base directories on HDFS after RM login to ensure 
RM recovery doesn't fail in secure mode. Contributed by Vinod K V.

[bikas] YARN-881. Priority#compareTo method seems to be wrong. (Jian He via 
bikas)

[vinodkv] MAPREDUCE-5466. Changed MR AM to not promote history files of 
intermediate AMs in case they are exiting because of errors and thus help 
history-server pick up the right history file for the last successful AM. 
Contributed by Jian He.

[kihwal] HDFS-4994. Audit log getContentSummary() calls. Contributed by Robert 
Parker.

[brandonli] HDFS-5069 Include hadoop-nfs and hadoop-hdfs-nfs into hadoop dist 
for NFS deployment. Contributed by Brandon Li

[llu] HADOOP-9784. Add a builder for HttpServer. (Junping Du via llu)

--
[...truncated 55973 lines...]
Adding reference: maven.project
Adding reference: maven.project.helper
Adding reference: maven.local.repository
[DEBUG] Initialize Maven Ant Tasks
parsing buildfile 
jar:file:/home/jenkins/.m2/repository/org/apache/maven/plugins/maven-antrun-plugin/1.6/maven-antrun-plugin-1.6.jar!/org/apache/maven/ant/tasks/antlib.xml
 with URI = 
jar:file:/home/jenkins/.m2/repository/org/apache/maven/plugins/maven-antrun-plugin/1.6/maven-antrun-plugin-1.6.jar!/org/apache/maven/ant/tasks/antlib.xml
 from a zip file
parsing buildfile 
jar:file:/home/jenkins/.m2/repository/org/apache/ant/ant/1.8.1/ant-1.8.1.jar!/org/apache/tools/ant/antlib.xml
 with URI = 
jar:file:/home/jenkins/.m2/repository/org/apache/ant/ant/1.8.1/ant-1.8.1.jar!/org/apache/tools/ant/antlib.xml
 from a zip file
Class org.apache.maven.ant.tasks.AttachArtifactTask loaded from parent loader 
(parentFirst)
 +Datatype attachartifact org.apache.maven.ant.tasks.AttachArtifactTask
Class org.apache.maven.ant.tasks.DependencyFilesetsTask loaded from parent 
loader (parentFirst)
 +Datatype dependencyfilesets org.apache.maven.ant.tasks.DependencyFilesetsTask
Setting project property: test.build.dir - 
https://builds.apache.org/job/Hadoop-Common-trunk/ws/trunk/hadoop-common-project/target/test-dir
Setting project property: test.exclude.pattern - _
Setting project property: hadoop.assemblies.version - 3.0.0-SNAPSHOT
Setting project property: test.exclude - _
Setting project property: distMgmtSnapshotsId - apache.snapshots.https
Setting project property: project.build.sourceEncoding - UTF-8
Setting project property: java.security.egd - file:///dev/urandom
Setting project property: distMgmtSnapshotsUrl - 
https://repository.apache.org/content/repositories/snapshots
Setting project property: distMgmtStagingUrl - 
https://repository.apache.org/service/local/staging/deploy/maven2
Setting project property: test.build.data - 
https://builds.apache.org/job/Hadoop-Common-trunk/ws/trunk/hadoop-common-project/target/test-dir
Setting project property: commons-daemon.version - 1.0.13
Setting project property: hadoop.common.build.dir - 
https://builds.apache.org/job/Hadoop-Common-trunk/ws/trunk/hadoop-common-project/../../hadoop-common-project/hadoop-common/target
Setting project property: testsThreadCount - 4
Setting project property: maven.test.redirectTestOutputToFile - true
Setting project property: jdiff.version - 1.0.9
Setting project property: build.platform - Linux-i386-32
Setting project property: distMgmtStagingName - Apache Release Distribution 
Repository
Setting project property: project.reporting.outputEncoding - UTF-8
Setting project property: protobuf.version - 2.5.0
Setting project property: failIfNoTests - false
Setting project property: protoc.path - ${env.HADOOP_PROTOC_PATH}
Setting project property: distMgmtStagingId - apache.staging.https
Setting project property: distMgmtSnapshotsName - Apache Development Snapshot 
Repository
Setting project property: ant.file - 
https://builds.apache.org/job/Hadoop-Common-trunk/ws/trunk/hadoop-common-project/pom.xml
[DEBUG] Setting properties with prefix: 
Setting project property: project.groupId - org.apache.hadoop
Setting project property: project.artifactId - hadoop-common-project
Setting project property: project.name - Apache Hadoop Common Project
Setting project property: project.description - Apache Hadoop Common Project
Setting project property: project.version - 3.0.0-SNAPSHOT
Setting project property: project.packaging - pom
Setting project property: project.build.directory - 
https://builds.apache.org/job/Hadoop-Common-trunk/ws/trunk/hadoop-common-project/target
Setting project property: project.build.outputDirectory -

Re: [VOTE] Release Apache Hadoop 2.0.6-alpha

2013-08-22 Thread Rob Parker 

+1 non-binding

verified signatures and checksum
built source
single node cluster tests

Rob

On 08/11/2013 12:46 AM, Konstantin Boudnik wrote:

All,

I have created a release candidate (rc0) for hadoop-2.0.6-alpha that I would
like to release.

This is a stabilization release that includes fixed for a couple a of issues
as outlined on the security list.

The RC is available at: http://people.apache.org/~cos/hadoop-2.0.6-alpha-rc0/
The RC tag in svn is here: 
http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.6-alpha-rc0

The maven artifacts are available via repository.apache.org.

Please try the release bits and vote; the vote will run for the usual 7 days.

Thanks for your voting
   Cos

[jira] [Created] (HADOOP-9897) Add method to get path start position without drive specifier in o.a.h.fs.Path

2013-08-22 Thread Binglin Chang (JIRA) 
Binglin Chang created HADOOP-9897:
-

 Summary: Add method to get path start position without drive 
specifier in o.a.h.fs.Path  
 Key: HADOOP-9897
 URL: https://issues.apache.org/jira/browse/HADOOP-9897
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Binglin Chang
Assignee: Binglin Chang
Priority: Trivial


There are a lot of code in Path to get start position after skipping drive 
specifier, like:

{code}
int start = hasWindowsDrive(uri.getPath()) ? 3 : 0;
{code}

First, using hard coded 3 or 0 is bad, second, the code always assumes path is 
with / prefix.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: [VOTE] Release Apache Hadoop 2.1.0-beta

2013-08-22 Thread Colin McCabe 
On Wed, Aug 21, 2013 at 3:49 PM, Stack st...@duboce.net wrote:
 On Wed, Aug 21, 2013 at 1:25 PM, Colin McCabe cmcc...@alumni.cmu.eduwrote:

 St.Ack wrote:

  + Once I figured where the logs were, found that JAVA_HOME was not being
  exported (don't need this in hadoop-2.0.5 for instance).  Adding an
  exported JAVA_HOME to my running shell which don't seem right but it took
  care of it (I gave up pretty quick on messing w/
  yarn.nodemanager.env-whitelist and yarn.nodemanager.admin-env -- I wasn't
  getting anywhere)

 I thought that we were always supposed to have JAVA_HOME set when
 running any of these commands.  At least, I do.  How else can the
 system disambiguate between different Java installs?  I need 2
 installs to test with JDK7.



 That is fair enough but I did not need to define this explicitly previously
 (for hadoop-2.0.5-alpha for instance) or the JAVA_HOME that was figured in
 start scripts was propagated and now is not (I have not dug in).



  + This did not seem to work for me:
  namehadoop.security.group.mapping/name
 
 valueorg.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback/va
  lue.

 We've seen this before.  I think your problem is that you have
 java.library.path set correctly (what System.loadLibrary checks), but
 your system library path does not include a necessary dependency of
 libhadoop.so-- most likely, libjvm.so.  Probably, we should fix
 NativeCodeLoader to actually make a function call in libhadoop.so
 before it declares everything OK.


 My expectation was that if native group lookup fails, as it does here, then
 the 'Fallback' would kick in and we'd do the Shell query.  This mechanism
 does not seem to be working.

I filed https://issues.apache.org/jira/browse/HADOOP-9895 to address this issue.

best,
Colin




 St.Ack

Re: [VOTE] Release Apache Hadoop 2.1.0-beta

2013-08-22 Thread Rob Parker 

+1 non-binding

Verified signatures and checksums one binary and source tar files
Built the source
Ran some tests on psuedo-distributed cluster.

Rob

On 08/15/2013 09:15 PM, Arun C Murthy wrote:

Folks,

I've created a release candidate (rc2) for hadoop-2.1.0-beta that I would like 
to get released - this fixes the bugs we saw since the last go-around (rc1).

The RC is available at: 
http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc2/
The RC tag in svn is here: 
http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.0-beta-rc2

The maven artifacts are available via repository.apache.org.

Please try the release and vote; the vote will run for the usual 7 days.

thanks,
Arun

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

RE: [VOTE] Release Apache Hadoop 2.1.0-beta

2013-08-22 Thread Bikas Saha 
+1 (non-binding)

Bikas

-Original Message-
From: Hitesh Shah [mailto:hit...@apache.org]
Sent: Wednesday, August 21, 2013 4:25 PM
To: yarn-...@hadoop.apache.org; Arun Murthy
Cc: common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org;
mapreduce-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.1.0-beta

+1 (binding)

Verified checksums, built from source and ran basic MR jobs on a
single-node cluster.

-- Hitesh

On Aug 15, 2013, at 2:15 PM, Arun C Murthy wrote:

 Folks,

 I've created a release candidate (rc2) for hadoop-2.1.0-beta that I
would like to get released - this fixes the bugs we saw since the last
go-around (rc1).

 The RC is available at:
 http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc2/
 The RC tag in svn is here:
 http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.0-beta-
 rc2

 The maven artifacts are available via repository.apache.org.

 Please try the release and vote; the vote will run for the usual 7 days.

 thanks,
 Arun

 --
 Arun C. Murthy
 Hortonworks Inc.
 http://hortonworks.com/



 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or
 entity to which it is addressed and may contain information that is
 confidential, privileged and exempt from disclosure under applicable
 law. If the reader of this message is not the intended recipient, you
 are hereby notified that any printing, copying, dissemination,
 distribution, disclosure or forwarding of this communication is
 strictly prohibited. If you have received this communication in error,
 please contact the sender immediately and delete it from your system.
Thank You.

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: [VOTE] Release Apache Hadoop 2.1.0-beta

2013-08-22 Thread Steve Loughran 
+1, binding

Review process

# symlink /usr/local/bin/protoc to the homebrew installed 2.5.0 version

# delete all 2.1.0-beta artifacts in the mvn repo:

  find ~/.m2 -name 2.1.0-beta -print | xargs rm -rf

# checkout hbase apache/branch-0.95 (commit # b58d596 )

# switch to ASF repo (arun's private repo is in the POM, with JARs with the
same sha1 sum, I'm just being rigorous)
repository
  idASF Staging/id
  urlhttps://repository.apache.org/content/groups/staging//url
/repository

# clean build of hbase tar against the beta artifacts

mvn clean install assembly:single -DskipTests -Dmaven.javadoc.skip=true
-Dhadoop.profile=2.0 -Dhadoop-two.version=2.1.0-beta

# Observe DL taking place

]
[INFO] --- maven-assembly-plugin:2.4:single (default-cli) @ hbase ---
[INFO] Assemblies have been skipped per configuration of the skipAssembly
parameter.
[INFO]

[INFO]

[INFO] Building HBase - Common 0.95.3-SNAPSHOT
[INFO]

Downloading:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-annotations/2.1.0-beta/hadoop-annotations-2.1.0-beta.pom
Downloaded:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-annotations/2.1.0-beta/hadoop-annotations-2.1.0-beta.pom(2
KB at 3.3 KB/sec)
Downloading:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-project/2.1.0-beta/hadoop-project-2.1.0-beta.pom
Downloaded:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-project/2.1.0-beta/hadoop-project-2.1.0-beta.pom(34
KB at 383.8 KB/sec)
Downloading:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-main/2.1.0-beta/hadoop-main-2.1.0-beta.pom
Downloaded:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-main/2.1.0-beta/hadoop-main-2.1.0-beta.pom(17
KB at 184.0 KB/sec)
Downloading:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-common/2.1.0-beta/hadoop-common-2.1.0-beta.pom
Downloaded:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-common/2.1.0-beta/hadoop-common-2.1.0-beta.pom(26
KB at 293.3 KB/sec)
Downloading:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-project-dist/2.1.0-beta/hadoop-project-dist-2.1.0-beta.pom
Downloaded:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-project-dist/2.1.0-beta/hadoop-project-dist-2.1.0-beta.pom(17
KB at 176.2 KB/sec)
Downloading:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-auth/2.1.0-beta/hadoop-auth-2.1.0-beta.pom
Downloaded:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-auth/2.1.0-beta/hadoop-auth-2.1.0-beta.pom(7
KB at 79.0 KB/sec)
Downloading:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-annotations/2.1.0-beta/hadoop-annotations-2.1.0-beta.jar
Downloading:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-common/2.1.0-beta/hadoop-common-2.1.0-beta.jar
Downloading:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-auth/2.1.0-beta/hadoop-auth-2.1.0-beta.jar
Downloaded:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-annotations/2.1.0-beta/hadoop-annotations-2.1.0-beta.jar(17
KB at 146.1 KB/sec)
Downloaded:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-auth/2.1.0-beta/hadoop-auth-2.1.0-beta.jar(47
KB at 241.7 KB/sec)
Downloaded:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-common/2.1.0-beta/hadoop-common-2.1.0-beta.jar(2657
KB at 2260.9 KB/sec)
[INFO]


# get md5 sum of hadoop-common-2.1.0-beta artifact in
https://repository.apache.org/content/groups/staging/ :
0166f5c94d3699b3a37efc16ebb1ceea3acb3b53

# verify version of artifact in local m2 repo
$ sha1sum
~/.m2/repository/org/apache/hadoop/hadoop-common/2.1.0-beta/hadoop-common-2.1.0-beta.jar
0166f5c94d3699b3a37efc16ebb1ceea3acb3b53

# in hbase/hbase-assembly/target , gunzip then untar the
hbase-0.95.3-SNAPSHOT-bin.tar file

# Patch the Hoya POM to use 2.1.0-beta instead of a local 2.1.1-SNAPSHOT

# run some of the hbase cluster deploy  flexing tests


 mvn clean test  -Pstaging

 (all tests pass after 20 min)

== functional tests =

# build and the Hoya JAR with classpath

mvn package -Pstaging

# D/L the binary .tar.gz file, and scp to an ubuntu VM with the hadoop conf
properties for net-accessible HDFS  YARN services, no memory limits on
containers

https://github.com/hortonworks/hoya/tree/master/src/test/configs/ubuntu
https://github.com/hortonworks/hoya/blob/master/src/test/configs/ubuntu/core-site.xml
https://github.com/hortonworks/hoya/blob/master/src/test/configs/ubuntu/yarn-site.xml

# stop the running hadoop-2.1.1-snapshot

[jira] [Created] (HADOOP-9898) Set SO_KEEPALIVE on all our sockets

2013-08-22 Thread Todd Lipcon (JIRA) 
Todd Lipcon created HADOOP-9898:
---

 Summary: Set SO_KEEPALIVE on all our sockets
 Key: HADOOP-9898
 URL: https://issues.apache.org/jira/browse/HADOOP-9898
 Project: Hadoop Common
  Issue Type: Bug
  Components: ipc, net
Affects Versions: 3.0.0
Reporter: Todd Lipcon
Priority: Minor


We recently saw an issue where network issues between slaves and the NN caused 
ESTABLISHED TCP connections to pile up and leak on the NN side. It looks like 
the RST packets were getting dropped, which meant that the client thought the 
connections were closed, while they hung open forever on the server.

Setting the SO_KEEPALIVE option on our sockets would prevent this kind of leak 
from going unchecked.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Hadoop is in Fedora

2013-08-22 Thread Mattmann, Chris A (398J) 
That is awesome news!

++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattm...@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++






-Original Message-
From: Matthew Farrellee m...@redhat.com
Reply-To: common-dev@hadoop.apache.org common-dev@hadoop.apache.org
Date: Tuesday, August 20, 2013 1:39 PM
To: Fedora Big Data SIG bigd...@lists.fedoraproject.org
Cc: Robert Rati rr...@redhat.com, common-dev@hadoop.apache.org
common-dev@hadoop.apache.org
Subject: Re: Hadoop is in Fedora

On 08/20/2013 04:35 PM, Robert Rati wrote:
 The hadoop package has passed review and has been built for Fedora 20.

 Rob

Well done!

[jira] [Created] (HADOOP-9899) Remove the debug message added by HADOOP-8855

2013-08-22 Thread Tsz Wo (Nicholas), SZE (JIRA) 
Tsz Wo (Nicholas), SZE created HADOOP-9899:
--

 Summary: Remove the debug message added by HADOOP-8855
 Key: HADOOP-9899
 URL: https://issues.apache.org/jira/browse/HADOOP-9899
 Project: Hadoop Common
  Issue Type: Bug
  Components: security
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
Priority: Minor


HADOOP-8855 added a debug message which was printed to System.out.
{code}
//KerberosAuthenticator.java
  private void sendToken(byte[] outToken) throws IOException, 
AuthenticationException {
new Exception(sendToken).printStackTrace(System.out);
...
}
{code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira