[jira] [Created] (HADOOP-8772) RawLocalFileStatus shells out for permission info

[Daryn Sharp (JIRA)]

Daryn Sharp created HADOOP-8772:
---

 Summary: RawLocalFileStatus shells out for permission info
 Key: HADOOP-8772
 URL: https://issues.apache.org/jira/browse/HADOOP-8772
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 0.23.3, 3.0.0, 2.2.0-alpha
Reporter: Daryn Sharp
Priority: Critical


{{RawLocalFileStatus}} shells out to run "ls" to get permissions info.  This 
very inefficient.

More importantly, mixing mulithreading and forking is risky.  Some version of 
glibc in RHEL will deadlock things such as "__GI__IO_list_lock" and 
"malloc_atfork".  All this unnecessary shelling out to access the local 
filesystem greatly increases the risk of deadlock.

Namely, the NM's user localizer is seen to jam more frequently than the TT & NM.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-8771) Correct CHANGES.txt file

[Eli Collins (JIRA)]

Eli Collins created HADOOP-8771:
---

 Summary: Correct CHANGES.txt file
 Key: HADOOP-8771
 URL: https://issues.apache.org/jira/browse/HADOOP-8771
 Project: Hadoop Common
  Issue Type: Task
Affects Versions: 2.0.0-alpha
Reporter: Eli Collins
Priority: Blocker


The trunk CHANGES.txt files have a bunch of stuff listed under trunk that's 
actually already in branch-2, let's move that down to the branch-2 section. 
Also, the branch-2 files need to be updated (currently says "Release 
2.0.1-alpha - UNRELEASED" even though 2.0.1 was already released).


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-8769) Tests failures on the ARM hosts

[Eli Collins (JIRA)]

Eli Collins created HADOOP-8769:
---

 Summary: Tests failures on the ARM hosts 
 Key: HADOOP-8769
 URL: https://issues.apache.org/jira/browse/HADOOP-8769
 Project: Hadoop Common
  Issue Type: Test
  Components: build
Affects Versions: 2.0.0-alpha
Reporter: Eli Collins


I created a [jenkins job|https://builds.apache.org/job/Hadoop-trunk-ARM] that 
runs on the ARM machines. The local build is now working and running tests 
(thanks Gavin!), however there are 40 test failures, looks like most are due to 
host configuration issues. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Branch 2 release names

[Andrew Purtell]

If it's all the same to you, I'd prefer you leave the branch, or at least a
tag, and just ignore it. We're pretty far away from branch-2.1.0 following
branch-2 but started from that point.

On Wed, Sep 5, 2012 at 8:52 AM, Eli Collins  wrote:

> On Tue, Sep 4, 2012 at 11:55 AM, Owen O'Malley  wrote:
> > While cleaning up the subversion branches, I thought more about the
> > branch 2 release names. I'm concerned if we backtrack and reuse
> > release numbers it will be extremely confusing to users. It also
> > creates problems for tools like Maven that parse version numbers and
> > expect a left to right release numbering scheme (eg. 2.1.1-alpha >
> > 2.1.0). It also seems better to keep on the 2.0.x minor release until
> > after we get a GA release off of the 2.0 branch.
> >
> > Therefore, I'd like to propose:
> > 1. rename branch-2.0.1-alpha -> branch-2.0
> > 2. delete branch-2.1.0-alpha
> > 3. stabilizing goes into branch-2.0 until it gets to GA
> > 4. features go into branch-2 and will be branched into branch-2.1 later
> > 5. The release tags can have the alpha/beta tags on them.
> >
> > Thoughts?
>

Re: Branch 2 release names

[Eli Collins]

On Tue, Sep 4, 2012 at 11:55 AM, Owen O'Malley  wrote:
> While cleaning up the subversion branches, I thought more about the
> branch 2 release names. I'm concerned if we backtrack and reuse
> release numbers it will be extremely confusing to users. It also
> creates problems for tools like Maven that parse version numbers and
> expect a left to right release numbering scheme (eg. 2.1.1-alpha >
> 2.1.0). It also seems better to keep on the 2.0.x minor release until
> after we get a GA release off of the 2.0 branch.
>
> Therefore, I'd like to propose:
> 1. rename branch-2.0.1-alpha -> branch-2.0
> 2. delete branch-2.1.0-alpha
> 3. stabilizing goes into branch-2.0 until it gets to GA
> 4. features go into branch-2 and will be branched into branch-2.1 later
> 5. The release tags can have the alpha/beta tags on them.
>
> Thoughts?
>

branch-2.0.1-alpha is pretty far behind branch-2 at this point, both
in terms of features merged to branch-2 (eg no auto failover or hsync)
and bug fixes (iiuc it is just 2.0 plus a couple changes). From my
hdfs pov the branch doesn't seem worth maintaining.  I'd tweak this as
follows:

1. delete branch-2.1.0-alpha
2. rename branch-2 -> branch-2.0 some time after 0.23.3 is released
3. stabilizing goes into branch-2.0 until it gets to GA
4. features go into branch-2 and will be branched into branch-2.1 later
5. The release tags can have the alpha/beta tags on them.

On the hdfs side most trunk work is for branch-2 so we're already
merging trunk -> branch-2, so delaying the third merge would help, and
we're using feature branches for the big stuff (HDFS-3077) so they're
being isolated that way.

Thanks,
Eli

Re: Branch 2 release names

[Robert Evans]

I must have misread it.  Thanks for clarifying.

--Bobby

From: Vinod Kumar Vavilapalli 
mailto:vino...@hortonworks.com>>
Reply-To: "common-dev@hadoop.apache.org" 
mailto:common-dev@hadoop.apache.org>>
Date: Tuesday, September 4, 2012 7:29 PM
To: "common-dev@hadoop.apache.org" 
mailto:common-dev@hadoop.apache.org>>
Subject: Re: Branch 2 release names

May be you misread the proposal. This is only about nuking 2.1.0-alpha and wait 
for 0.23.3 to be stabilized and released. Once that happens, we can create a 
branch-2.1 off branch-2.

Does that sound okay?

Thanks,
+Vinod Kumar Vavilapalli
Hortonworks Inc.
http://hortonworks.com/

On Sep 4, 2012, at 3:05 PM, Robert Evans wrote:

I am fine with that too, but it is going to be a fairly large amount of
work to pull in all of the bug fixes into 2.0 that have gone into 0.23.
There was already a lot of discussion about just rebasing 2.1 instead of
trying to merge everything back into it and 2.1 is a lot further along
then 2.0 is.  Just something to be aware of.

--Bobby Evans

[jira] [Created] (HADOOP-8767) secondary namenode on slave machines

[giovanni delussu (JIRA)]

giovanni delussu created HADOOP-8767:


 Summary: secondary namenode on slave machines
 Key: HADOOP-8767
 URL: https://issues.apache.org/jira/browse/HADOOP-8767
 Project: Hadoop Common
  Issue Type: Bug
  Components: bin
Affects Versions: 1.0.3
Reporter: giovanni delussu
Priority: Minor


when the default value for HADOOP_SLAVES is changed in hadoop-env.sh the hdfs 
starting (with start-dfs.sh) creates secondary namenodes on all the machines in 
the file conf/slaves instead of conf/masters.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Build failed in Jenkins: Hadoop-Common-trunk #524

[Apache Jenkins Server]

See 

Changes:

[todd] HDFS-1490. TransferFSImage should timeout. Contributed by Dmytro Molkov 
and Vinay.

[eli] HADOOP-8764. CMake: HADOOP-8737 broke ARM build. Contributed by Trevor 
Robinson

[todd] HDFS-2793. Add an admin command to trigger an edit log roll. Contributed 
by Todd Lipcon.

[vinodkv] YARN-42. Modify NM's non-aggregating logs' handler to stop properly 
so that NMs don't get NPEs on startup errors. Contributed by Devaraj K.

[vinodkv] YARN-79. Implement close on all clients to YARN so that RPC clients 
don't throw exceptions on shut-down. Contributed by Vinod Kumar Vavilapalli.

[szetszwo] HDFS-3888. Clean up BlockPlacementPolicyDefault.  Contributed by 
Jing Zhao

[szetszwo] HDFS-3887. Remove redundant chooseTarget methods in 
BlockPlacementPolicy.  Contributed by Jing Zhao

[tucu] HDFS-3866. HttpFS POM should have property where to download tomcat from 
(zero45 via tucu)

[suresh] YARN-84. Use Builder to build RPC server. Contributed by Brandon Li

[bobby] Updated CHANGES.txt for 0.23.4

[vinodkv] YARN-10. Fix DistributedShell module to not have a dependency on 
hadoop-mapreduce-client-core. Contributed by Hitesh Shah.

--
[...truncated 26411 lines...]
[DEBUG]   (s) debug = false
[DEBUG]   (s) effort = Default
[DEBUG]   (s) failOnError = true
[DEBUG]   (s) findbugsXmlOutput = false
[DEBUG]   (s) findbugsXmlOutputDirectory = 

[DEBUG]   (s) fork = true
[DEBUG]   (s) includeTests = false
[DEBUG]   (s) localRepository =id: local
  url: file:///home/jenkins/.m2/repository/
   layout: none

[DEBUG]   (s) maxHeap = 512
[DEBUG]   (s) nested = false
[DEBUG]   (s) outputDirectory = 

[DEBUG]   (s) outputEncoding = UTF-8
[DEBUG]   (s) pluginArtifacts = 
[org.codehaus.mojo:findbugs-maven-plugin:maven-plugin:2.3.2:, 
com.google.code.findbugs:bcel:jar:1.3.9:compile, 
org.codehaus.gmaven:gmaven-mojo:jar:1.3:compile, 
org.codehaus.gmaven.runtime:gmaven-runtime-api:jar:1.3:compile, 
org.codehaus.gmaven.feature:gmaven-feature-api:jar:1.3:compile, 
org.codehaus.gmaven.runtime:gmaven-runtime-1.5:jar:1.3:compile, 
org.codehaus.gmaven.feature:gmaven-feature-support:jar:1.3:compile, 
org.codehaus.groovy:groovy-all-minimal:jar:1.5.8:compile, 
org.apache.ant:ant:jar:1.7.1:compile, 
org.apache.ant:ant-launcher:jar:1.7.1:compile, jline:jline:jar:0.9.94:compile, 
org.codehaus.plexus:plexus-interpolation:jar:1.1:compile, 
org.codehaus.gmaven:gmaven-plugin:jar:1.3:compile, 
org.codehaus.gmaven.runtime:gmaven-runtime-loader:jar:1.3:compile, 
org.codehaus.gmaven.runtime:gmaven-runtime-support:jar:1.3:compile, 
org.sonatype.gshell:gshell-io:jar:2.0:compile, 
com.thoughtworks.qdox:qdox:jar:1.10:compile, 
org.apache.maven.shared:file-management:jar:1.2.1:compile, 
org.apache.maven.shared:maven-shared-io:jar:1.1:compile, 
commons-lang:commons-lang:jar:2.4:compile, 
org.slf4j:slf4j-api:jar:1.5.10:compile, 
org.sonatype.gossip:gossip:jar:1.2:compile, 
org.apache.maven.reporting:maven-reporting-impl:jar:2.1:compile, 
commons-validator:commons-validator:jar:1.2.0:compile, 
commons-beanutils:commons-beanutils:jar:1.7.0:compile, 
commons-digester:commons-digester:jar:1.6:compile, 
commons-logging:commons-logging:jar:1.0.4:compile, oro:oro:jar:2.0.8:compile, 
xml-apis:xml-apis:jar:1.0.b2:compile, 
org.codehaus.groovy:groovy-all:jar:1.7.4:compile, 
org.apache.maven.reporting:maven-reporting-api:jar:3.0:compile, 
org.apache.maven.doxia:doxia-core:jar:1.1.3:compile, 
org.apache.maven.doxia:doxia-logging-api:jar:1.1.3:compile, 
xerces:xercesImpl:jar:2.9.1:compile, 
commons-httpclient:commons-httpclient:jar:3.1:compile, 
commons-codec:commons-codec:jar:1.2:compile, 
org.apache.maven.doxia:doxia-sink-api:jar:1.1.3:compile, 
org.apache.maven.doxia:doxia-decoration-model:jar:1.1.3:compile, 
org.apache.maven.doxia:doxia-site-renderer:jar:1.1.3:compile, 
org.apache.maven.doxia:doxia-module-xhtml:jar:1.1.3:compile, 
org.apache.maven.doxia:doxia-module-fml:jar:1.1.3:compile, 
org.codehaus.plexus:plexus-i18n:jar:1.0-beta-7:compile, 
org.codehaus.plexus:plexus-velocity:jar:1.1.7:compile, 
org.apache.velocity:velocity:jar:1.5:compile, 
commons-collections:commons-collections:jar:3.2:compile, 
org.apache.maven.shared:maven-doxia-tools:jar:1.2.1:compile, 
commons-io:commons-io:jar:1.4:compile, 
com.google.code.findbugs:findbugs-ant:jar:1.3.9:compile, 
com.google.code.findbugs:findbugs:jar:1.3.9:compile, 
com.google.code.findbugs:jsr305:jar:1.3.9:compile, 
com.google.code.findbugs:jFormatString:jar:1.3.9:compile, 
com.google.code.findbugs:annotations:jar:1.3.9:compile, 
dom4j:dom4j:jar:1.6.1:compile, jaxen:jaxen:jar:1.1.1:compile, 
jdom:jdom:jar:1.0:compile, xom:xom:jar:1.0:compile, 
xerces:xmlParserAPIs:jar:2.6.2:compile, xalan:xalan:jar:2.6.0:compile, 

Build failed in Jenkins: Hadoop-Common-0.23-Build #363

[Apache Jenkins Server]

See 

Changes:

[bobby] Updated version numbers for 0.23.4

--
[...truncated 11626 lines...]
Running 
org.apache.hadoop.io.file.tfile.TestTFileNoneCodecsJClassComparatorByteArrays
Tests run: 25, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.412 sec
Running org.apache.hadoop.io.file.tfile.TestTFileComparator2
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.57 sec
Running org.apache.hadoop.io.file.tfile.TestTFile
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.433 sec
Running org.apache.hadoop.io.file.tfile.TestTFileSplit
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.701 sec
Running org.apache.hadoop.io.file.tfile.TestTFileSeqFileComparison
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.271 sec
Running org.apache.hadoop.io.file.tfile.TestTFileNoneCodecsStreams
Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.226 sec
Running org.apache.hadoop.io.file.tfile.TestTFileComparators
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.588 sec
Running org.apache.hadoop.io.TestMapFile
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.712 sec
Running org.apache.hadoop.io.TestBytesWritable
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.156 sec
Running org.apache.hadoop.io.TestArrayFile
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.561 sec
Running org.apache.hadoop.io.TestVersionedWritable
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.133 sec
Running org.apache.hadoop.io.TestArrayWritable
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.138 sec
Running org.apache.hadoop.io.nativeio.TestNativeIO
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.006 sec
Running org.apache.hadoop.io.TestWritableUtils
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.192 sec
Running org.apache.hadoop.io.TestIOUtils
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.377 sec
Running org.apache.hadoop.io.serializer.avro.TestAvroSerialization
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.611 sec
Running org.apache.hadoop.io.serializer.TestWritableSerialization
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.356 sec
Running org.apache.hadoop.io.serializer.TestSerializationFactory
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.333 sec
Running org.apache.hadoop.test.TestMultithreadedTestUtil
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.239 sec
Running org.apache.hadoop.net.TestSocketIOWithTimeout
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.287 sec
Running org.apache.hadoop.net.TestSwitchMapping
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.342 sec
Running org.apache.hadoop.net.TestNetUtils
Tests run: 36, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.575 sec
Running org.apache.hadoop.net.TestStaticMapping
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.636 sec
Running org.apache.hadoop.net.TestScriptBasedMapping
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.891 sec
Running org.apache.hadoop.net.TestDNS
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.229 sec

Results :

Tests run: 1560, Failures: 0, Errors: 0, Skipped: 4

[INFO] 
[INFO] --- maven-clover2-plugin:3.0.5:clover (clover) @ hadoop-common ---
[INFO] Using /default-clover-report descriptor.
[INFO] Using Clover report descriptor: /tmp/mvn856092105075768478resource
[INFO] Clover Version 3.0.2, built on April 13 2010 (build-790)
[INFO] Loaded from: 
/home/jenkins/.m2/repository/com/cenqua/clover/clover/3.0.2/clover-3.0.2.jar
[INFO] Clover: Open Source License registered to Apache.
[INFO] Clover is enabled with initstring 
'
[WARNING] Clover historical directory 
[
 does not exist, skipping Clover historical report generation 
([
[INFO] Clover Version 3.0.2, built on April 13 2010 (build-790)
[INFO] Loaded from: 
/home/jenkins/.m2/repository/com/cenqua/clover/clover/3.0.2/clover-3.0.2.jar
[INFO] Clover: Open Source License registered to Apache.
[INFO] Loading coverage database from: 
'
[WARNING] Failed to load coverage recording 
RecordingTranscripts.FileRef[datafile=