[jira] [Created] (HADOOP-8620) Add -Drequire.fuse and -Drequire.snappy

[Colin Patrick McCabe (JIRA)]

Colin Patrick McCabe created HADOOP-8620:


 Summary: Add -Drequire.fuse and -Drequire.snappy
 Key: HADOOP-8620
 URL: https://issues.apache.org/jira/browse/HADOOP-8620
 Project: Hadoop Common
  Issue Type: Improvement
  Components: build
Affects Versions: 2.0.1-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor


We have some optional build components which don't get built if they are not 
installed on the build machine.  One of those components is fuse_dfs.  Another 
is the snappy support in libhadoop.so.

Unfortunately, since these components are silently ignored if they are not 
present, it's easy to unintentionally create an incomplete build.

We should add two flags, -Drequire.fuse and -Drequire.snappy, that do exactly 
what the names suggest.  This will make the build more repeatable.  Those who 
want a complete build can specify these system properties to maven.  If the 
build cannot be created as requested, it will be a hard error.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HADOOP-8618) Windows build failing after 1.0.3 got merged into branch-1-win

[Suresh Srinivas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-8618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Srinivas resolved HADOOP-8618.
-

  Resolution: Fixed
Hadoop Flags: Reviewed

I committed the patch.

> Windows build failing after 1.0.3 got merged into branch-1-win
> --
>
> Key: HADOOP-8618
> URL: https://issues.apache.org/jira/browse/HADOOP-8618
> Project: Hadoop Common
>  Issue Type: Sub-task
>Affects Versions: 1-win
>Reporter: Bikas Saha
>Assignee: Bikas Saha
> Attachments: HADOOP-8618-1.patch, HADOOP-8618-2.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-8619) WritableComparator must implement no-arg constructor

[Radim Kolar (JIRA)]

Radim Kolar created HADOOP-8619:
---

 Summary: WritableComparator must implement no-arg constructor
 Key: HADOOP-8619
 URL: https://issues.apache.org/jira/browse/HADOOP-8619
 Project: Hadoop Common
  Issue Type: Improvement
  Components: io
Affects Versions: 3.0.0
Reporter: Radim Kolar
 Fix For: 3.0.0, 2.0.0-alpha, 0.23.0
 Attachments: writable-comparator.txt

Because of reasons listed here: 
http://findbugs.sourceforge.net/bugDescriptions.html#SE_COMPARATOR_SHOULD_BE_SERIALIZABLE

comparators should be serializable. To make deserialization work, it is 
required that all superclasses have no-arg constructor. 
http://findbugs.sourceforge.net/bugDescriptions.html#SE_NO_SUITABLE_CONSTRUCTOR

Simply add no=arg constructor to  WritableComparator.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-8618) Windows build failing after 1.0.3 got merged into branch-1-win

[Bikas Saha (JIRA)]

Bikas Saha created HADOOP-8618:
--

 Summary: Windows build failing after 1.0.3 got merged into 
branch-1-win
 Key: HADOOP-8618
 URL: https://issues.apache.org/jira/browse/HADOOP-8618
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Bikas Saha
Assignee: Bikas Saha




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-8617) backport pure Java CRC32 calculator changes to branch-1

[Brandon Li (JIRA)]

Brandon Li created HADOOP-8617:
--

 Summary: backport pure Java CRC32 calculator changes to branch-1
 Key: HADOOP-8617
 URL: https://issues.apache.org/jira/browse/HADOOP-8617
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Brandon Li
Assignee: Brandon Li


Multiple efforts have been made gradually to improve the CRC performance in 
Hadoop. This JIRA is to back port these changes to branch-1, which include 
HADOOP-6166, HADOOP-6148, HADOOP-7333.

The related HDFS and MAPREDUCE patches (e.g., HDFS-496 and MR-782) will be 
uploaded to their original JIRAs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Build failed in Jenkins: Hadoop-Common-trunk #482

[Apache Jenkins Server]

See 

Changes:

[atm] HDFS-3711. Manually convert remaining tests to JUnit4. Contributed by 
Andrew Wang.

--
[...truncated 26763 lines...]
[DEBUG]   (s) debug = false
[DEBUG]   (s) effort = Default
[DEBUG]   (s) failOnError = true
[DEBUG]   (s) findbugsXmlOutput = false
[DEBUG]   (s) findbugsXmlOutputDirectory = 

[DEBUG]   (s) fork = true
[DEBUG]   (s) includeTests = false
[DEBUG]   (s) localRepository =id: local
  url: file:///home/jenkins/.m2/repository/
   layout: none

[DEBUG]   (s) maxHeap = 512
[DEBUG]   (s) nested = false
[DEBUG]   (s) outputDirectory = 

[DEBUG]   (s) outputEncoding = UTF-8
[DEBUG]   (s) pluginArtifacts = 
[org.codehaus.mojo:findbugs-maven-plugin:maven-plugin:2.3.2:, 
com.google.code.findbugs:bcel:jar:1.3.9:compile, 
org.codehaus.gmaven:gmaven-mojo:jar:1.3:compile, 
org.codehaus.gmaven.runtime:gmaven-runtime-api:jar:1.3:compile, 
org.codehaus.gmaven.feature:gmaven-feature-api:jar:1.3:compile, 
org.codehaus.gmaven.runtime:gmaven-runtime-1.5:jar:1.3:compile, 
org.codehaus.gmaven.feature:gmaven-feature-support:jar:1.3:compile, 
org.codehaus.groovy:groovy-all-minimal:jar:1.5.8:compile, 
org.apache.ant:ant:jar:1.7.1:compile, 
org.apache.ant:ant-launcher:jar:1.7.1:compile, jline:jline:jar:0.9.94:compile, 
org.codehaus.plexus:plexus-interpolation:jar:1.1:compile, 
org.codehaus.gmaven:gmaven-plugin:jar:1.3:compile, 
org.codehaus.gmaven.runtime:gmaven-runtime-loader:jar:1.3:compile, 
org.codehaus.gmaven.runtime:gmaven-runtime-support:jar:1.3:compile, 
org.sonatype.gshell:gshell-io:jar:2.0:compile, 
com.thoughtworks.qdox:qdox:jar:1.10:compile, 
org.apache.maven.shared:file-management:jar:1.2.1:compile, 
org.apache.maven.shared:maven-shared-io:jar:1.1:compile, 
commons-lang:commons-lang:jar:2.4:compile, 
org.slf4j:slf4j-api:jar:1.5.10:compile, 
org.sonatype.gossip:gossip:jar:1.2:compile, 
org.apache.maven.reporting:maven-reporting-impl:jar:2.1:compile, 
commons-validator:commons-validator:jar:1.2.0:compile, 
commons-beanutils:commons-beanutils:jar:1.7.0:compile, 
commons-digester:commons-digester:jar:1.6:compile, 
commons-logging:commons-logging:jar:1.0.4:compile, oro:oro:jar:2.0.8:compile, 
xml-apis:xml-apis:jar:1.0.b2:compile, 
org.codehaus.groovy:groovy-all:jar:1.7.4:compile, 
org.apache.maven.reporting:maven-reporting-api:jar:3.0:compile, 
org.apache.maven.doxia:doxia-core:jar:1.1.3:compile, 
org.apache.maven.doxia:doxia-logging-api:jar:1.1.3:compile, 
xerces:xercesImpl:jar:2.9.1:compile, 
commons-httpclient:commons-httpclient:jar:3.1:compile, 
commons-codec:commons-codec:jar:1.2:compile, 
org.apache.maven.doxia:doxia-sink-api:jar:1.1.3:compile, 
org.apache.maven.doxia:doxia-decoration-model:jar:1.1.3:compile, 
org.apache.maven.doxia:doxia-site-renderer:jar:1.1.3:compile, 
org.apache.maven.doxia:doxia-module-xhtml:jar:1.1.3:compile, 
org.apache.maven.doxia:doxia-module-fml:jar:1.1.3:compile, 
org.codehaus.plexus:plexus-i18n:jar:1.0-beta-7:compile, 
org.codehaus.plexus:plexus-velocity:jar:1.1.7:compile, 
org.apache.velocity:velocity:jar:1.5:compile, 
commons-collections:commons-collections:jar:3.2:compile, 
org.apache.maven.shared:maven-doxia-tools:jar:1.2.1:compile, 
commons-io:commons-io:jar:1.4:compile, 
com.google.code.findbugs:findbugs-ant:jar:1.3.9:compile, 
com.google.code.findbugs:findbugs:jar:1.3.9:compile, 
com.google.code.findbugs:jsr305:jar:1.3.9:compile, 
com.google.code.findbugs:jFormatString:jar:1.3.9:compile, 
com.google.code.findbugs:annotations:jar:1.3.9:compile, 
dom4j:dom4j:jar:1.6.1:compile, jaxen:jaxen:jar:1.1.1:compile, 
jdom:jdom:jar:1.0:compile, xom:xom:jar:1.0:compile, 
xerces:xmlParserAPIs:jar:2.6.2:compile, xalan:xalan:jar:2.6.0:compile, 
com.ibm.icu:icu4j:jar:2.6.1:compile, asm:asm:jar:3.1:compile, 
asm:asm-analysis:jar:3.1:compile, asm:asm-commons:jar:3.1:compile, 
asm:asm-util:jar:3.1:compile, asm:asm-tree:jar:3.1:compile, 
asm:asm-xml:jar:3.1:compile, jgoodies:plastic:jar:1.2.0:compile, 
org.codehaus.plexus:plexus-resources:jar:1.0-alpha-4:compile, 
org.codehaus.plexus:plexus-utils:jar:1.5.1:compile]
[DEBUG]   (s) project = MavenProject: 
org.apache.hadoop:hadoop-common-project:3.0.0-SNAPSHOT @ 

[DEBUG]   (s) relaxed = false
[DEBUG]   (s) remoteArtifactRepositories = [   id: apache.snapshots.https
  url: https://repository.apache.org/content/repositories/snapshots
   layout: default
snapshots: [enabled => true, update => daily]
 releases: [enabled => true, update => daily]
,id: repository.jboss.org
  url: http://repository.jboss.org/nexus/content/groups/public/
   layout: default
snapshots: [enabled => false, update => daily]
 releases: [enabled => true, 

Re: Build failed in Jenkins: Hadoop-Common-trunk #481

[Eli Collins]

I fixed up the command jenkins was running to hopefully fix this error.


On Tue, Jul 24, 2012 at 2:27 AM, Apache Jenkins Server
 wrote:
> See 
>
> Changes:
>
> [eli] HDFS-3709. TestStartup tests still binding to the ephemeral port. 
> Contributed by Eli Collins
>
> [bobby] MAPREDUCE-3893. allow capacity scheduler configs max-apps and 
> max-am-pct per queue (tgraves via bobby)
>
> [todd] HDFS-3697. Enable fadvise readahead by default. Contributed by Todd 
> Lipcon.
>
> --
> [...truncated 18195 lines...]
> [DEBUG]   (s) reportsDirectory = 
> 
> [DEBUG]   (s) runOrder = filesystem
> [DEBUG]   (s) session = org.apache.maven.execution.MavenSession@98f352
> [DEBUG]   (s) skip = false
> [DEBUG]   (s) skipTests = false
> [DEBUG]   (s) systemPropertyVariables = {hadoop.log.dir=null, 
> hadoop.tmp.dir=null, java.net.preferIPv4Stack=true, 
> java.security.egd=file:///dev/urandom, 
> java.security.krb5.conf=
>  test.build.classes=null, 
> test.build.data=
>  
> test.build.dir=
>  test.build.webapps=null, test.cache.data=null}
> [DEBUG]   (s) testClassesDirectory = 
> 
> [DEBUG]   (s) testFailureIgnore = false
> [DEBUG]   (s) testNGArtifactName = org.testng:testng
> [DEBUG]   (s) testSourceDirectory = 
> 
> [DEBUG]   (s) trimStackTrace = true
> [DEBUG]   (s) useFile = true
> [DEBUG]   (s) useManifestOnlyJar = true
> [DEBUG]   (s) useSystemClassLoader = true
> [DEBUG]   (s) useUnlimitedThreads = false
> [DEBUG]   (s) workingDirectory = 
> 
> [DEBUG] -- end configuration --
> [INFO] No tests to run.
> [INFO] Surefire report directory: 
> 
> [DEBUG] dummy:dummy:jar:1.0 (selected for null)
> [DEBUG]   org.apache.maven.surefire:surefire-booter:jar:2.12:compile 
> (selected for compile)
> [DEBUG] org.apache.maven.surefire:surefire-api:jar:2.12:compile (selected 
> for compile)
> [DEBUG] Adding to surefire booter test classpath: 
> /home/jenkins/.m2/repository/org/apache/maven/surefire/surefire-booter/2.12/surefire-booter-2.12.jar
>  Scope: compile
> [DEBUG] Adding to surefire booter test classpath: 
> /home/jenkins/.m2/repository/org/apache/maven/surefire/surefire-api/2.12/surefire-api-2.12.jar
>  Scope: compile
> [DEBUG] Setting system property [java.net.preferIPv4Stack]=[true]
> [DEBUG] Setting system property 
> [java.security.krb5.conf]=[
> [DEBUG] Setting system property [tar]=[true]
> [DEBUG] Setting system property 
> [test.build.data]=[
> [DEBUG] Setting system property 
> [user.dir]=[
> [DEBUG] Setting system property 
> [localRepository]=[/home/jenkins/.m2/repository]
> [DEBUG] Setting system property 
> [test.build.dir]=[
> [DEBUG] Setting system property 
> [basedir]=[
> [DEBUG] Setting system property [java.security.egd]=[file:///dev/urandom]
> [DEBUG] Using JVM: /home/jenkins/tools/java/jdk1.6.0_27/jre/bin/java
> [DEBUG] Setting environment variable 
> [LD_LIBRARY_PATH]=[/home/jenkins/tools/java/jdk1.6.0_27/jre/lib/i386/server:/home/jenkins/tools/java/jdk1.6.0_27/jre/lib/i386:/home/jenkins/tools/java/jdk1.6.0_27/jre/../lib/i386:
> [DEBUG] Setting environment variable [MALLOC_ARENA_MAX]=[4]
> [DEBUG] dummy:dummy:jar:1.0 (selected for null)
> [DEBUG]   org.apache.maven.surefire:surefire-junit3:jar:2.12:test (selected 
> for test)
> [DEBUG] org.a

Re: Hive partitioner

[Abhishek]

Data is skewed here so does it help?

Thanks
Abhi


Sent from my iPhone

On Jul 24, 2012, at 9:50 AM, Nitin Pawar  wrote:

> Tried rarely.
> 
> It also depends on the data :)
> 
> On Tue, Jul 24, 2012 at 6:26 PM, Abhishek  wrote:
>> Hi nitin,
>> 
>> Thanks for the reply.By setting this in hive does it improve the performance 
>> ?
>> 
>> Regards
>> Abhi
>> 
>> Sent from my iPhone
>> 
>> On Jul 23, 2012, at 3:18 AM, Nitin Pawar  wrote:
>> 
>>> you may try this
>>> 
>>> set hive.mapred.partitioner=
>>> org.apache.hadoop.mapred.lib.TotalOrderPartitioner;
>>> set total.order.partitioner.natural.order=[false|true];
>>> set total.order.partitioner.path=[path];
>>> 
>>> On Mon, Jul 23, 2012 at 8:56 AM, abhiTowson cal
>>>  wrote:
 
 hi all,
 
 How to use total order partitioner hive?
 
 Regards
 Abhishek
>>> 
>>> 
>>> 
>>> 
>>> --
>>> Nitin Pawar
> 
> 
> 
> -- 
> Nitin Pawar

Re: Hive partitioner

[Nitin Pawar]

Tried rarely.

It also depends on the data :)

On Tue, Jul 24, 2012 at 6:26 PM, Abhishek  wrote:
> Hi nitin,
>
> Thanks for the reply.By setting this in hive does it improve the performance ?
>
> Regards
> Abhi
>
> Sent from my iPhone
>
> On Jul 23, 2012, at 3:18 AM, Nitin Pawar  wrote:
>
>> you may try this
>>
>> set hive.mapred.partitioner=
>> org.apache.hadoop.mapred.lib.TotalOrderPartitioner;
>> set total.order.partitioner.natural.order=[false|true];
>> set total.order.partitioner.path=[path];
>>
>> On Mon, Jul 23, 2012 at 8:56 AM, abhiTowson cal
>>  wrote:
>>>
>>> hi all,
>>>
>>> How to use total order partitioner hive?
>>>
>>> Regards
>>> Abhishek
>>
>>
>>
>>
>> --
>> Nitin Pawar



-- 
Nitin Pawar

Re: Hive partitioner

[Abhishek]

Hi nitin,

Thanks for the reply.By setting this in hive does it improve the performance ?

Regards
Abhi

Sent from my iPhone

On Jul 23, 2012, at 3:18 AM, Nitin Pawar  wrote:

> you may try this
> 
> set hive.mapred.partitioner=
> org.apache.hadoop.mapred.lib.TotalOrderPartitioner;
> set total.order.partitioner.natural.order=[false|true];
> set total.order.partitioner.path=[path];
> 
> On Mon, Jul 23, 2012 at 8:56 AM, abhiTowson cal
>  wrote:
>> 
>> hi all,
>> 
>> How to use total order partitioner hive?
>> 
>> Regards
>> Abhishek
> 
> 
> 
> 
> --
> Nitin Pawar

Build failed in Jenkins: Hadoop-Common-trunk #481

[Apache Jenkins Server]

See 

Changes:

[eli] HDFS-3709. TestStartup tests still binding to the ephemeral port. 
Contributed by Eli Collins

[bobby] MAPREDUCE-3893. allow capacity scheduler configs max-apps and 
max-am-pct per queue (tgraves via bobby)

[todd] HDFS-3697. Enable fadvise readahead by default. Contributed by Todd 
Lipcon.

--
[...truncated 18195 lines...]
[DEBUG]   (s) reportsDirectory = 

[DEBUG]   (s) runOrder = filesystem
[DEBUG]   (s) session = org.apache.maven.execution.MavenSession@98f352
[DEBUG]   (s) skip = false
[DEBUG]   (s) skipTests = false
[DEBUG]   (s) systemPropertyVariables = {hadoop.log.dir=null, 
hadoop.tmp.dir=null, java.net.preferIPv4Stack=true, 
java.security.egd=file:///dev/urandom, 
java.security.krb5.conf=
 test.build.classes=null, 
test.build.data=
 
test.build.dir=
 test.build.webapps=null, test.cache.data=null}
[DEBUG]   (s) testClassesDirectory = 

[DEBUG]   (s) testFailureIgnore = false
[DEBUG]   (s) testNGArtifactName = org.testng:testng
[DEBUG]   (s) testSourceDirectory = 

[DEBUG]   (s) trimStackTrace = true
[DEBUG]   (s) useFile = true
[DEBUG]   (s) useManifestOnlyJar = true
[DEBUG]   (s) useSystemClassLoader = true
[DEBUG]   (s) useUnlimitedThreads = false
[DEBUG]   (s) workingDirectory = 

[DEBUG] -- end configuration --
[INFO] No tests to run.
[INFO] Surefire report directory: 

[DEBUG] dummy:dummy:jar:1.0 (selected for null)
[DEBUG]   org.apache.maven.surefire:surefire-booter:jar:2.12:compile (selected 
for compile)
[DEBUG] org.apache.maven.surefire:surefire-api:jar:2.12:compile (selected 
for compile)
[DEBUG] Adding to surefire booter test classpath: 
/home/jenkins/.m2/repository/org/apache/maven/surefire/surefire-booter/2.12/surefire-booter-2.12.jar
 Scope: compile
[DEBUG] Adding to surefire booter test classpath: 
/home/jenkins/.m2/repository/org/apache/maven/surefire/surefire-api/2.12/surefire-api-2.12.jar
 Scope: compile
[DEBUG] Setting system property [java.net.preferIPv4Stack]=[true]
[DEBUG] Setting system property 
[java.security.krb5.conf]=[
[DEBUG] Setting system property [tar]=[true]
[DEBUG] Setting system property 
[test.build.data]=[
[DEBUG] Setting system property 
[user.dir]=[
[DEBUG] Setting system property [localRepository]=[/home/jenkins/.m2/repository]
[DEBUG] Setting system property 
[test.build.dir]=[
[DEBUG] Setting system property 
[basedir]=[
[DEBUG] Setting system property [java.security.egd]=[file:///dev/urandom]
[DEBUG] Using JVM: /home/jenkins/tools/java/jdk1.6.0_27/jre/bin/java
[DEBUG] Setting environment variable 
[LD_LIBRARY_PATH]=[/home/jenkins/tools/java/jdk1.6.0_27/jre/lib/i386/server:/home/jenkins/tools/java/jdk1.6.0_27/jre/lib/i386:/home/jenkins/tools/java/jdk1.6.0_27/jre/../lib/i386:
[DEBUG] Setting environment variable [MALLOC_ARENA_MAX]=[4]
[DEBUG] dummy:dummy:jar:1.0 (selected for null)
[DEBUG]   org.apache.maven.surefire:surefire-junit3:jar:2.12:test (selected for 
test)
[DEBUG] org.apache.maven.surefire:surefire-api:jar:2.12:test (selected for 
test)
[DEBUG] Adding to surefire test classpath: 
/home/jenkins/.m2/repository/org/apache/maven/surefire/surefire-junit3/2.12/surefire-junit3-2.12.jar
 Scope: test
[DEBUG] Adding to surefire test classpath: 
/home/jenkins/.m2/reposi