[jira] [Created] (HADOOP-8744) Fix the OSX native build

[Eli Collins (JIRA)]

Eli Collins created HADOOP-8744:
---

 Summary: Fix the OSX native build
 Key: HADOOP-8744
 URL: https://issues.apache.org/jira/browse/HADOOP-8744
 Project: Hadoop Common
  Issue Type: Bug
  Components: build
Affects Versions: 2.2.0-alpha
Reporter: Eli Collins


Per HADOOP-8737 let's fix the OSX native build, which was working as of 
HADOOP-3659 (v 0.21).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-8743) Change version in branch-2 to 2.2.0-SNAPSHOT, Update CHANGES.txt

[Siddharth Seth (JIRA)]

Siddharth Seth created HADOOP-8743:
--

 Summary: Change version in branch-2 to 2.2.0-SNAPSHOT, Update 
CHANGES.txt
 Key: HADOOP-8743
 URL: https://issues.apache.org/jira/browse/HADOOP-8743
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Siddharth Seth


The version in branch-2 is currently set to 2.0.1-SNAPSHOT, which has already 
been released.
This is to update the version to 2.2.0-SNAPSHOT to match the target versions in 
jira.

Also, CHANGES.txt is in bit of a mess. common, hdfs are missing sections for 
2.0.1, as well as 2.1.0.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-8742) Support compiling the native build with a different javac than the one under JAVA_HOME

[Colin Patrick McCabe (JIRA)]

Colin Patrick McCabe created HADOOP-8742:


 Summary: Support compiling the native build with a different javac 
than the one under JAVA_HOME
 Key: HADOOP-8742
 URL: https://issues.apache.org/jira/browse/HADOOP-8742
 Project: Hadoop Common
  Issue Type: New Feature
  Components: native
Reporter: Colin Patrick McCabe
Priority: Trivial


Maven supports compiling software with a different {{java}} than the one under 
the current {{JAVA_HOME}}.  See 
http://maven.apache.org/plugins/maven-compiler-plugin/examples/compile-using-different-jdk.html
 for details.

Although we have never supported this in the past, in the future we could 
support compiling the native code with a JDK other than the one referenced by 
the JDK in use by Maven.  Probably we would add a system property that could be 
passed in on the command-line.  Perhaps {{-Dnative.jdk.path=...}}.

Another thing to keep in mind is that the native build currently considers 
certain properties of the current Java environment when building.  For example, 
the native build checks {{sun.arch.data.model}} to see if it should build a 
32-bit or 64-bit native build.  We would have to make sure that this worked 
correctly with any "separate native JDK" feature we added.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Number of reducers

[Harsh J]

Ah, well, my bad. See instead the description for mapred.reduce.tasks
in mapred-default.xml, which states this: "Typically set to 99% of the
cluster's reduce capacity, so that if a node fails the reduces can
still be executed in a single wave."

FWIW, I set it manually to the level of parallelism I require (given
my partitioned data, etc.).

On Tue, Aug 28, 2012 at 8:43 PM, abhiTowson cal
 wrote:
> hi harsh,
>
> Thanks for the reply.I get your first and second points and coming to
> third point how is it specific to a job?
> My question was specific to job.
>
> Regards
> Abhishek
>
>
>
> On Mon, Aug 27, 2012 at 11:29 PM, Harsh J  wrote:
>> Hi,
>>
>> On Tue, Aug 28, 2012 at 8:32 AM, Abhishek  wrote:
>>> Hi all,
>>>
>>> I just want to know that, based on what factor map reduce framework decides 
>>> number of reducers to launch for a job
>>
>> The framework does not auto-determine the number of reducers for a
>> job. That is purely user-or-client-program-supplied presently.
>>
>>> By default only one reducer will be launched for a given job is this right? 
>>> If we explicitly does not mention number to launch via command line or 
>>> driver class.
>>
>> Yes, by default the number of reduce tasks is configured to be one.
>>
>>> If i choose to decide number of reducers to mention explicitly, what should 
>>> I consider.Because choosing in appropriate number of reducer hampers the 
>>> performance.
>>
>> See http://wiki.apache.org/hadoop/HowManyMapsAndReduces
>>
>> --
>> Harsh J



-- 
Harsh J

Re: Number of reducers

[abhiTowson cal]

hi harsh,

Thanks for the reply.I get your first and second points and coming to
third point how is it specific to a job?
My question was specific to job.

Regards
Abhishek



On Mon, Aug 27, 2012 at 11:29 PM, Harsh J  wrote:
> Hi,
>
> On Tue, Aug 28, 2012 at 8:32 AM, Abhishek  wrote:
>> Hi all,
>>
>> I just want to know that, based on what factor map reduce framework decides 
>> number of reducers to launch for a job
>
> The framework does not auto-determine the number of reducers for a
> job. That is purely user-or-client-program-supplied presently.
>
>> By default only one reducer will be launched for a given job is this right? 
>> If we explicitly does not mention number to launch via command line or 
>> driver class.
>
> Yes, by default the number of reduce tasks is configured to be one.
>
>> If i choose to decide number of reducers to mention explicitly, what should 
>> I consider.Because choosing in appropriate number of reducer hampers the 
>> performance.
>
> See http://wiki.apache.org/hadoop/HowManyMapsAndReduces
>
> --
> Harsh J

[jira] [Created] (HADOOP-8741) Broken links from "Cluster setup" to *-default.html

[Bertrand Dechoux (JIRA)]

Bertrand Dechoux created HADOOP-8741:


 Summary: Broken links from "Cluster setup" to *-default.html
 Key: HADOOP-8741
 URL: https://issues.apache.org/jira/browse/HADOOP-8741
 Project: Hadoop Common
  Issue Type: Bug
  Components: documentation
Affects Versions: 1.0.3
Reporter: Bertrand Dechoux
Priority: Minor


Hi,

The links from the cluster setup pages to the configuration files are broken.
http://hadoop.apache.org/common/docs/stable/cluster_setup.html

Read-only default configuration
http://hadoop.apache.org/common/docs/current/core-default.html
should be
http://hadoop.apache.org/common/docs/r1.0.3/core-default.html

The same holds for the three configuration : core, hdfs and mapred.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Review Request for HADOOP-8476 and HADOOP-8160

[Suresh Srinivas]

Vinay, will follow up these jiras sometime this week.

On Sat, Aug 25, 2012 at 2:19 AM, Vinayakumar B wrote:

> Hi,
>
>
>
> Please some one review patches for
>
>https://issues.apache.org/jira/browse/HADOOP-8476
>
>https://issues.apache.org/jira/browse/HADOOP-8160
>
>
>
>
>
> Thanks in advance.
>
>
>
> Regards,
>
> Vinay
>
>
>
>


-- 
http://hortonworks.com/download/

Build failed in Jenkins: Hadoop-Common-trunk #516

[Apache Jenkins Server]

See 

Changes:

[vinodkv] YARN-31. Fix TestDelegationTokenRenewer to not depend on test order 
so as to pass tests on jdk7. Contributed by Thomas Graves.

[vinodkv] MAPREDUCE-4579. Split TestTaskAttempt into two so as to pass tests on 
jdk7. Contributed by Thomas Graves.

[eli] Fixup CHANGELOG for HDFS-3856.

[eli] HDFS-3856. TestHDFSServerPorts failure is causing surefire fork failure. 
Contributed by Colin Patrick McCabe

[sseth] MAPREDUCE-4580. Change MapReduce to use the yarn-client module. 
(Contributed by Vinod Kumar Vavilapalli)

[harsh] HADOOP-8719. Workaround for kerberos-related log errors upon running 
any hadoop command on OSX. Contributed by Jianbin Wei. (harsh)

[sseth] YARN-37. Change TestRMAppTransitions to use the DrainDispatcher. 
(Contributed by Mayank Bansal)

[sseth] YARN-29. Add a yarn-client module. (Contributed by Vinod Kumar 
Vavilapalli)

--
[...truncated 27247 lines...]
[DEBUG]   (s) debug = false
[DEBUG]   (s) effort = Default
[DEBUG]   (s) failOnError = true
[DEBUG]   (s) findbugsXmlOutput = false
[DEBUG]   (s) findbugsXmlOutputDirectory = 

[DEBUG]   (s) fork = true
[DEBUG]   (s) includeTests = false
[DEBUG]   (s) localRepository =id: local
  url: file:///home/jenkins/.m2/repository/
   layout: none

[DEBUG]   (s) maxHeap = 512
[DEBUG]   (s) nested = false
[DEBUG]   (s) outputDirectory = 

[DEBUG]   (s) outputEncoding = UTF-8
[DEBUG]   (s) pluginArtifacts = 
[org.codehaus.mojo:findbugs-maven-plugin:maven-plugin:2.3.2:, 
com.google.code.findbugs:bcel:jar:1.3.9:compile, 
org.codehaus.gmaven:gmaven-mojo:jar:1.3:compile, 
org.codehaus.gmaven.runtime:gmaven-runtime-api:jar:1.3:compile, 
org.codehaus.gmaven.feature:gmaven-feature-api:jar:1.3:compile, 
org.codehaus.gmaven.runtime:gmaven-runtime-1.5:jar:1.3:compile, 
org.codehaus.gmaven.feature:gmaven-feature-support:jar:1.3:compile, 
org.codehaus.groovy:groovy-all-minimal:jar:1.5.8:compile, 
org.apache.ant:ant:jar:1.7.1:compile, 
org.apache.ant:ant-launcher:jar:1.7.1:compile, jline:jline:jar:0.9.94:compile, 
org.codehaus.plexus:plexus-interpolation:jar:1.1:compile, 
org.codehaus.gmaven:gmaven-plugin:jar:1.3:compile, 
org.codehaus.gmaven.runtime:gmaven-runtime-loader:jar:1.3:compile, 
org.codehaus.gmaven.runtime:gmaven-runtime-support:jar:1.3:compile, 
org.sonatype.gshell:gshell-io:jar:2.0:compile, 
com.thoughtworks.qdox:qdox:jar:1.10:compile, 
org.apache.maven.shared:file-management:jar:1.2.1:compile, 
org.apache.maven.shared:maven-shared-io:jar:1.1:compile, 
commons-lang:commons-lang:jar:2.4:compile, 
org.slf4j:slf4j-api:jar:1.5.10:compile, 
org.sonatype.gossip:gossip:jar:1.2:compile, 
org.apache.maven.reporting:maven-reporting-impl:jar:2.1:compile, 
commons-validator:commons-validator:jar:1.2.0:compile, 
commons-beanutils:commons-beanutils:jar:1.7.0:compile, 
commons-digester:commons-digester:jar:1.6:compile, 
commons-logging:commons-logging:jar:1.0.4:compile, oro:oro:jar:2.0.8:compile, 
xml-apis:xml-apis:jar:1.0.b2:compile, 
org.codehaus.groovy:groovy-all:jar:1.7.4:compile, 
org.apache.maven.reporting:maven-reporting-api:jar:3.0:compile, 
org.apache.maven.doxia:doxia-core:jar:1.1.3:compile, 
org.apache.maven.doxia:doxia-logging-api:jar:1.1.3:compile, 
xerces:xercesImpl:jar:2.9.1:compile, 
commons-httpclient:commons-httpclient:jar:3.1:compile, 
commons-codec:commons-codec:jar:1.2:compile, 
org.apache.maven.doxia:doxia-sink-api:jar:1.1.3:compile, 
org.apache.maven.doxia:doxia-decoration-model:jar:1.1.3:compile, 
org.apache.maven.doxia:doxia-site-renderer:jar:1.1.3:compile, 
org.apache.maven.doxia:doxia-module-xhtml:jar:1.1.3:compile, 
org.apache.maven.doxia:doxia-module-fml:jar:1.1.3:compile, 
org.codehaus.plexus:plexus-i18n:jar:1.0-beta-7:compile, 
org.codehaus.plexus:plexus-velocity:jar:1.1.7:compile, 
org.apache.velocity:velocity:jar:1.5:compile, 
commons-collections:commons-collections:jar:3.2:compile, 
org.apache.maven.shared:maven-doxia-tools:jar:1.2.1:compile, 
commons-io:commons-io:jar:1.4:compile, 
com.google.code.findbugs:findbugs-ant:jar:1.3.9:compile, 
com.google.code.findbugs:findbugs:jar:1.3.9:compile, 
com.google.code.findbugs:jsr305:jar:1.3.9:compile, 
com.google.code.findbugs:jFormatString:jar:1.3.9:compile, 
com.google.code.findbugs:annotations:jar:1.3.9:compile, 
dom4j:dom4j:jar:1.6.1:compile, jaxen:jaxen:jar:1.1.1:compile, 
jdom:jdom:jar:1.0:compile, xom:xom:jar:1.0:compile, 
xerces:xmlParserAPIs:jar:2.6.2:compile, xalan:xalan:jar:2.6.0:compile, 
com.ibm.icu:icu4j:jar:2.6.1:compile, asm:asm:jar:3.1:compile, 
asm:asm-analysis:jar:3.1:compile, asm:asm-commons:jar:3.1:compile, 
asm:asm-util:jar:3.1:compile, asm:asm-tree:jar:3.1:compile, 
asm:asm-xml:jar:3.1:compile, jgoodies:plastic:jar:1.2.0:compile, 
org.codehaus.plexus:p