Can you share with us in pastebin all conf files that you are using for
YARN?
On 06/13/2012 05:26 PM, anil gupta wrote:
Hi Marcus,
Sorry i forgot to mention that Job history server is installed and
running and AFAIK resourcemanager is responsible for running MR jobs.
Historyserver is only used to get info about MR jobs.
Thanks,
Anil
On Wed, Jun 13, 2012 at 2:04 PM, Marcos Ortiz <mlor...@uci.cu
<mailto:mlor...@uci.cu>> wrote:
According to the CDH 4 official documentation, you should install
a JobHistory server for your MRv2 (YARN)
cluster.
https://ccp.cloudera.com/display/CDH4DOC/Deploying+MapReduce+v2+%28YARN%29+on+a+Cluster
How to configure the HistoryServer
https://ccp.cloudera.com/display/CDH4DOC/Deploying+MapReduce+v2+%28YARN%29+on+a+Cluster#DeployingMapReducev2%28YARN%29onaCluster-Step3
On 06/13/2012 03:16 PM, anil gupta wrote:
Hi All
I am using cdh4 for running a HBase cluster on CentOs6.0. I have 5
nodes in my cluster(2 Admin Node and 3 DN).
My resourcemanager is up and running and showing that all
three DN are
running the nodemanager. HDFS is also working fine and showing
3 DN's.
But when i fire the pi example job. It starts to run in Local
mode.
Here is the console output:
sudo -u hdfs yarn jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-
examples.jar pi 10 1000000000
Number of Maps = 10
Samples per Map = 1000000000
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Wrote input for Map #5
Wrote input for Map #6
Wrote input for Map #7
Wrote input for Map #8
Wrote input for Map #9
Starting Job
12/06/13 12:03:27 WARN conf.Configuration: session.id
<http://session.id> is deprecated.
Instead, use dfs.metrics.session-id
12/06/13 12:03:27 INFO jvm.JvmMetrics: Initializing JVM
Metrics with
processName=JobTracker, sessionId=
12/06/13 12:03:27 INFO util.NativeCodeLoader: Loaded the
native-hadoop
library
12/06/13 12:03:27 WARN mapred.JobClient: Use
GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the
same.
12/06/13 12:03:28 INFO mapred.FileInputFormat: Total input
paths to
process : 10
12/06/13 12:03:29 INFO mapred.JobClient: Running job:
job_local_0001
12/06/13 12:03:29 INFO mapred.LocalJobRunner: OutputCommitter
set in
config null
12/06/13 12:03:29 INFO mapred.LocalJobRunner: OutputCommitter is
org.apache.hadoop.mapred.FileOutputCommitter
12/06/13 12:03:29 WARN mapreduce.Counters: Group
org.apache.hadoop.mapred.Task$Counter is deprecated. Use
org.apache.hadoop.mapreduce.TaskCounter instead
12/06/13 12:03:29 INFO util.ProcessTree: setsid exited with
exit code
0
12/06/13 12:03:29 INFO mapred.Task: Using
ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@3d46e381
12/06/13 12:03:29 WARN mapreduce.Counters: Counter name
MAP_INPUT_BYTES is deprecated. Use FileInputFormatCounters as
group
name and BYTES_READ as counter name instead
12/06/13 12:03:29 INFO mapred.MapTask: numReduceTasks: 1
12/06/13 12:03:29 INFO mapred.MapTask: io.sort.mb = 100
12/06/13 12:03:30 INFO mapred.MapTask: data buffer =
79691776/99614720
12/06/13 12:03:30 INFO mapred.MapTask: record buffer =
262144/327680
12/06/13 12:03:30 INFO mapred.JobClient: map 0% reduce 0%
12/06/13 12:03:35 INFO mapred.LocalJobRunner: Generated 95735000
samples.
12/06/13 12:03:36 INFO mapred.JobClient: map 100% reduce 0%
12/06/13 12:03:38 INFO mapred.LocalJobRunner: Generated 151872000
samples.
Here is the content of yarn-site.xml:
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce.shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<description>List of directories to store localized files in.</
description>
<name>yarn.nodemanager.local-dirs</name>
<value>/disk/yarn/local</value>
</property>
<property>
<description>Where to store container logs.</description>
<name>yarn.nodemanager.log-dirs</name>
<value>/disk/yarn/logs</value>
</property>
<property>
<description>Where to aggregate logs to.</description>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/var/log/hadoop-yarn/apps</value>
</property>
<property>
<description>Classpath for typical applications.</description>
<name>yarn.application.classpath</name>
<value>
$HADOOP_CONF_DIR,
$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,
$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,
$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,
$YARN_HOME/*,$YARN_HOME/lib/*
</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>ihub-an-g1:8025</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>ihub-an-g1:8040</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>ihub-an-g1:8030</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>ihub-an-g1:8141</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>ihub-an-g1:8088</value>
</property>
<property>
<name>mapreduce.jobhistory.intermediate-done-dir</name>
<value>/disk/mapred/jobhistory/intermediate/done</value>
</property>
<property>
<name>>mapreduce.jobhistory.done-dir</name>
<value>/disk/mapred/jobhistory/done</value>
</property>
</configuration>
Can anyone tell me what is the problem over here? Appreciate your
help.
Thanks,
Anil Gupta
--
Marcos Luis Ortíz Valmaseda
Data Engineer&& Sr. System Administrator at UCI
http://marcosluis2186.posterous.com
http://www.linkedin.com/in/marcosluis2186
Twitter: @marcosluis2186
10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS
INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci
--
Thanks & Regards,
Anil Gupta
--
Marcos Luis Ortíz Valmaseda
Data Engineer&& Sr. System Administrator at UCI
http://marcosluis2186.posterous.com
http://www.linkedin.com/in/marcosluis2186
Twitter: @marcosluis2186
10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS
INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci
