Re: Yarn job runs in Local Mode even though the cluster is running in Distributed Mode

Marcos Ortiz Wed, 13 Jun 2012 14:05:14 -0700

According to the CDH 4 official documentation, you should install aJobHistory server for your MRv2 (YARN)

cluster.
https://ccp.cloudera.com/display/CDH4DOC/Deploying+MapReduce+v2+%28YARN%29+on+a+Cluster


How to configure the HistoryServer
https://ccp.cloudera.com/display/CDH4DOC/Deploying+MapReduce+v2+%28YARN%29+on+a+Cluster#DeployingMapReducev2%28YARN%29onaCluster-Step3



On 06/13/2012 03:16 PM, anil gupta wrote:

Hi All

I am using cdh4 for running a HBase cluster on CentOs6.0. I have 5
nodes in my cluster(2 Admin Node and 3 DN).
My resourcemanager is up and running and showing that all three DN are
running the nodemanager. HDFS is also working fine and showing 3 DN's.

But when i fire the pi example job. It starts to run in Local mode.
Here is the console output:
sudo -u hdfs yarn jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-
examples.jar pi 10 1000000000
Number of Maps  = 10
Samples per Map = 1000000000
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Wrote input for Map #5
Wrote input for Map #6
Wrote input for Map #7
Wrote input for Map #8
Wrote input for Map #9
Starting Job
12/06/13 12:03:27 WARN conf.Configuration: session.id is deprecated.
Instead, use dfs.metrics.session-id
12/06/13 12:03:27 INFO jvm.JvmMetrics: Initializing JVM Metrics with
processName=JobTracker, sessionId=
12/06/13 12:03:27 INFO util.NativeCodeLoader: Loaded the native-hadoop
library
12/06/13 12:03:27 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the
same.
12/06/13 12:03:28 INFO mapred.FileInputFormat: Total input paths to
process : 10
12/06/13 12:03:29 INFO mapred.JobClient: Running job: job_local_0001
12/06/13 12:03:29 INFO mapred.LocalJobRunner: OutputCommitter set in
config null
12/06/13 12:03:29 INFO mapred.LocalJobRunner: OutputCommitter is
org.apache.hadoop.mapred.FileOutputCommitter
12/06/13 12:03:29 WARN mapreduce.Counters: Group
org.apache.hadoop.mapred.Task$Counter is deprecated. Use
org.apache.hadoop.mapreduce.TaskCounter instead
12/06/13 12:03:29 INFO util.ProcessTree: setsid exited with exit code
0
12/06/13 12:03:29 INFO mapred.Task:  Using ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@3d46e381
12/06/13 12:03:29 WARN mapreduce.Counters: Counter name
MAP_INPUT_BYTES is deprecated. Use FileInputFormatCounters as group
name and  BYTES_READ as counter name instead
12/06/13 12:03:29 INFO mapred.MapTask: numReduceTasks: 1
12/06/13 12:03:29 INFO mapred.MapTask: io.sort.mb = 100
12/06/13 12:03:30 INFO mapred.MapTask: data buffer = 79691776/99614720
12/06/13 12:03:30 INFO mapred.MapTask: record buffer = 262144/327680
12/06/13 12:03:30 INFO mapred.JobClient:  map 0% reduce 0%
12/06/13 12:03:35 INFO mapred.LocalJobRunner: Generated 95735000
samples.
12/06/13 12:03:36 INFO mapred.JobClient:  map 100% reduce 0%
12/06/13 12:03:38 INFO mapred.LocalJobRunner: Generated 151872000
samples.

Here is the content of yarn-site.xml:

<configuration>
   <property>
     <name>yarn.nodemanager.aux-services</name>
     <value>mapreduce.shuffle</value>
   </property>

   <property>
     <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
     <value>org.apache.hadoop.mapred.ShuffleHandler</value>
   </property>

   <property>
     <name>yarn.log-aggregation-enable</name>
     <value>true</value>
   </property>

   <property>
     <description>List of directories to store localized files in.</
description>
     <name>yarn.nodemanager.local-dirs</name>
     <value>/disk/yarn/local</value>
   </property>

   <property>
     <description>Where to store container logs.</description>
     <name>yarn.nodemanager.log-dirs</name>
     <value>/disk/yarn/logs</value>
   </property>

   <property>
     <description>Where to aggregate logs to.</description>
     <name>yarn.nodemanager.remote-app-log-dir</name>
     <value>/var/log/hadoop-yarn/apps</value>
   </property>

   <property>
     <description>Classpath for typical applications.</description>
      <name>yarn.application.classpath</name>
      <value>
         $HADOOP_CONF_DIR,
         $HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,
         $HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,
         $HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,
         $YARN_HOME/*,$YARN_HOME/lib/*
      </value>
   </property>
<property>
         <name>yarn.resourcemanager.resource-tracker.address</name>
         <value>ihub-an-g1:8025</value>
</property>
<property>
         <name>yarn.resourcemanager.address</name>
         <value>ihub-an-g1:8040</value>
</property>
<property>
         <name>yarn.resourcemanager.scheduler.address</name>
         <value>ihub-an-g1:8030</value>
</property>
<property>
         <name>yarn.resourcemanager.admin.address</name>
         <value>ihub-an-g1:8141</value>
</property>
<property>
         <name>yarn.resourcemanager.webapp.address</name>
         <value>ihub-an-g1:8088</value>
</property>
<property>
         <name>mapreduce.jobhistory.intermediate-done-dir</name>
         <value>/disk/mapred/jobhistory/intermediate/done</value>
</property>
<property>
         <name>>mapreduce.jobhistory.done-dir</name>
         <value>/disk/mapred/jobhistory/done</value>
</property>
</configuration>

Can anyone tell me what is the problem over here? Appreciate your
help.
Thanks,
Anil Gupta


--
Marcos Luis Ortíz Valmaseda
 Data Engineer&&  Sr. System Administrator at UCI
 http://marcosluis2186.posterous.com
 http://www.linkedin.com/in/marcosluis2186
 Twitter: @marcosluis2186


10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS 
INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION

http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci

Re: Yarn job runs in Local Mode even though the cluster is running in Distributed Mode

Reply via email to