Dears,

Sorry, I did not mean to cross post.  But the previous article was
accidentally posted to the HBase user list.  I would like to bring it back
to the Hadoop user since it is confusing me a lot and it is mainly MapReduce
related.

Currently running version hadoop-0.18.1 on 25 nodes.  Map and Reduce Task
Capacity is 92.  When I do this in my MapReduce program:

============= SAMPLE CODE =============
        JobConf jconf = new JobConf(conf, TestTask.class);
        jconf.setJobName("my.test.TestTask");
        jconf.setOutputKeyClass(Text.class);
        jconf.setOutputValueClass(Text.class);
        jconf.setOutputFormat(TextOutputFormat.class);
        jconf.setMapperClass(MyMapper.class);
        jconf.setCombinerClass(MyReducer.class);
        jconf.setReducerClass(MyReducer.class);
        jconf.setInputFormat(TextInputFormat.class);
        try {
            jconf.setNumMapTasks(5);
            jconf.setNumReduceTasks(3);
            JobClient.runJob(jconf);
        } catch (Exception e) {
            e.printStackTrace();
        }
============= ============= =============

When I run the job, I'm always getting 300 mappers and 1 reducers from the
JobTracker webpage running on the default port 50030.
No matter how I configure the numbers in methods "setNumMapTasks" and
"setNumReduceTasks", I get the same result.
Does anyone know why this is happening?
Am I missing something or misunderstand something in the picture?  =(

Here's a reference to the parameters we have override in "hadoop-site.xml".
===============
<property>
  <name>mapred.tasktracker.map.tasks.maximum</name>
  <value>4</value>
</property>

<property>
  <name>mapred.tasktracker.reduce.tasks.maximum</name>
  <value>4</value>
</property>
================
other parameters are default from hadoop-default.xml.

Any idea how this is happening?

Any inputs are appreciated.

Thanks,
-Andy

Reply via email to