Dears, Sorry, I did not mean to cross post. But the previous article was accidentally posted to the HBase user list. I would like to bring it back to the Hadoop user since it is confusing me a lot and it is mainly MapReduce related.
Currently running version hadoop-0.18.1 on 25 nodes. Map and Reduce Task Capacity is 92. When I do this in my MapReduce program: ============= SAMPLE CODE ============= JobConf jconf = new JobConf(conf, TestTask.class); jconf.setJobName("my.test.TestTask"); jconf.setOutputKeyClass(Text.class); jconf.setOutputValueClass(Text.class); jconf.setOutputFormat(TextOutputFormat.class); jconf.setMapperClass(MyMapper.class); jconf.setCombinerClass(MyReducer.class); jconf.setReducerClass(MyReducer.class); jconf.setInputFormat(TextInputFormat.class); try { jconf.setNumMapTasks(5); jconf.setNumReduceTasks(3); JobClient.runJob(jconf); } catch (Exception e) { e.printStackTrace(); } ============= ============= ============= When I run the job, I'm always getting 300 mappers and 1 reducers from the JobTracker webpage running on the default port 50030. No matter how I configure the numbers in methods "setNumMapTasks" and "setNumReduceTasks", I get the same result. Does anyone know why this is happening? Am I missing something or misunderstand something in the picture? =( Here's a reference to the parameters we have override in "hadoop-site.xml". =============== <property> <name>mapred.tasktracker.map.tasks.maximum</name> <value>4</value> </property> <property> <name>mapred.tasktracker.reduce.tasks.maximum</name> <value>4</value> </property> ================ other parameters are default from hadoop-default.xml. Any idea how this is happening? Any inputs are appreciated. Thanks, -Andy