Hi,
i was trying to create a test based on mapreduce job in a local mode
testing various partitioning issues.
But curiously, whenever i switch mapreduce into local node, i can't
seem to be able to configure multiple reduce tasks.
Indeed, upon some investigation i found that the following fragment in
LocalJobRunner resets all reducers to 1 :
/* 177 */ int numReduceTasks = this.job.getNumReduceTasks();
/* 178 */ if ((numReduceTasks > 1) || (numReduceTasks < 0))
/* */ {
/* 180 */ numReduceTasks = 1;
/* 181 */ this.job.setNumReduceTasks(1);
/* */ }
/* 183 */ outputCommitter.setupJob(jContext);
/* 184 */ this.status.setSetupProgress(1.0F);
/* */
/* 186 */ Map mapOutputFiles = new HashMap();
/* */
Is this a fundamental limitation of the local mapreduce mode? what if
i need to write up a unit test that checks various partitioning
functions? Is there a workaround?
Also, i don't remember these problems when writing tests based on
local mapreduce in previous versions (this is cdh3b4) , although i
cannot be sure if i ran into exactly same situation before.
thanks.
-Dmitriy
