yes. that is a general solution to control counts of output files.
however, if you need to control counts of outputs dynamically, how could
you do?
if an output file name is 'A', counts of this output files are needed to
be 5.
if an output file name is 'B', counts of this output files are
You can control the number of reducers by calling
job.setNumReduceTasks() before you launch it.
-Joey
On Thu, May 12, 2011 at 6:33 PM, Jun Young Kim juneng...@gmail.com wrote:
yes. that is a general solution to control counts of output files.
however, if you need to control counts of outputs
hi, all.
I have 60 reducers which are generating same output files.
from output-r--1 to output-r-00059.
under this situation, I want to control the count of output files.
for example, is it possible to concatenate all output files to 10 ?
from output-r-1 to output-r-00010.
thanks
Short, blind answer: You could run 10 reducers.
Otherwise, you'll have to run another job that picks up a few files
each in mapper and merges them out. But having 60 files shouldn't
really be a problem if they are sufficiently large (at least 80% of a
block size perhaps -- you can tune # of
