Hello,
I am slightly confused about the number of reducers executed and the
size of data each receives.
Setup:
I have a setup of 5 task trackers.
In my hadoop-site:
(1)
<property>
<name>mapred.reduce.tasks</name>
<value>7</value>
<description>The default number of reduce tasks per job. Typically
set
to a prime close to the number of available hosts. Ignored when
mapred.job.tracker is "local".
</description>
</property>
(2)
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<value>7</value>
<description>The maximum number of map tasks that will be run
simultaneously by a task tracker.
</description>
</property>
(3)
However from http://hadoop.apache.org/core/docs/r0.18.1/api/index.html
"The total number of partitions is the same as the number of reduce
tasks for the job.."
Q:
So does that mean (from (1) & (2)) , there will be a total of 7 reduce
tasks distributed across 5 machines such that
no machine receives more than 7 reduce jobs?
If so, suppose i have millions of unique keys which need to be
reduced(e.g urls/hashes), these will be partitioned into 7 groups
(from (3))
and distributed across 5 machines?
Which is equivalent to saying that the number of reduces tasks run
across all machines will be equal to 7?
Wouldn't that be too large a number of keys for each reduce task?
Are these possible solutions:
Solution:
I) Fixed machines (5), but increase mapred.reduce.tasks (loss in
performance?)
2) Increase number of machines (not possible for me, but a theoretical
solution) and set mapred.reduce.tasks to a commensurate number
Many thanks for your time
Saptarshi
Saptarshi Guha | [EMAIL PROTECTED] | http://www.stat.purdue.edu/~sguha
More people are flattered into virtue than bullied out of vice.
-- R. S. Surtees
