Moving to mapreduce-user@, bcc gene...@. Please do not use the
general@ list for project specific discussions.
On Jul 21, 2010, at 10:15 AM, Syed Wasti wrote:
It says “:In M/R job Reducers are initialized with Mappers at the
job initialization, but the reduce method is called in reduce phase
when all the maps had been finished. So in large jobs where Reducer
loads data (>100 MB for business logic) in-memory on initialization,
the performance can be increased by lazily initializing Reducers
i.e. loading data in reduce method controlled by an initialize flag
variable which assures that it is loaded only once. By lazily
initializing Reducers which require memory (for business logic) on
initialization, number of maps can be increased.”
The part about 'loading data in reduce method controlled by an
initialize flag variable which assures that it is loaded only once'
makes no sense to me.
However, you can 'slowstart' reduces by ensuring sufficient maps are
complete before _any_ reduces are launched... from mapred-default.xml:
<property>
<name>mapred.reduce.slowstart.completed.maps</name>
<value>0.05</value>
<description>Fraction of the number of maps in the job which should
be
complete before reduces are scheduled for the job.
</description>
</property>
Arun
