Hello Marek, I think you can increase number of reducers for your MR job so as to reduce the amount of intermediate key-value pairs assigned to each reducer. Note also that the number of reducers is dependent on your job and how the output should be produced.
On Mon, Feb 6, 2012 at 11:37 AM, Marek Miglinski <mmiglin...@seven.com>wrote: > Hey, > > I have a mapreduce job (transactions loader) and the main problem of it is > "reduce->copy" and "reduce->sort" phase which takes all IO and uses all > disk resources, what are the possible ways to reduce this load? My cloud > settings are: > > ioSortFactor=80 > ioSortMb=800 > (mapredChildJavaOpts=Xmx1152m) > > I can lower those settings, what else can I tweak? > > > Thanks, > Marek M. > -- Best Regards, Mostafa Ead
