On Nov 13, 2008, at 4:23 PM, Arun C Murthy wrote:


On Nov 13, 2008, at 1:59 PM, Brian Bockelman wrote:

Hey all,

When we run a large task (lots of intermediate output), we have a real problem with it "beating up" nodes during the shuffle phase. About half of our nodes are completely overwhelmed by the number of reducers requesting data to be copied (somewhere between 50 and 75 files appear to be open); this causes wait_io on the node to go up to about 50-75% of the CPU. The number of active threads is about 45. The amount of intermediate data is about 10GB per node.

Anyone else run into this problem? Any hints? My first thought is to ask the user to use a combiner so there are less files to work with in the first place.


Combiners are useful, they help in cutting down the amount of data which needs to be shuffled.


I poked around the user's code, and he indeed had 36 billion outputs, and no real need to not use a Combiner. I'll sit and have a good talk with him tomorrow...

Have you tried compressing map-outputs?
http://hadoop.apache.org/core/docs/current/mapred_tutorial.html#Data+Compression

A heavy-handed option is to cut down the no. of per-reducer threads which do the shuffle: mapred.reduce.parallel.copies. We recommend ~20, try with a lesser number than that...


Ah, I might think about this one. What's the thinking behind the ~20 number?

Brian

Reply via email to