On 07/20/2012 09:20 AM, Dave Shine wrote:
I believe this is referred to as a “key skew problem”, which I know is heavily dependent on the actual data being processed. Can anyone point me to any blog posts, white papers, etc. that might give me some options on how to deal with this issue?
I don't know about blog posts or white papers, but the canonical answer here is usually using a different Partitioner.
The default one takes the .hash() of each Mapper output key and reduces it modulo the number of Reducers you've specified (43, here). So the first place I'd look is to see if there's some reason you're getting so many more outputs with one key-hash-mod-43 than the others.
A common answer here is that one key alone has a huge number of outputs, in which case it's hard to do anything better with it. Another case is that your key class' hash function is bad at telling apart a certain class of keys that occur with some regularity. Since 43 is an odd prime, I would not expect a moderately evenly distributed hash to suddenly get spikes at certain values mod-43.
So if you want to (and can) rejigger your hashes to spread things more evenly, great. If not, you're down to writing your own partitioner. It's slightly different depending on which API you're using, but either way you basically have to write a function called getPartition that takes a mapper output record (key and value) and the number of reducers and returns the index (from 0 to numReducers-1) of the reducer that should handle that record. And unless you REALLY know what you're doing, the function should probably only depend on the key.
Good luck.
