Hi, Would reducing the output from the map tasks solve the problem ? i.e. are reducers slowing down because a lot of data is being shuffled ?
If that's the case, you could see if the map output size will reduce by using the framework's combiner or an in-mapper combining technique. Thanks Hemanth On Wed, Oct 3, 2012 at 6:34 AM, Huanchen Zhang <iamzhan...@gmail.com> wrote: > Hello, > > I have a small portion of map tasks whose output is much larger than > others (more spills). So the reducer is mainly waiting for these a few map > tasks. Is there a good solution for this problem ? > > Thank you. > > Best, > Huanchen
