Fair scheduler won't help unless you set it to allow preemptive executions which may not be a good thing...
Fair scheduler will wait until the current task completes before assigning a new task to the open slot. So if you have a long running job... You're SOL. A combiner will definitely help but you will still have the issue of long running jobs. You could put you job in a queue that limits the number of slots... But then you will definitely increase the time to run your job. If you could suspend a task... But that's anon-trivial solution... Sent from a remote device. Please excuse any typos... Mike Segel On May 18, 2011, at 5:04 PM, "W.P. McNeill" <bill...@gmail.com> wrote: > I'm using fair scheduler and JVM reuse. It is just plain a big job. > > I'm not using a combiner right now, but that's something to look at. > > What about bumping the mapred.reduce.tasks up to some huge number? I think > that shouldn't make a difference, but I'm hearing conflicting information on > this.
