Sorry, I meant 0.21+, not 0.20+ for the Apache releases. Matei
On May 31, 2011, at 4:05 PM, Matei Zaharia wrote: > Preemption is only available in Hadoop 0.20+ or in distributions of Hadoop > that have applied that patch, such as Cloudera's distribution. If you are > running one of these, check out > http://hadoop.apache.org/mapreduce/docs/r0.21.0/fair_scheduler.html for > information on how to enable preemption. > > Matei > > On May 31, 2011, at 12:20 PM, Edward Capriolo wrote: > >> On Tue, May 31, 2011 at 2:50 PM, W.P. McNeill <bill...@gmail.com> wrote: >> >>> I'm launching long-running tasks on a cluster running the Fair Scheduler. >>> As I understand it, the Fair Scheduler is preemptive. What I expect to see >>> is that my long-running jobs sometimes get killed to make room for other >>> people's jobs. This never happens instead my long-running jobs hog mapper >>> and reducer slots and starve other people out. >>> >>> Am I misunderstanding how the Fair Scheduler works? >>> >> >> Try adding >> >> <minSharePreemptionTimeout>120</minSharePreemptionTimeout> >> <fairSharePreemptionTimeout>180</fairSharePreemptionTimeout> >> >> To one of your pools and see if that pool pre-empts other pools >
