Preemption is only available in Hadoop 0.20+ or in distributions of Hadoop that have applied that patch, such as Cloudera's distribution. If you are running one of these, check out http://hadoop.apache.org/mapreduce/docs/r0.21.0/fair_scheduler.html for information on how to enable preemption.
Matei On May 31, 2011, at 12:20 PM, Edward Capriolo wrote: > On Tue, May 31, 2011 at 2:50 PM, W.P. McNeill <bill...@gmail.com> wrote: > >> I'm launching long-running tasks on a cluster running the Fair Scheduler. >> As I understand it, the Fair Scheduler is preemptive. What I expect to see >> is that my long-running jobs sometimes get killed to make room for other >> people's jobs. This never happens instead my long-running jobs hog mapper >> and reducer slots and starve other people out. >> >> Am I misunderstanding how the Fair Scheduler works? >> > > Try adding > > <minSharePreemptionTimeout>120</minSharePreemptionTimeout> > <fairSharePreemptionTimeout>180</fairSharePreemptionTimeout> > > To one of your pools and see if that pool pre-empts other pools
