Re: Intermediary Data on Fair Scheduler

2009-08-13 Thread Todd Lipcon
Hi Mithila, I assume you're referring to fair scheduler preemption. In the preemption scenario, tasks are completely killed, not paused. It's not like a preemptive scheduler in your OS where things are context switched. This is why the preemption is not enabled by default and has tuning

Re: Intermediary Data on Fair Scheduler

2009-08-13 Thread Mithila Nagendra
Hi Todd So does this mean that when two jobs are assigned to a pool, where one job has 1 map task and 1 reduce task, whereas the other has 5 map and 5 reduce tasks, how will the switch between these jobs take place? Lets say the scheduler starts with the bigger job, runs 1 map task, when it

Re: Intermediary Data on Fair Scheduler

2009-08-13 Thread Todd Lipcon
On Thu, Aug 13, 2009 at 11:32 AM, Mithila Nagendra mnage...@asu.edu wrote: Hi Todd So does this mean that when two jobs are assigned to a pool, where one job has 1 map task and 1 reduce task, whereas the other has 5 map and 5 reduce tasks, how will the switch between these jobs take place?

Re: Intermediary Data on Fair Scheduler

2009-08-13 Thread Mithila Nagendra
This helps a lot! Thank you Todd. Best Regards Mithila On Thu, Aug 13, 2009 at 11:40 AM, Todd Lipcon t...@cloudera.com wrote: On Thu, Aug 13, 2009 at 11:32 AM, Mithila Nagendra mnage...@asu.edu wrote: Hi Todd So does this mean that when two jobs are assigned to a pool, where one job