jiang licht wrote:
Yeah, will give it a try :)
Thanks,
-Michael
Hadoop 0.21 has the buy-cpu-time scheduler too, which does have
preemption and is apparently simpler than the main schedulers, which
Yahoo! and facebook use. Given the performance of the main schedulers
matters a lot to those two companies, they will be nervous about any
patch which could impact their performance. Starting off with the
simpler code (and using the SVN_TRUNK scheduler plugin API) would be a
good idea.
Any documentation you can make on writing/testing schedulers would be
nice too, its a good plugin point, and datacentre scheduling is right up
there with VM and data placement as cutting edge problems
--- On Mon, 6/28/10, He Chen <[email protected]> wrote:
From: He Chen <[email protected]>
Subject: Re: hybrid map/reducer scheduler?
To: [email protected]
Date: Monday, June 28, 2010, 7:09 PM
You can write your own one based on them. They are open source.
On Mon, Jun 28, 2010 at 6:13 PM, jiang licht <[email protected]> wrote:
In addition to default FIFO scheduler, there are fair scheduler and
capacity scheduler. In some sense, fair scheduler can be considered a
user-based scheduling while capacity scheduler does a queue-based
scheduling. Is there or will there be a hybrid scheduler that combines the
good parts of the two (or a capacity scheduler that allows preemption, then
different users are asked to submit jobs to different queues, in this way
implicitly follow user-based scheduling as well, more or less)?
Thanks,
--Michael
