Hi Wang,
In Hama, every task has same category and priority. You would have already
observed that GroomServer acts as a container/observer for every BSP task.
Every GroomServer reports the status of itself and these tasks to
BSPMaster. When GroomServer comes up, it is configured to run a particular
number of tasks on the machine('bsp.tasks.maximum' property mentioned by
Edward above) . So a scheduler has to decide the best choice of
GroomServers with the available slots to allocate the tasks in a job.
In the previous code refactor, we added
org.apache.hama.bsp.taskallocation.TaskAllocationStrategy.
You may modify this interface for the purpose. The idea is to make it
pluggable. Currently we have something that does a best effort attempt for
data locality. However, bsp solutions are heavy in messaging. Hence, we
have not yet handled the task group locality along with data locality; in
simple words, the tasks in a job should preferrably be scheduled in the
same rack instead of separate ones.
The other aspects where you want to define the resources (CPU, RAM,
(GPU/x86) ) per node, you can make them a property for GroomServer to
report to BSPMaster and the scheduler can decide to allocate resources
based on a global view of cluster. You may also look into YARN or Mesos
before proceeding.
Hope it helps :)
Thanks,
Suraj
On Wed, Apr 10, 2013 at 5:09 AM, Edward J. Yoon <[email protected]>wrote:
> If I understand/remember 'Fair Scheduler' correctly, it adjusts the
> balance of the multiple MR jobs by killing tasks (changing to peding
> status).
>
> In Hama BSP case, I don't have any idea yet. Let's think about it together.
>
> On Wed, Apr 10, 2013 at 5:54 PM, Shuo Wang <[email protected]>
> wrote:
> > If I want to implement the fair scheduler in Hama, how can I allocate the
> > resources between different jobs.I know that hadoop use slot to represent
> > the resource, it actually allocates the slots.
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>
