[
https://issues.apache.org/jira/browse/HADOOP-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12672186#action_12672186
]
Vivek Ratan commented on HADOOP-5199:
-------------------------------------
It may not be clear from what I've written so far, but I'm *not* proposing yet
another scheduler. I'm suggesting that there is a large amount of common
code/functionality in the three existing schedulers today, which we can combine
into one common base scheduler (which I refer to as the 'Hadoop Scheduler').
Features such as detecting how many M/R tasks to allocate in one heartbeat,
pools and capacities, getting one or more node-local map tasks, checking for
user limits and memory requirements, etc. are all common functionality that
this base scheduler can implement in one place. At the same time, I'm
suggesting that we have two different ways of ordering jobs (one matching what
Fairshare does, the other similar to the default and Capacity schedulers). This
design (the base scheduler and the two ways of ordering jobs) is intended to
replace _contrib/capacity-scheduler_ and _contrib/fairscheduler_ while making
sure we support as much of existing functionality as possible. Putting common
code in one place will help efforts such as global scheduling and others that
impact all schedulers, as there is only one set of code to fix.
We've had some discussions in the past, across various Jiras, to consolidate
common code/functionality and this is an attempt to do so.
> A proposal to merge common functionality of various Schedulers
> --------------------------------------------------------------
>
> Key: HADOOP-5199
> URL: https://issues.apache.org/jira/browse/HADOOP-5199
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Vivek Ratan
> Attachments: 5199.1.patch
>
>
> There are at least 3 Schedulers in Hadoop today: Default, Capacity, and
> Fairshare. Over time, we're seeing a lot of functionality common to all
> three. Many bug fixes, improvements to existing functionality, and new
> functionality are applicable to all three schedulers. This trend seems to be
> getting stronger, as we notice similar problems, solutions, and ideas. This
> is a proposal to detect and consolidate such common functionality.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
