[
https://issues.apache.org/jira/browse/YARN-3318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14487631#comment-14487631
]
Craig Welch commented on YARN-3318:
-----------------------------------
bq. ...Do we really see non-comparator based ordering-policy. We are
unnecessarily adding two abstractions - adding policies and comparators...
In the context of the code so far, the "comparator based" approach is specific
to compounding comparators to achieve functionality (priority + fifo, fair +
fifo, etc). This was the initial motivation for the two level api &
configuration, the broader surface of the policy which would allow for
different collection types, sorting on demand, etc, (the original policy) and
the narrower one within that (comparator) for the cases where comparator logic
was sufficient, e.g. sharing a collection (for composition) and a collection
type (a tree, for efficient resorting of individual elements when required) was
possible.
The two level api & configuration was not well received. Offline Wangda has
indicated that he thinks there are policies coming up which will need the
wider, initial api, with control over the collection, sorting, etc. Supporting
policy composition for those cases would be very awkward & is not really worth
pursuing.
The various competing tradeoffs, the aversion to a multilevel api, the need for
the higher level api, and the ability to compose policies creates something of
a tension, I don't think it's realistic to try and accomplish it all together,
the result will be Frankensteinian at best. Something has to go. Originally,
I chose the multilevel api to resolve the dilemma, I like that choice, it seems
unpopular with the crowd. Given that, the other "optional" dynamic is the
ability to compose policies (there's no requirement for either of these as far
as I can tell, it is a "bonus feature"). While I like the composition
approach, it can't be maintained as such with the broader api and without the
multilevel config/api. As one of these has to go and it appears it can't be
the broader api or the multilevel api I suppose it will have to be composition.
Internally there can be some composition of course, but it won't be
transparent/exposed/configurable as it was initially.
I'll put out a patch with that in a bit.
> Create Initial OrderingPolicy Framework and FifoOrderingPolicy
> --------------------------------------------------------------
>
> Key: YARN-3318
> URL: https://issues.apache.org/jira/browse/YARN-3318
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
> Reporter: Craig Welch
> Assignee: Craig Welch
> Attachments: YARN-3318.13.patch, YARN-3318.14.patch,
> YARN-3318.17.patch, YARN-3318.34.patch, YARN-3318.35.patch,
> YARN-3318.36.patch, YARN-3318.39.patch, YARN-3318.45.patch,
> YARN-3318.47.patch, YARN-3318.48.patch
>
>
> Create the initial framework required for using OrderingPolicies and an
> initial FifoOrderingPolicy
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
