[
https://issues.apache.org/jira/browse/YARN-7612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16284608#comment-16284608
]
Wangda Tan commented on YARN-7612:
----------------------------------
Thanks [~asuresh] for working on the patch.
Here's my overall understanding of the patch:
a. AM requests SchedulingRequest.
b. PlacementProcessor intercepts AM's request, send SchedulingRequests to
Algorithm (b1) and set RejectedRequest (b2) to AMResponse.
b1. Algorithm generates PlacedRequest (proposal) and rejected requests, and
PlacementProcessor will process responses from Algorithm.
In general, I think the workflow looks good. PlacementProcessor is the true
global scheduler and algorithm is the pluggable module, and common logic (like
how to handle proposal / rejected requests) are handled by PlacementProcessor
itself.
There're a couple of high-level suggestions regarding API design.
1. Algorithm interface is not the true global scheduler: the BatchedRequests is
requesting for one app. Ideally we should be able to assign containers based on
requests from multiple apps, correct?
2. In addition to that, it's better not to pass
tagsManager/constraintsManager/node selector to Algorithm. Instead, we should
have a separate init method in Algorithm API to store these util classes.
3. The interaction between Algorithm & PlacementProcessor is too tightly
coupled: Processor called algorithm once and get a response, in the response it
includes attempt id, etc. (I'm not quite sure why it needs the attempt id,
could you explain a bit?).
I would suggest considering following Algorithm API option:
1. The algorithm holds reference to scheduler states (including scheduler
itself, AllocationTagsManager, ConstraintsManager, etc.)
2. The algorithm makes decision truly based on scheduler state (no additional
inputs).
3. The result of Algorithm can be pulled by Processor or alternatively, notify
Processor. I would prefer the latter one, which Algorithm can have a
PlacementResultNotifier reference (passed in init method). And if any placement
or reject decision is made, Algorithm will invoke the PlacementResultNotifer.
> Add Placement Processor and planner framework
> ---------------------------------------------
>
> Key: YARN-7612
> URL: https://issues.apache.org/jira/browse/YARN-7612
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Arun Suresh
> Assignee: Arun Suresh
> Attachments: YARN-7612-YARN-6592.001.patch,
> YARN-7612-YARN-6592.002.patch, YARN-7612-v2.wip.patch, YARN-7612.wip.patch
>
>
> This introduces a Placement Processor and a Planning algorithm framework to
> handle placement constraints and scheduling requests from an app and places
> them on nodes.
> The actual planning algorithm(s) will be handled in a YARN-7613.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
