[
https://issues.apache.org/jira/browse/YARN-7612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300467#comment-16300467
]
Arun Suresh commented on YARN-7612:
-----------------------------------
Thanks for the review [~sunilg]
With regard to the PlacementConstraintManager comments (2 and 3 above): I just
put the stub here to validate and demonstrate the end-2-end workflow.
[~kkaranasos] will handle them in YARN-6596.
bq. I think enablePlacementConstraints could be application specific as well. I
think if we can get YARN-7666 in, we could use that envs to specify whether an
app needed to enable such constraints (more clearly to disable if needed).
True. Yup looks like YARN-7666 can help there (I just skimmed it - will look in
detail). But Currently, and to keep things less complicated, I just assumed the
app can control this with the presence or absence of the PlacementConstraint
mappings in the register call.
bq. NodeCandidateSelector is added in this patch. How different it is with
existing CandidateNodeSet impls. We are adding a MultiNodeSelector in
YARN-7494. and same could be used for all such complex node select strategies.
Could we make use of same.?
So, it maybe possible to converge at some point - but I find there is a
fundamental difference, atleast with regard to our use-case here. To clarify:
The NodeCandidateSelector here is basically a readOnly wrapper over the
clusterNodeTracker, and thus is a collection of all nodes. We would like the
Algorithm to decide how to filter the nodes using the NodeFilter which it will
pass to the NodeCandidateSelector - while the CandidateNodeSet is already
pre-filtered set of nodes.
That said - I am open to finding some way to converge. But lets not block this.
We can refactor later I feel.
bq. BatchedRequests is grouping a bunch of requests. Whats the criteria to
include requests to a batch? Are expecting more policies in this?..
So for some context: We realized while prototyping was that giving the
Algorithm a group of related requests at a time will lead to more optimal
placements. We initially tried timed batching - essentially batch all requests
from this app in this time window. But that, I feel is not very deterministic.
Instead - what we are proposing here is that, initially, a 'batch' would be all
SchedulingRequests received by the processor in 1 allocate call. That way, the
AM can provide some hint of required optimality of placements if it sends
related SchedulingRequests together in the same allocate call.
In the current patch, along with the above batching strategy, I also try to
batch all Scheduler Rejected requests of the same app by placementAttempt -
That way, the Algorithm can use this information to place all inputs
differently in the current run/place call if the placementAttempt > 1 (or can
have specific strategy if it hits a specific rejection count say: if
placementAttempt = 3, then just place on any node etc.)
bq. RM_PLACEMENT_RETRY_ATTEMPTS is at YARN level. Will it be more or less an
applications choice?
True, but initially I think lets just keep it a cluster configuration. It
should be trivial to add a retry value in the register/allocate call.
> Add Placement Processor Framework
> ---------------------------------
>
> Key: YARN-7612
> URL: https://issues.apache.org/jira/browse/YARN-7612
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Arun Suresh
> Assignee: Arun Suresh
> Attachments: YARN-7612-YARN-6592.001.patch,
> YARN-7612-YARN-6592.002.patch, YARN-7612-YARN-6592.003.patch,
> YARN-7612-YARN-6592.004.patch, YARN-7612-YARN-6592.005.patch,
> YARN-7612-YARN-6592.006.patch, YARN-7612-YARN-6592.007.patch,
> YARN-7612-YARN-6592.008.patch, YARN-7612-v2.wip.patch, YARN-7612.wip.patch
>
>
> This introduces a Placement Processor and a Planning algorithm framework to
> handle placement constraints and scheduling requests from an app and places
> them on nodes.
> The actual planning algorithm(s) will be handled in a YARN-7613.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
