[
https://issues.apache.org/jira/browse/YARN-7612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16301671#comment-16301671
]
Arun Suresh commented on YARN-7612:
-----------------------------------
Thanks for the review [~sunilg]
bq. I had one smaller concern here. I think its better to handle a duplicate
check provided yarn.resourcemanager.application-master-service.processors could
be also configured with new PlacementProcessor
Ah... good point. Will add a check in the next patch to ensure user cannot add
it.
bq. So all rejectedRequests are going in one batch. This seems fine. But in
case where we have too many failures on placement, do you see a corner case
where number of requests in batch is bigger.
Yup - that could happen if say the app sends a SchedReq with numAllocations
very large (with anti-affinity but num nodes in cluster much smaller). I was
planning on bunching SchedulingRequests with the same porperties (priority,
allocationReqId, ExecType Resources etc.) by doing a numAllocation increment
and retaining the same SchedulerRequest object. I was hoping to handle all
these efficiency improvements in a separate JIRA (as am sure more will pop up
once we start doing scalability tests)
bq. In PlacementProcessor, schedulePlacedRequests is invoked after
reDispatchRetryableRequests. So * requestsToRetry* will be mostly cleaned up,
correct?. In that case, does addToRetryList need to handle it again?
That should be OK right ? if it gets cleared up, then it will be picked up in
the next allocate call.
Regarding the code snippet in the SampleAlgorithm - Yeah, it looks confusing.
But like I mentioned, the Sample algorithm is just to verify end-2-end not to
be used in production - And it makes certain assumptions about the
PlacementConstraint. I've added a comment there. Maybe I will just move it to
the test package and use it for the tests.
> Add Placement Processor Framework
> ---------------------------------
>
> Key: YARN-7612
> URL: https://issues.apache.org/jira/browse/YARN-7612
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Arun Suresh
> Assignee: Arun Suresh
> Attachments: YARN-7612-YARN-6592.001.patch,
> YARN-7612-YARN-6592.002.patch, YARN-7612-YARN-6592.003.patch,
> YARN-7612-YARN-6592.004.patch, YARN-7612-YARN-6592.005.patch,
> YARN-7612-YARN-6592.006.patch, YARN-7612-YARN-6592.007.patch,
> YARN-7612-YARN-6592.008.patch, YARN-7612-YARN-6592.009.patch,
> YARN-7612-YARN-6592.010.patch, YARN-7612-v2.wip.patch, YARN-7612.wip.patch
>
>
> This introduces a Placement Processor and a Planning algorithm framework to
> handle placement constraints and scheduling requests from an app and places
> them on nodes.
> The actual planning algorithm(s) will be handled in a YARN-7613.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
