[
https://issues.apache.org/jira/browse/YARN-10380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhuqi updated YARN-10380:
-------------------------
Attachment: (was: YARN-10380.001.patch)
> Import logic of multi-node allocation in CapacityScheduler
> ----------------------------------------------------------
>
> Key: YARN-10380
> URL: https://issues.apache.org/jira/browse/YARN-10380
> Project: Hadoop YARN
> Issue Type: Sub-task
> Affects Versions: 3.3.0
> Reporter: Wangda Tan
> Assignee: zhuqi
> Priority: Critical
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> *1) Entry point:*
> When we do multi-node allocation, we're using the same logic of async
> scheduling:
> {code:java}
> // Allocate containers of node [start, end)
> for (FiCaSchedulerNode node : nodes) {
> if (current++ >= start) {
> if (shouldSkipNodeSchedule(node, cs, printSkipedNodeLogging)) {
> continue;
> }
> cs.allocateContainersToNode(node.getNodeID(), false);
> }
> } {code}
> Is it the most effective way to do multi-node scheduling? Should we allocate
> based on partitions? In above logic, if we have thousands of node in one
> partition, we will repeatly access all nodes of the partition thousands of
> times.
> I would suggest looking at making entry-point for node-heartbeat,
> async-scheduling (single node), and async-scheduling (multi-node) to be
> different.
> Node-heartbeat and async-scheduling (single node) can be still similar and
> share most of the code.
> async-scheduling (multi-node): should iterate partition first, using pseudo
> code like:
> {code:java}
> for (partition : all partitions) {
> allocateContainersOnMultiNodes(getCandidate(partition))
> } {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
