[
https://issues.apache.org/jira/browse/YARN-11736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17907516#comment-17907516
]
ASF GitHub Bot commented on YARN-11736:
---------------------------------------
TaoYang526 commented on PR #7121:
URL: https://github.com/apache/hadoop/pull/7121#issuecomment-2557882266
I have just committed optimized code to help this policy achieving high
scheduling quality and meanwhile ensuring high performance with cache enabled.
There are 2 important improvements:
- Introduce PreferredIterator with next-fit policy to avoid always choosing
the first node when cache enabled.
- Separate sorted nodes to 3 zones: preferred, normal and ignored to achieve
high scheduling quality, front nodes in preferred zone will have more
opportunities to be chosen then nodes in normal zone, and rear nodes in ignored
zone can be ignored in the scheduling process for now to avoid make things
worse.
> Enhance MultiNodeLookupPolicy to allow configuration of extended comparators
> for better usability.
> --------------------------------------------------------------------------------------------------
>
> Key: YARN-11736
> URL: https://issues.apache.org/jira/browse/YARN-11736
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: multi-node-placement
> Reporter: Tao Yang
> Assignee: Tao Yang
> Priority: Major
> Labels: pull-request-available
>
> Currently when multi-nodes is enabled, there is only 1 implementation of
> _MultiNodeLookupPolicy_ interface: {_}ResourceUsageMultiNodeLookupPolicy{_},
> which will sort nodes by allocated resources in ascending order.
> If cluster has nodes with different resource-spec, the resource utilization
> of smaller nodes will be significantly high, while larger nodes will
> experience low resource utilization. This may rise the hotspot risk and
> reduce the scheduling effectiveness.
> So I propose to add a new policy called _MultiComparatorPolicy_ to meet
> requirements from some complex scenarios, which should contains serveral
> inherit comparators and can be extended later, and supports configuring
> specified comparators for different policy instances.
>
> {*}Implementation Details{*}:
> 1. _MultiNodeSorter#initPolicy_ will pass the policyConf which is cloned from
> scheduler configuration and attached the name of current policy, so that we
> can fetch the specified configuration for this policy inside the
> implementations of MultiNodeLookupPolicy.
> 2. new implementation of {_}MultiNodeLookupPolicy{_}: _MultiComparatorPolicy_
> 2.1) contains several inherit comparators and can be extendable later.
> comparator keys: _ALLOCATED_RESOURCE / UNALLOCATED_RESOURCE /
> DOMINANT_RESOURCE_RATIO / NODE_ID,_ order-directions: _ASC / DESC._
> 2.2) supports configuring specified comparators with
> order-direction(ASC/DESC) for different policy instances via conf-key:
> {+}{color:#172b4d}yarn.scheduler.capacity.multi-node-sorting.policy.<policy-name>.comparators{color}{+},
> value format is
> "<comparator_key_1>[:<order_direction_1>],<comparator_key_2>[:<order_direction_2>],...".
> For example, "DOMINANT_ALLOCATED_RATIO,NODE_ID:DESC" means that for policy
> test, nodes should be sorted by dominant-resource-ratio in ascending order,
> by nodeID desc in descending order.
> 3. Refactor variable names in
> AbstractCSQueue/CSQueue/FiCaSchedulerApp/AppPlacementAllocator after
> supporting multiple policy instances with the same policy class.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
