[
https://issues.apache.org/jira/browse/YARN-11736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Yang updated YARN-11736:
----------------------------
Description:
Currently when multi-nodes is enabled, there is only 1 implementation of
_MultiNodeLookupPolicy_ interface: {_}ResourceUsageMultiNodeLookupPolicy{_},
which will sort nodes by allocated resources in ascending order.
If cluster has nodes with different resource-spec, the resource utilization of
smaller nodes will be significantly high, while larger nodes will experience
low resource utilization. This may rise the hotspot risk and reduce the
scheduling effectiveness.
So I propose to add a new policy called _MultiComparatorPolicy_ to meet
requirements from some complex scenarios, which should contains serveral
inherit comparators and can be extended later, and supports configuring
specified comparators for different policy instances.
{*}Implementation Details{*}:
1. _MultiNodeSorter#initPolicy_ will pass the policyConf which is cloned from
scheduler configuration and attached the name of current policy, so that we can
fetch the specified configuration for this policy inside the implementations of
MultiNodeLookupPolicy.
2. new implementation of {_}MultiNodeLookupPolicy{_}: _MultiComparatorPolicy_
2.1) contains several inherit comparators and can be extendable later.
comparator keys: _ALLOCATED_RESOURCE / UNALLOCATED_RESOURCE /
DOMINANT_RESOURCE_RATIO / NODE_ID,_ order-directions: _ASC / DESC._
2.2) supports configuring specified comparators with
order-direction(ASC/DESC) for different policy instances via conf-key:
{+}{color:#172b4d}yarn.scheduler.capacity.multi-node-sorting.policy.<policy-name>.comparators{color}{+},
value format is
"<comparator_key_1>[:<order_direction_1>],<comparator_key_2>[:<order_direction_2>],...".
For example, "DOMINANT_ALLOCATED_RATIO,NODE_ID:DESC" means that for policy
test, nodes should be sorted by dominant-resource-ratio in ascending order, by
nodeID desc in descending order.
3. Refactor variable names in
AbstractCSQueue/CSQueue/FiCaSchedulerApp/AppPlacementAllocator after supporting
multiple policy instances with the same policy class.
was:
Currently when multi-nodes is enabled, there is only 1 implementation of
_MultiNodeLookupPolicy_ interface: {_}ResourceUsageMultiNodeLookupPolicy{_},
which will sort nodes by allocated resources in ascending order.
If cluster has nodes with different resource-spec, the resource utilization of
smaller nodes will be significantly high, while larger nodes will experience
low resource utilization. This may rise the hotspot risk and reduce the
scheduling effectiveness.
So I propose to add a new policy called _MultiComparatorPolicy_ to meet
requirements from some complex scenarios, which should contains serveral
inherit comparators and can be extended later, and supports configuring
specified comparators for different policy instances.
{*}Implementation Details{*}:
1. _MultiNodeSorter#initPolicy_ will pass the policyConf which is cloned from
scheduler configuration and attached the name of current policy, so that we can
fetch the specified configuration for this policy inside the implementations of
MultiNodeLookupPolicy.
2. new implementation of {_}MultiNodeLookupPolicy{_}: _MultiComparatorPolicy_
2.1) contains several inherit comparators and can be extendable later.
comparator keys: _ALLOCATED_RESOURCE / UNALLOCATED_RESOURCE /
DOMINANT_RESOURCE_RATIO / NODE_ID,_ order-directions: _ASC / DESC._
2.2) supports configuring specified comparators with
order-direction(ASC/DESC) for different policy instances via conf-key:
{+}{color:#172b4d}yarn.scheduler.capacity.multi-node-sorting.policy.<policy-name>.comparators{color}{+},
value format is
"<comparator_key_1>[:<order_direction_1>],<comparator_key_2>[:<order_direction_2>],...".
For example, "DOMINANT_ALLOCATED_RATIO,NODE_ID:DESC" means that for policy
test, nodes should be sorted by dominant-resource-ratio in ascending order, by
nodeID desc in descending order.
3. Refactor variable names in after supporting multiple policy instances with
the same policy class.
> Enhance MultiNodeLookupPolicy to allow configuration of extended comparators
> for better usability.
> --------------------------------------------------------------------------------------------------
>
> Key: YARN-11736
> URL: https://issues.apache.org/jira/browse/YARN-11736
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: multi-node-placement
> Reporter: Tao Yang
> Assignee: Tao Yang
> Priority: Major
> Labels: pull-request-available
>
> Currently when multi-nodes is enabled, there is only 1 implementation of
> _MultiNodeLookupPolicy_ interface: {_}ResourceUsageMultiNodeLookupPolicy{_},
> which will sort nodes by allocated resources in ascending order.
> If cluster has nodes with different resource-spec, the resource utilization
> of smaller nodes will be significantly high, while larger nodes will
> experience low resource utilization. This may rise the hotspot risk and
> reduce the scheduling effectiveness.
> So I propose to add a new policy called _MultiComparatorPolicy_ to meet
> requirements from some complex scenarios, which should contains serveral
> inherit comparators and can be extended later, and supports configuring
> specified comparators for different policy instances.
>
> {*}Implementation Details{*}:
> 1. _MultiNodeSorter#initPolicy_ will pass the policyConf which is cloned from
> scheduler configuration and attached the name of current policy, so that we
> can fetch the specified configuration for this policy inside the
> implementations of MultiNodeLookupPolicy.
> 2. new implementation of {_}MultiNodeLookupPolicy{_}: _MultiComparatorPolicy_
> 2.1) contains several inherit comparators and can be extendable later.
> comparator keys: _ALLOCATED_RESOURCE / UNALLOCATED_RESOURCE /
> DOMINANT_RESOURCE_RATIO / NODE_ID,_ order-directions: _ASC / DESC._
> 2.2) supports configuring specified comparators with
> order-direction(ASC/DESC) for different policy instances via conf-key:
> {+}{color:#172b4d}yarn.scheduler.capacity.multi-node-sorting.policy.<policy-name>.comparators{color}{+},
> value format is
> "<comparator_key_1>[:<order_direction_1>],<comparator_key_2>[:<order_direction_2>],...".
> For example, "DOMINANT_ALLOCATED_RATIO,NODE_ID:DESC" means that for policy
> test, nodes should be sorted by dominant-resource-ratio in ascending order,
> by nodeID desc in descending order.
> 3. Refactor variable names in
> AbstractCSQueue/CSQueue/FiCaSchedulerApp/AppPlacementAllocator after
> supporting multiple policy instances with the same policy class.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
