[
https://issues.apache.org/jira/browse/YARN-11736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17890843#comment-17890843
]
ASF GitHub Bot commented on YARN-11736:
---------------------------------------
TaoYang526 commented on code in PR #7121:
URL: https://github.com/apache/hadoop/pull/7121#discussion_r1806055092
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/placement/MultiNodeSorter.java:
##########
@@ -74,23 +75,30 @@ public synchronized MultiNodeLookupPolicy<N>
getMultiNodeLookupPolicy() {
}
public void serviceInit(Configuration conf) throws Exception {
- LOG.info("Initializing MultiNodeSorter=" + policySpec.getPolicyClassName()
+ LOG.info("Initializing MultiNodeSorter policyName=" +
policySpec.getPolicyName()
+ + ", policyClassName=" + policySpec.getPolicyClassName()
+ ", with sorting interval=" + policySpec.getSortingInterval());
- initPolicy(policySpec.getPolicyClassName());
+ initPolicy(policySpec);
super.serviceInit(conf);
}
@SuppressWarnings("unchecked")
- void initPolicy(String policyName) throws YarnException {
+ void initPolicy(MultiNodePolicySpec policySpec) throws YarnException {
+ String policyName = policySpec.getPolicyName();
+ String policyClassName = policySpec.getPolicyClassName();
Class<?> policyClass;
try {
- policyClass = Class.forName(policyName);
+ policyClass = Class.forName(policyClassName);
} catch (ClassNotFoundException e) {
throw new YarnException(
- "Invalid policy name:" + policyName + e.getMessage());
+ "Invalid policy class name:" + policyClassName + e.getMessage());
}
+ Configuration policyConf = new Configuration(this.getConfig());
Review Comment:
In MultiNodeSortingManager#createAllPolicies, we can see all the
MultiNodeSorter instances owns a shared config, policyName will be set in
policyConf, which is a instance-level configuration, so that policyInstance can
get the configurations belong to itself.
> Enhance MultiNodeLookupPolicy to allow configuration of extended comparators
> for better usability.
> --------------------------------------------------------------------------------------------------
>
> Key: YARN-11736
> URL: https://issues.apache.org/jira/browse/YARN-11736
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: multi-node-placement
> Reporter: Tao Yang
> Assignee: Tao Yang
> Priority: Major
> Labels: pull-request-available
>
> Currently when multi-nodes is enabled, there is only 1 implementation of
> _MultiNodeLookupPolicy_ interface: {_}ResourceUsageMultiNodeLookupPolicy{_},
> which will sort nodes by allocated resources in ascending order.
> If cluster has nodes with different resource-spec, the resource utilization
> of smaller nodes will be significantly high, while larger nodes will
> experience low resource utilization. This may rise the hotspot risk and
> reduce the scheduling effectiveness.
> So I propose to add a new policy called _MultiComparatorPolicy_ to meet
> requirements from some complex scenarios, which should contains serveral
> inherit comparators and can be extended later, and supports configuring
> specified comparators for different policy instances.
>
> {*}Implementation Details{*}:
> 1. _MultiNodeSorter#initPolicy_ will pass the policyConf which is cloned from
> scheduler configuration and attached the name of current policy, so that we
> can fetch the specified configuration for this policy inside the
> implementations of MultiNodeLookupPolicy.
> 2. new implementation of {_}MultiNodeLookupPolicy{_}: _MultiComparatorPolicy_
> 2.1) contains several inherit comparators and can be extendable later.
> comparator keys: _ALLOCATED_RESOURCE / UNALLOCATED_RESOURCE /
> DOMINANT_RESOURCE_RATIO / NODE_ID,_ order-directions: _ASC / DESC._
> 2.2) supports configuring specified comparators with
> order-direction(ASC/DESC) for different policy instances via conf-key:
> {+}{color:#172b4d}yarn.scheduler.capacity.multi-node-sorting.policy.<policy-name>.comparators{color}{+},
> value format is
> "<comparator_key_1>[:<order_direction_1>],<comparator_key_2>[:<order_direction_2>],...".
> For example, "DOMINANT_ALLOCATED_RATIO,NODE_ID:DESC" means that for policy
> test, nodes should be sorted by dominant-resource-ratio in ascending order,
> by nodeID desc in descending order.
> 3. Refactor variable names in
> AbstractCSQueue/CSQueue/FiCaSchedulerApp/AppPlacementAllocator after
> supporting multiple policy instances with the same policy class.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
