[
https://issues.apache.org/jira/browse/YARN-11000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17737533#comment-17737533
]
ASF GitHub Bot commented on YARN-11000:
---------------------------------------
tomicooler opened a new pull request, #5783:
URL: https://github.com/apache/hadoop/pull/5783
### Description of PR
The new queue resource calculation logic is capable of calculating the queue
resources based on mixed resource types. The queues under one parent can also
mix their modes (e.g.: root.queue1=percentage root.queue2=weight
root.queue3=absolute).
Precedence is: absolute -> percentage -> weight.
Note that the calculation is done by a single resource type at once.
**Design doc:**
[YARN-10888](https://issues.apache.org/jira/browse/YARN-10888)
**Example**
```
yarn.scheduler.capacity.root.default.capacity = [memory=1w,
vcores=4]
yarn.scheduler.capacity.root.test_1.capacity =
[memory=16384, vcores=100%]
yarn.scheduler.capacity.root.test_2.capacity = [memory=3w,
vcores=12]
yarn.scheduler.capacity.root.test_1.test_1_1.capacity = [memory=1w,
vcores=1w]
yarn.scheduler.capacity.root.test_1.test_1_2.capacity = [memory=50%,
vcores=2]
yarn.scheduler.capacity.root.test_1.test_1_3.capacity = [memory=12288,
vcores=86%]
```
Further examples:
`hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySchedulerMixedMode.java`
The feature is behind a **feature flag**, it is disabled by default.
```
yarn.scheduler.capacity.legacy-queue-mode.enabled = true/false
```
Note that the mixed mode is not yet supported for AQCv2 (or v1) templates.
I'll file a Jira ticket for that.
### Key changes
1. The **capacity** and **absoluteCapacity** (and **maxApplications** and
other derived properties) of a queue is calculated from the
**effectiveMinResource** (= the current available resource for a given queue
based on the queue hierarchy configuration and the available cluster resource).
2. When there is no cluster resources available yet, the maxApplications are
set to the configured values, so applications can be submitted.
3. Changes in the scheduler-response: the capacity and absoluteCapacity
resembles the effective values based on the queue configs and cluster
resources. The normalizedWeight is not filled in the scheduler-response when
queues are in weight mode.
### How was this patch tested?
- I'll run all the unit test in both Legacy and the "Uniform" queue
calculation mode in this PR.
- I'll create a follow-up ticket to run some e2e like automatically on both
mode.
### Acknowledges
The current state of this PR was a team effort, thanks to:
@9uapaw
@brumi1024
@K0K0V0K
### For code changes:
- [x] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'YARN-11000. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
> Replace queue resource calculation logic in updateClusterResource
> -----------------------------------------------------------------
>
> Key: YARN-11000
> URL: https://issues.apache.org/jira/browse/YARN-11000
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
> Reporter: Andras Gyori
> Assignee: Andras Gyori
> Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> YARN-10965 introduces a brand new queue calculation system. In order to
> simplify the review process, this issue replaces the current logic with the
> newly introduced one.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
