[jira] [Commented] (YARN-11633) [Federation] Improve LoadBasedRouterPolicy To Use Available vcores
[ https://issues.apache.org/jira/browse/YARN-11633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17801961#comment-17801961 ] ASF GitHub Bot commented on YARN-11633: --- slfan1989 commented on PR #6356: URL: https://github.com/apache/hadoop/pull/6356#issuecomment-1874691119 > @slfan1989 Thank you for your reply, I will close this PR. Where can I contact you, such as wechat, and I will ask you some questions. Sorry for the late reply, we can contact by email. @me below pr or jira, if I can understand the issue you describe, I will reply. > [Federation] Improve LoadBasedRouterPolicy To Use Available vcores > -- > > Key: YARN-11633 > URL: https://issues.apache.org/jira/browse/YARN-11633 > Project: Hadoop YARN > Issue Type: Improvement > Components: federation >Affects Versions: 3.3.6 >Reporter: yanbin.zhang >Priority: Major > Labels: pull-request-available > > When selecting a subcluster, consider not only available memory but also > available vcore -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11633) [Federation] Improve LoadBasedRouterPolicy To Use Available vcores
[ https://issues.apache.org/jira/browse/YARN-11633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799230#comment-17799230 ] ASF GitHub Bot commented on YARN-11633: --- singer-bin commented on PR #6356: URL: https://github.com/apache/hadoop/pull/6356#issuecomment-1865382265 @slfan1989 Thank you for your reply, I will close this PR. Where can I contact you, such as wechat, and I will ask you some questions. > [Federation] Improve LoadBasedRouterPolicy To Use Available vcores > -- > > Key: YARN-11633 > URL: https://issues.apache.org/jira/browse/YARN-11633 > Project: Hadoop YARN > Issue Type: Improvement > Components: federation >Affects Versions: 3.3.6 >Reporter: yanbin.zhang >Priority: Major > Labels: pull-request-available > > When selecting a subcluster, consider not only available memory but also > available vcore -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11633) [Federation] Improve LoadBasedRouterPolicy To Use Available vcores
[
https://issues.apache.org/jira/browse/YARN-11633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17796959#comment-17796959
]
ASF GitHub Bot commented on YARN-11633:
---
slfan1989 commented on code in PR #6356:
URL: https://github.com/apache/hadoop/pull/6356#discussion_r1427433693
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/policies/router/LoadBasedRouterPolicy.java:
##
@@ -66,12 +66,19 @@ protected SubClusterId chooseSubCluster(
Map weights =
getPolicyInfo().getRouterPolicyWeights();
SubClusterIdInfo chosen = null;
long currBestMem = -1;
+long currBestVcore = -1;
for (Map.Entry entry :
preSelectSubclusters.entrySet()) {
SubClusterIdInfo id = new SubClusterIdInfo(entry.getKey());
if (weights.containsKey(id) && weights.get(id) > 0) {
long availableMemory = getAvailableMemory(entry.getValue());
-if (availableMemory > currBestMem) {
+long availableVcore = getAvailableVcore(entry.getValue());
+if (availableMemory > currBestMem && availableVcore > currBestVcore) {
Review Comment:
```
if (availableMemory > currBestMem) {
}
```
This code is needed to select the largest subcluster of availableMemory.
Memory is a more strictly limited resource, because without memory, the
application cannot execute. Sufficient memory can also be considered as the
cluster is idle.
If we add cpu constraints (availableVcore), subclusters with smaller memory
may be selected, which is not an expected situation.
> [Federation] Improve LoadBasedRouterPolicy To Use Available vcores
> --
>
> Key: YARN-11633
> URL: https://issues.apache.org/jira/browse/YARN-11633
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: federation
>Affects Versions: 3.3.6
>Reporter: yanbin.zhang
>Priority: Major
> Labels: pull-request-available
>
> When selecting a subcluster, consider not only available memory but also
> available vcore
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11633) [Federation] Improve LoadBasedRouterPolicy To Use Available vcores
[
https://issues.apache.org/jira/browse/YARN-11633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17796954#comment-17796954
]
ASF GitHub Bot commented on YARN-11633:
---
slfan1989 commented on PR #6356:
URL: https://github.com/apache/hadoop/pull/6356#issuecomment-1857019443
@singer-bin Thank you forYARN's scheduler compares based on memory by
default. We can refer to `FairSharePolicy.java`
Therefore, using memory for comparison in LoadBasedRouterPolicy is in line
with the design rules of the scheduler.
```
/**
* Makes scheduling decisions by trying to equalize shares of memory.
*/
@Private
@Unstable
public class FairSharePolicy extends SchedulingPolicy {
@VisibleForTesting
public static final String NAME = "fair";
private static final Logger LOG =
LoggerFactory.getLogger(FairSharePolicy.class);
private static final String MEMORY =
ResourceInformation.MEMORY_MB.getName();
private static final DefaultResourceCalculator RESOURCE_CALCULATOR =
new DefaultResourceCalculator();
private static final FairShareComparator COMPARATOR =
new FairShareComparator();
@Override
public String getName() {
return NAME;
}
...
```
> [Federation] Improve LoadBasedRouterPolicy To Use Available vcores
> --
>
> Key: YARN-11633
> URL: https://issues.apache.org/jira/browse/YARN-11633
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: federation
>Affects Versions: 3.3.6
>Reporter: yanbin.zhang
>Priority: Major
> Labels: pull-request-available
>
> When selecting a subcluster, consider not only available memory but also
> available vcore
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11633) [Federation] Improve LoadBasedRouterPolicy To Use Available vcores
[
https://issues.apache.org/jira/browse/YARN-11633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17796543#comment-17796543
]
ASF GitHub Bot commented on YARN-11633:
---
hadoop-yetus commented on PR #6356:
URL: https://github.com/apache/hadoop/pull/6356#issuecomment-1855118902
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 50s | | trunk passed |
| +1 :green_heart: | compile | 0m 26s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 20s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 18s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 28s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 24s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 57s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 44s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 20s | | the patch passed |
| +1 :green_heart: | compile | 0m 18s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 18s | | the patch passed |
| +1 :green_heart: | compile | 0m 18s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 18s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 12s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 21s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 17s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 51s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 37s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 34s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 23s | | The patch does not
generate ASF License warnings. |
| | | 82m 39s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6356/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6356 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux ca39b28900b4 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1695037944fbcdc881e9b5a1db8e0411b53bd7d4 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6356/1/testReport/ |
| Max. process+thread count | 734 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6356/1/console |
| versions | git=2.25.1 maven=3.6.3
[jira] [Commented] (YARN-11633) [Federation] Improve LoadBasedRouterPolicy To Use Available vcores
[ https://issues.apache.org/jira/browse/YARN-11633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17796523#comment-17796523 ] ASF GitHub Bot commented on YARN-11633: --- singer-bin opened a new pull request, #6356: URL: https://github.com/apache/hadoop/pull/6356 ### Description of PR JIRA: [YARN-11633](https://issues.apache.org/jira/browse/YARN-11633). [Federation] Improve LoadBasedRouterPolicy To Use Available vcores. ### How was this patch tested? unnecessary ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? > [Federation] Improve LoadBasedRouterPolicy To Use Available vcores > -- > > Key: YARN-11633 > URL: https://issues.apache.org/jira/browse/YARN-11633 > Project: Hadoop YARN > Issue Type: Improvement > Components: federation >Affects Versions: 3.3.6 >Reporter: yanbin.zhang >Priority: Major > > When selecting a subcluster, consider not only available memory but also > available vcore -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
