[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265713#comment-17265713
]
Hadoop QA commented on YARN-10506:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
13s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 2
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
53s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 29s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
51s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 53s{color}
|
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/483/artifact/out/diff-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt{color}
| {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 generated 2 new + 40
unchanged - 2 fixed = 42 total (was 42) {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green}{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 0 new + 738 unchanged - 1 fixed = 738 total (was 739)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265652#comment-17265652 ] Wangda Tan commented on YARN-10506: --- Thanks [~zhuqi], the latest patch looks good to me! I will wait for thoughts from others, if no additional opposite opinions I will get it in by tomorrow. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506-012.patch, YARN-10506-013.patch, YARN-10506.001.patch, > YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch, > YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch, > YARN-10506.007.patch, YARN-10506.009.patch, YARN-10506.011.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265651#comment-17265651 ] zhuqi edited comment on YARN-10506 at 1/15/21, 2:56 AM: I update this and fix the check style in 013 patch. Waiting for the CI. Thanks all for patient review. was (Author: zhuqi): I update this and fix the check style in 013 patch. Thanks all for patient review. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506-012.patch, YARN-10506-013.patch, YARN-10506.001.patch, > YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch, > YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch, > YARN-10506.007.patch, YARN-10506.009.patch, YARN-10506.011.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265651#comment-17265651 ] zhuqi commented on YARN-10506: -- I update this and fix the check style in 013 patch. Thanks all for patient review. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506-012.patch, YARN-10506-013.patch, YARN-10506.001.patch, > YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch, > YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch, > YARN-10506.007.patch, YARN-10506.009.patch, YARN-10506.011.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuqi updated YARN-10506: - Attachment: YARN-10506-013.patch > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506-012.patch, YARN-10506-013.patch, YARN-10506.001.patch, > YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch, > YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch, > YARN-10506.007.patch, YARN-10506.009.patch, YARN-10506.011.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265618#comment-17265618 ] zhuqi commented on YARN-10506: -- It makes sense to me. Thanks to [~shuzirra] [~pbacsko] for your patient comment to make it clear. [~wangda] i will update according to the latest proposal. *1) yarn.scheduler.capacity..auto-queue-creation-v2.enabled* To make sure the flag is right set. *2) Removing "create" from the ApplicationPlacementContext:* Remove it, and change related test. Make the property inheritable or not, will be another jira. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506-012.patch, YARN-10506.001.patch, YARN-10506.002.patch, > YARN-10506.003.patch, YARN-10506.004.patch, YARN-10506.005.patch, > YARN-10506.006-combined.patch, YARN-10506.006.patch, YARN-10506.007.patch, > YARN-10506.009.patch, YARN-10506.011.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10512) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include mode of operation for CS
[
https://issues.apache.org/jira/browse/YARN-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265613#comment-17265613
]
Hadoop QA commented on YARN-10512:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
39s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 2
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
17s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 39s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m
10s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
8s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
50s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 35s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/481/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
| {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 27 new + 94 unchanged - 0 fixed = 121 total (was 94)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/481/artifact/out/whitespace-eol.txt{color}
| {color:red} The patch has 9 line(s) that end in whitespace. Use git apply
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green}
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265521#comment-17265521 ] Wangda Tan commented on YARN-10506: --- Sounds good, thanks [~shuzirra], [~pbacsko]. It makes sense to me, [~zhuqi] can you add your thoughts about the latest proposal? Thanks, > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506-012.patch, YARN-10506.001.patch, YARN-10506.002.patch, > YARN-10506.003.patch, YARN-10506.004.patch, YARN-10506.005.patch, > YARN-10506.006-combined.patch, YARN-10506.006.patch, YARN-10506.007.patch, > YARN-10506.009.patch, YARN-10506.011.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10535) Make queue placement in CapacityScheduler compliant with auto-queue-placement
[
https://issues.apache.org/jira/browse/YARN-10535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265520#comment-17265520
]
Hadoop QA commented on YARN-10535:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
36s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
36s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 32s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m
11s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
9s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
33s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/482/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
33s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/482/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 33s{color}
|
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/482/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
33s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/482/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 33s{color}
|
[jira] [Commented] (YARN-10512) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include mode of operation for CS
[ https://issues.apache.org/jira/browse/YARN-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265403#comment-17265403 ] Szilard Nemeth commented on YARN-10512: --- Added first patch. Just realized that the tests should not be in TestRMWebServices, but in TestRMWebServicesCapacitySched. Will move the testcases there tomorrow and upload a new patch. > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include > mode of operation for CS > -- > > Key: YARN-10512 > URL: https://issues.apache.org/jira/browse/YARN-10512 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-10512.001.patch > > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > We would like to expose the mode of operation with the RM's /scheduler REST > endpoint. > The field name will be 'mode'. > All queue representations in the response will be uniformly hold any of the > mode values of: "percentage", "absolute", "weight". -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10512) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include mode of operation for CS
[ https://issues.apache.org/jira/browse/YARN-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-10512: -- Description: Under this umbrella (YARN-10496), weight-mode has been implemented for CS with YARN-10504. We would like to expose the mode of operation with the RM's /scheduler REST endpoint. The field name will be 'mode'. All queue representations in the response will be uniformly hold any of the mode values of: "percentage", "absolute", "weight". was: Under this umbrella (YARN-10496), weight-mode has been implemented for CS with YARN-10504. We would like to expose the mode of operation with the RM's /scheduler REST endpoint. The field name will be 'mode'. > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include > mode of operation for CS > -- > > Key: YARN-10512 > URL: https://issues.apache.org/jira/browse/YARN-10512 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-10512.001.patch > > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > We would like to expose the mode of operation with the RM's /scheduler REST > endpoint. > The field name will be 'mode'. > All queue representations in the response will be uniformly hold any of the > mode values of: "percentage", "absolute", "weight". -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10512) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include mode of operation for CS
[ https://issues.apache.org/jira/browse/YARN-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-10512: -- Attachment: YARN-10512.001.patch > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include > mode of operation for CS > -- > > Key: YARN-10512 > URL: https://issues.apache.org/jira/browse/YARN-10512 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-10512.001.patch > > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > We would like to expose the mode of operation with the RM's /scheduler REST > endpoint. > The field name will be 'mode'. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10535) Make queue placement in CapacityScheduler compliant with auto-queue-placement
[ https://issues.apache.org/jira/browse/YARN-10535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265402#comment-17265402 ] Gergely Pollak commented on YARN-10535: --- The first iteration of the patch includes all modifications needed to be compliant with the changes being introduced in YARN-10506. This patch will not compile until YARN-10506 is merged, but I upload it for initial reviews. Also I'll add some new test cases for the new possible cases introduced in YARN-10506. > Make queue placement in CapacityScheduler compliant with auto-queue-placement > - > > Key: YARN-10535 > URL: https://issues.apache.org/jira/browse/YARN-10535 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Wangda Tan >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-10535.001.patch > > > Once YARN-10506 is done, we need to call the API from the queue placement > policy to create queues. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10535) Make queue placement in CapacityScheduler compliant with auto-queue-placement
[ https://issues.apache.org/jira/browse/YARN-10535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergely Pollak updated YARN-10535: -- Attachment: YARN-10535.001.patch > Make queue placement in CapacityScheduler compliant with auto-queue-placement > - > > Key: YARN-10535 > URL: https://issues.apache.org/jira/browse/YARN-10535 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Wangda Tan >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-10535.001.patch > > > Once YARN-10506 is done, we need to call the API from the queue placement > policy to create queues. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10535) Make queue placement in CapacityScheduler compliant with auto-queue-placement
[ https://issues.apache.org/jira/browse/YARN-10535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergely Pollak updated YARN-10535: -- Summary: Make queue placement in CapacityScheduler compliant with auto-queue-placement (was: Make changes in queue placement policy to be compliant with auto-queue-placement in CapacityScheduler) > Make queue placement in CapacityScheduler compliant with auto-queue-placement > - > > Key: YARN-10535 > URL: https://issues.apache.org/jira/browse/YARN-10535 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Wangda Tan >Assignee: Gergely Pollak >Priority: Major > > Once YARN-10506 is done, we need to call the API from the queue placement > policy to create queues. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10535) Make changes in queue placement policy to be compliant with auto-queue-placement in CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-10535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergely Pollak updated YARN-10535: -- Summary: Make changes in queue placement policy to be compliant with auto-queue-placement in CapacityScheduler (was: Make changes in queue placement policy to use auto-queue-placement API in CapacityScheduler) > Make changes in queue placement policy to be compliant with > auto-queue-placement in CapacityScheduler > - > > Key: YARN-10535 > URL: https://issues.apache.org/jira/browse/YARN-10535 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Wangda Tan >Assignee: Gergely Pollak >Priority: Major > > Once YARN-10506 is done, we need to call the API from the queue placement > policy to create queues. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10512) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include mode of operation for CS
[ https://issues.apache.org/jira/browse/YARN-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-10512: -- Description: Under this umbrella (YARN-10496), weight-mode has been implemented for CS with YARN-10504. We would like to expose the mode of operation with the RM's /scheduler REST endpoint. The field name will be 'mode'. > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include > mode of operation for CS > -- > > Key: YARN-10512 > URL: https://issues.apache.org/jira/browse/YARN-10512 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Szilard Nemeth >Priority: Major > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > We would like to expose the mode of operation with the RM's /scheduler REST > endpoint. > The field name will be 'mode'. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10512) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include mode of operation for CS
[ https://issues.apache.org/jira/browse/YARN-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-10512: -- Summary: CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include mode of operation for CS (was: CS Flexible Auto Queue Creation: Modify RM's /scheduler endpoint to include mode of operation for CS) > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include > mode of operation for CS > -- > > Key: YARN-10512 > URL: https://issues.apache.org/jira/browse/YARN-10512 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Szilard Nemeth >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265208#comment-17265208
]
Gergely Pollak edited comment on YARN-10506 at 1/14/21, 8:29 PM:
-
Thank you [~wangda] [~pbacsko] for your insights!
Actually the queue creation checks were present in the legacy engine
([https://github.com/apache/hadoop/blob/9d5144e66d71551ad6e1a8d3f1670e49ba181999/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java#L333-L340]),
if I see correctly since 3.0. But the first commit seems much older:
[https://github.com/apache/hadoop/commit/0987a7b8c2c1e4c2095821d98a7db19644df]
.
I've just kept it during the refactor, and as Peter pointed out now we actually
build on it, since we need to be able to determine if a rule will be executed
successfully (ie. the queue can be created), and if it cannot we can move onto
the next rule. This fallback behaviour is part of the FS Placement engine as
well, so we need it in order to server FS customers, so it was somewhat lucky
that it was already in place. (In the original implementation we just used this
check to throw exceptions, which was quite pointless, since the CS would have
rejected the application if the queue was uncreatable anyway)
So this is why I say we can safely remove those flags from the application
placement context, this way we can reduce the code complexity.
This way we only need what [~pbacsko] mentioned:
{code:java}
1) "create" flag on the rule itself in the JSON - true/false
2) "yarh.scheduler.capacity..auto-queue-creation-v2.enabled" -
true/false{code}
Also I'm currently adopting this change in the CSMappingPlacementRule, and I
don't use these flags. What I'll need in the future a better, centralised way
to determine if a path can be created, and both CS and the MappingEngine should
use the same methods, to do it. Currently we are implementing the same logic
differently, and it's much harder to maintain the code this way, but during the
followup code cleanup jiras we'll take care of this.
About the race condition, I think (and I might be very wrong, because race
conditions can be really sneaky), we cannot do much about, theoretically it can
happen that the Mapping engine determines a rule can be executed, because the
target queue exists or there is a proper dynamic parent, then a config change
occurs, and by the time we get to the actual creation the queue structure
changes, at this point the rejection is the proper action to take since the
submission IS against the configuration. And I don't see a way around it since
at the time the MappingEngine makes the decision that decision is the correct
one, but we shouldn't enforce this decision if the configuration changes. But
this race condition should only occur when the user changes the queue
configuration, and in this case they should expect failing application
submissions for the queues which have been changed or removed.
was (Author: shuzirra):
Actually the queue creation checks were present in the legacy engine
(https://github.com/apache/hadoop/blob/9d5144e66d71551ad6e1a8d3f1670e49ba181999/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java#L333-L340),
if I see correctly since 3.0. But the first commit seems much older:
[https://github.com/apache/hadoop/commit/0987a7b8c2c1e4c2095821d98a7db19644df]
.
I've just kept it during the refactor, and as Peter pointed out now we actually
build on it, since we need to be able to determine if a rule will be executed
successfully (ie. the queue can be created), and if it cannot we can move onto
the next rule. This fallback behaviour is part of the FS Placement engine as
well, so we need it in order to server FS customers, so it was somewhat lucky
that it was already in place. (In the original implementation we just used this
check to throw exceptions, which was quite pointless, since the CS would have
rejected the application if the queue was uncreatable anyway)
So this is why I say we can safely remove those flags from the application
placement context, this way we can reduce the code complexity.
This way we only need what [~pbacsko] mentioned:
{code:java}
1) "create" flag on the rule itself in the JSON - true/false
2) "yarh.scheduler.capacity..auto-queue-creation-v2.enabled" -
true/false{code}
Also I'm currently adopting this change in the CSMappingPlacementRule, and I
don't use these flags. What I'll need in the future a better, centralised way
to determine if a path can be created, and both CS and the MappingEngine should
use the same methods, to do it. Currently we are implementing the same logic
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265208#comment-17265208
]
Gergely Pollak commented on YARN-10506:
---
Actually the queue creation checks were present in the legacy engine
(https://github.com/apache/hadoop/blob/9d5144e66d71551ad6e1a8d3f1670e49ba181999/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java#L333-L340),
if I see correctly since 3.0. But the first commit seems much older:
[https://github.com/apache/hadoop/commit/0987a7b8c2c1e4c2095821d98a7db19644df]
.
I've just kept it during the refactor, and as Peter pointed out now we actually
build on it, since we need to be able to determine if a rule will be executed
successfully (ie. the queue can be created), and if it cannot we can move onto
the next rule. This fallback behaviour is part of the FS Placement engine as
well, so we need it in order to server FS customers, so it was somewhat lucky
that it was already in place. (In the original implementation we just used this
check to throw exceptions, which was quite pointless, since the CS would have
rejected the application if the queue was uncreatable anyway)
So this is why I say we can safely remove those flags from the application
placement context, this way we can reduce the code complexity.
This way we only need what [~pbacsko] mentioned:
{code:java}
1) "create" flag on the rule itself in the JSON - true/false
2) "yarh.scheduler.capacity..auto-queue-creation-v2.enabled" -
true/false{code}
Also I'm currently adopting this change in the CSMappingPlacementRule, and I
don't use these flags. What I'll need in the future a better, centralised way
to determine if a path can be created, and both CS and the MappingEngine should
use the same methods, to do it. Currently we are implementing the same logic
differently, and it's much harder to maintain the code this way, but during the
followup code cleanup jiras we'll take care of this.
About the race condition, I think (and I might be very wrong, because race
conditions can be really sneaky), we cannot do much about, theoretically it can
happen that the Mapping engine determines a rule can be executed, because the
target queue exists or there is a proper dynamic parent, then a config change
occurs, and by the time we get to the actual creation the queue structure
changes, at this point the rejection is the proper action to take since the
submission IS against the configuration. And I don't see a way around it since
at the time the MappingEngine makes the decision that decision is the correct
one, but we shouldn't enforce this decision if the configuration changes. But
this race condition should only occur when the user changes the queue
configuration, and in this case they should expect failing application
submissions for the queues which have been changed or removed.
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10506-006-10504-010.patch,
> YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch,
> YARN-10506-012.patch, YARN-10506.001.patch, YARN-10506.002.patch,
> YARN-10506.003.patch, YARN-10506.004.patch, YARN-10506.005.patch,
> YARN-10506.006-combined.patch, YARN-10506.006.patch, YARN-10506.007.patch,
> YARN-10506.009.patch, YARN-10506.011.patch
>
>
> The queue creation logic should be updated to use weight mode and support the
> flexible creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265149#comment-17265149
]
Wangda Tan commented on YARN-10506:
---
[~pbacsko], I see, I think it makes sense.
{quote}The logic must be there, otherwise we can't proceed to the next rule if
the fallback is set to "skip". If you perform the validations (which are
currently done by {{CSMappingPlacementRule}}) outside of the placement engine,
you lose the ability to execute different actions (skip/reject/place to
default).
{quote}
Because it is different from the previous CS queue mapping: previously, when a
rule matches, it up to the scheduler to make a decision such as create queue,
accept or reject the app.
Now we need to make sure rule engine makes the right decision before sending
the request to scheduler. We need to be careful about possible racing
conditions. But I think you're right, when admin change
.auto-queue-creation-v2.enabled in the queue hierarchy, it is possible
to cause undesirable result. But syncing between the two systems (rule engine
and scheduler) will have delays in any case, so removing create flag from
ApplicationPlacementContext sounds not bad.
I will let [~zhuqi], [~gandras] and you guys to decide what is the best
approach to take, I'm OK with either approaches. But let's reach a conclusion
fast since this patch blocks a number of follow up tickets.
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10506-006-10504-010.patch,
> YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch,
> YARN-10506-012.patch, YARN-10506.001.patch, YARN-10506.002.patch,
> YARN-10506.003.patch, YARN-10506.004.patch, YARN-10506.005.patch,
> YARN-10506.006-combined.patch, YARN-10506.006.patch, YARN-10506.007.patch,
> YARN-10506.009.patch, YARN-10506.011.patch
>
>
> The queue creation logic should be updated to use weight mode and support the
> flexible creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265144#comment-17265144
]
Peter Bacsko commented on YARN-10506:
-
"_It is possible when the rule checks, the queue allow creation, but when
ApplicationPlacementContext arrived at scheduler, the queue is refreshed"_
This indeed sounds like a race condition, but if the queue is removed in the
meantime, then the submission would fail anyway. The refresh itself is perfomed
while holding a write lock, so that in itself should not be a problem.
"RuleEngine can just pass ApplicationPLacementContext to Scheduler and let
scheduler making the decision"
Could you elaborate on what "decision making" you're referring to? I think
inside the engine, it's important that it has to determine whether the target
queue is available or not, if not, can we create it, etc. The logic must be
there, otherwise we can't proceed to the next rule if the fallback is set to
"skip". If you perform the validations (which are currently done by
{{CSMappingPlacementRule}}) outside of the placement engine, you lose the
ability to execute different actions (skip/reject/place to default).
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10506-006-10504-010.patch,
> YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch,
> YARN-10506-012.patch, YARN-10506.001.patch, YARN-10506.002.patch,
> YARN-10506.003.patch, YARN-10506.004.patch, YARN-10506.005.patch,
> YARN-10506.006-combined.patch, YARN-10506.006.patch, YARN-10506.007.patch,
> YARN-10506.009.patch, YARN-10506.011.patch
>
>
> The queue creation logic should be updated to use weight mode and support the
> flexible creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265085#comment-17265085 ] Wangda Tan commented on YARN-10506: --- Thanks [~shuzirra], [~pbacsko], posting this comment while in the middle of something else, I hope I didn't miss important information. Here you shared several things: *1) Add a create flag on the rule itself in the JSON:* I agreed. *2) yarh.scheduler.capacity..auto-queue-creation-v2.enabled* I agreed. Also, I agree we should look at if we can make the property inheritable or not, I agree but I suggest to move the implementation to a separate Jira. *3) Removing "create" from the ApplicationPlacementContext:* I initially thought it is needed because the rule engine may not do exhaust checks and the concurrency issue. _(It is possible when the rule checks, the queue allow creation, but when ApplicationPlacementContext arrived at scheduler, the queue is refreshed)_. So I think it is still valuable to let the scheduler do the creation based on demand, and fail the App submission atomically. Also, if we relies on Scheduler to check creatable or not, it can reduce complexities of RuleEngine to do additional check. RuleEngine can just pass ApplicationPLacementContext to Scheduler and let scheduler making the decision. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506-012.patch, YARN-10506.001.patch, YARN-10506.002.patch, > YARN-10506.003.patch, YARN-10506.004.patch, YARN-10506.005.patch, > YARN-10506.006-combined.patch, YARN-10506.006.patch, YARN-10506.007.patch, > YARN-10506.009.patch, YARN-10506.011.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264977#comment-17264977 ] Peter Bacsko edited comment on YARN-10506 at 1/14/21, 3:56 PM: --- *Clarification comment* It turned out I misunderstood [~shuzirra]'s comment above. I thought he was speaking about the "create" flag the comes from the rule itself. I should have read it more carefully. Anyway we just had an internal meeting so now I have a better understanding. _"How we deal with "create" flag of ApplicationPlacementContext?"_ I don't think we need _any_ flags in this class. We need two settings basically: 1) "create" flag on the rule itself in the JSON - true/false 2) "yarh.scheduler.capacity..auto-queue-creation-v2.enabled" - true/false The placement engine (even the old one) can access both to make a decision about whether the queue is allowed to be created or not. If it is, it returns the full path inside the "ApplicationPlacementContext". Otherwise it decides what to do by checking the "fallbackResult" setting of the rule: returns "root.default", proceeds to the next rule or rejects the application. In fact, Gergo told me that *the new engine already works the way I proposed*. We don't need any additional input or flag to make an explicit, valid decision about the placement. Also, [~gandras] + [~snemeth] reached the same conclusion. was (Author: pbacsko): *Clarification comment* I turned out I misunderstood [~shuzirra]'s comment above. I thought he was speaking about the "create" flag the comes from the rule itself. I should have read it more carefully. Anyway we just had an internal meeting so now I have a better understanding. _"How we deal with "create" flag of ApplicationPlacementContext?"_ I don't think we need _any_ flags in this class. We need two settings basically: 1) "create" flag on the rule itself in the JSON - true/false 2) "yarh.scheduler.capacity..auto-queue-creation-v2.enabled" - true/false The placement engine (even the old one) can access both to make a decision about whether the queue is allowed to be created or not. If it is, it returns the full path inside the "ApplicationPlacementContext". Otherwise it decides what to do by checking the "fallbackResult" setting of the rule: returns "root.default", proceeds to the next rule or rejects the application. In fact, Gergo told me that *the new engine already works the way I proposed*. We don't need any additional input or flag to make an explicit, valid decision about the placement. Also, [~gandras] + [~snemeth] reached the same conclusion. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506-012.patch, YARN-10506.001.patch, YARN-10506.002.patch, > YARN-10506.003.patch, YARN-10506.004.patch, YARN-10506.005.patch, > YARN-10506.006-combined.patch, YARN-10506.006.patch, YARN-10506.007.patch, > YARN-10506.009.patch, YARN-10506.011.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264977#comment-17264977 ] Peter Bacsko commented on YARN-10506: - *Clarification comment* I turned out I misunderstood [~shuzirra]'s comment above. I thought he was speaking about the "create" flag the comes from the rule itself. I should have read it more carefully. Anyway we just had an internal meeting so now I have a better understanding. _"How we deal with "create" flag of ApplicationPlacementContext?"_ I don't think we need _any_ flags in this class. We need two settings basically: 1) "create" flag on the rule itself in the JSON - true/false 2) "yarh.scheduler.capacity..auto-queue-creation-v2.enabled" - true/false The placement engine (even the old one) can access both to make a decision about whether the queue is allowed to be created or not. If it is, it returns the full path inside the "ApplicationPlacementContext". Otherwise it decides what to do by checking the "fallbackResult" setting of the rule: returns "root.default", proceeds to the next rule or rejects the application. In fact, Gergo told me that *the new engine already works the way I proposed*. We don't need any additional input or flag to make an explicit, valid decision about the placement. Also, [~gandras] + [~snemeth] reached the same conclusion. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506-012.patch, YARN-10506.001.patch, YARN-10506.002.patch, > YARN-10506.003.patch, YARN-10506.004.patch, YARN-10506.005.patch, > YARN-10506.006-combined.patch, YARN-10506.006.patch, YARN-10506.007.patch, > YARN-10506.009.patch, YARN-10506.011.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10564) Support Auto Queue Creation template configurations
[
https://issues.apache.org/jira/browse/YARN-10564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Gyori reassigned YARN-10564:
---
Assignee: Andras Gyori (was: zhuqi)
> Support Auto Queue Creation template configurations
> ---
>
> Key: YARN-10564
> URL: https://issues.apache.org/jira/browse/YARN-10564
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10564.poc.001.patch
>
>
> Similar to how the template configuration works for ManagedParents, we need
> to support templates for the new auto queue creation logic. Proposition is to
> allow wildcards in template configs such as:
> {noformat}
> yarn.scheduler.capacity.root.*.*.weight 10{noformat}
> which would mean, that set weight to 10 of every leaf of every parent under
> root.
> We should possibly take an approach, that could support arbitrary depth of
> template configuration, because we might need to lift the limitation of auto
> queue nesting.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264855#comment-17264855
]
Gergely Pollak edited comment on YARN-10506 at 1/14/21, 3:46 PM:
-
Hi [~gandras] [~wangda], [~zhuqi]
*1) How we deal with "create" flag of ApplicationPlacementContext?* **
I don't think we need this flag at all. The CapacityScheduler has the
information where can be dynamic queues created, it is supplied via the
configuration option {{queue-path.auto-queue-creation-v2.enabled}}
and
{\{queue-path.auto-queue-creation.enabled }}
So at any point the CS will be aware if a supplied queue path is creatable.
Mapping rules will return a path based on their configuration, but I don't
think the placement rule should be overriding the CS configuration. This can
cause some undesired issues, and we really don't get anything with it. It will
be a little more inconvenient for the user since they have to configure the
mapping rules AND the CS queues properly, but it doesn't look like a big deal.
On the other hand there are multiple places in the code which check if a queue
can be created and all these places use the queue hierarchy to determine
whether a queue is a managed parent, or not, however if the Mapping Rules are
allowed to circumvent this, it might lead to unforeseen inconsistencies. (As a
matter of fact the CSMappingPlacementRule currently uses the information from
CS to determine if a queue can be created or not, so it wouldn't set any of
these flags differently than CS is configured anyway)
So I would say let's omit the creation flags from the
ApoplicationPlacementContext, let the CSMappingPlacementRule rely on the
information provided by the CS about queue creation eligibility, and let it
pass back a "regular" ApplicationPlacementContext with the desired queue path,
then the CS will be able to create it if the configuration allows. This is a
much easier and stable solution, while we don't really put extra administration
overhead to the user, and also can be easily extended in the future.
As I see circumventing protections are never a good idea even if we are talking
about internal interfaces, and these flags would exactly do that, they would
force the CS to create queues which are against it's configuration.
was (Author: shuzirra):
Hi [~gandras] [~wangda], [~zhuqi]
*1) How we deal with "create" flag of ApplicationPlacementContext?* **
I don't think we need this flag at all. The CapacityScheduler has the
information where can be dynamic queues created, it is supplied via the
configuration option {{queue-path.auto-queue-creation-v2.enabled}}
and
{\{queue-path.auto-queue-creation.enabled }}
So at any point the CS will be aware if a supplied queue path is creatable.
Mapping rules will return a path based on their configuration, but I don't
think the placement rule should be overriding the CS configuration. This can
cause some undesired issues, and we really don't get anything with it. It will
be a little more inconvenient for the user since they have to configure the
mapping rules AND the CS queues properly, but it doesn't look like a big deal.
On the other hand there are multiple places in the code which check if a queue
can be created and all these places use the queue hierarchy to determine
whether a queue is a managed parent, or not, however if the Mapping Rules are
allowed to circumvent this, it might lead to unforeseen inconsistencies. (As a
matter of fact the CSMappingPlacementRule currently uses the information from
CS to determine if a queue can be created or not, so it wouldn't set any of
these flags differently then CS is configured anyway)
So I would say let's omit the creation flags from the
ApoplicationPlacementContext, let the CSMappingPlacementRule rely on the
information provided by the CS about queue creation eligibility, and let it
pass back a "regular" ApplicationPlacementContext with the desired queue path,
then the CS will be able to create it if the configuration allows. This is a
much easier and stable solution, while we don't really put extra administration
overhead to the user, and also can be easily extended in the future.
As I see circumventing protections are never a good idea even if we are talking
about internal interfaces, and these flags would exactly do that, they would
force the CS to create queues which are against it's configuration.
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>
[jira] [Comment Edited] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264855#comment-17264855
]
Gergely Pollak edited comment on YARN-10506 at 1/14/21, 3:46 PM:
-
Hi [~gandras] [~wangda], [~zhuqi]
*1) How we deal with "create" flag of ApplicationPlacementContext?* **
I don't think we need this flag at all. The CapacityScheduler has the
information where can be dynamic queues created, it is supplied via the
configuration option {{queue-path.auto-queue-creation-v2.enabled}}
and
{\{queue-path.auto-queue-creation.enabled }}
So at any point the CS will be aware if a supplied queue path is creatable.
Mapping rules will return a path based on their configuration, but I don't
think the placement rule should be overriding the CS configuration. This can
cause some undesired issues, and we really don't get anything with it. It will
be a little more inconvenient for the user since they have to configure the
mapping rules AND the CS queues properly, but it doesn't look like a big deal.
On the other hand there are multiple places in the code which check if a queue
can be created and all these places use the queue hierarchy to determine
whether a queue is a managed parent, or not, however if the Mapping Rules are
allowed to circumvent this, it might lead to unforeseen inconsistencies. (As a
matter of fact the CSMappingPlacementRule currently uses the information from
CS to determine if a queue can be created or not, so it wouldn't set any of
these flags differently than CS is configured anyway)
So I would say let's omit the creation flags from the
ApplicationPlacementContext, let the CSMappingPlacementRule rely on the
information provided by the CS about queue creation eligibility, and let it
pass back a "regular" ApplicationPlacementContext with the desired queue path,
then the CS will be able to create it if the configuration allows. This is a
much easier and stable solution, while we don't really put extra administration
overhead to the user, and also can be easily extended in the future.
As I see circumventing protections are never a good idea even if we are talking
about internal interfaces, and these flags would exactly do that, they would
force the CS to create queues which are against it's configuration.
was (Author: shuzirra):
Hi [~gandras] [~wangda], [~zhuqi]
*1) How we deal with "create" flag of ApplicationPlacementContext?* **
I don't think we need this flag at all. The CapacityScheduler has the
information where can be dynamic queues created, it is supplied via the
configuration option {{queue-path.auto-queue-creation-v2.enabled}}
and
{\{queue-path.auto-queue-creation.enabled }}
So at any point the CS will be aware if a supplied queue path is creatable.
Mapping rules will return a path based on their configuration, but I don't
think the placement rule should be overriding the CS configuration. This can
cause some undesired issues, and we really don't get anything with it. It will
be a little more inconvenient for the user since they have to configure the
mapping rules AND the CS queues properly, but it doesn't look like a big deal.
On the other hand there are multiple places in the code which check if a queue
can be created and all these places use the queue hierarchy to determine
whether a queue is a managed parent, or not, however if the Mapping Rules are
allowed to circumvent this, it might lead to unforeseen inconsistencies. (As a
matter of fact the CSMappingPlacementRule currently uses the information from
CS to determine if a queue can be created or not, so it wouldn't set any of
these flags differently than CS is configured anyway)
So I would say let's omit the creation flags from the
ApoplicationPlacementContext, let the CSMappingPlacementRule rely on the
information provided by the CS about queue creation eligibility, and let it
pass back a "regular" ApplicationPlacementContext with the desired queue path,
then the CS will be able to create it if the configuration allows. This is a
much easier and stable solution, while we don't really put extra administration
overhead to the user, and also can be easily extended in the future.
As I see circumventing protections are never a good idea even if we are talking
about internal interfaces, and these flags would exactly do that, they would
force the CS to create queues which are against it's configuration.
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority:
[jira] [Commented] (YARN-10564) Support Auto Queue Creation template configurations
[
https://issues.apache.org/jira/browse/YARN-10564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264929#comment-17264929
]
zhuqi commented on YARN-10564:
--
[~gandras]
Ok, you can take it based the poc , i'm glad to work with you.
And also, i am working on YARN-10532 with some simple policy now.
If you want to fill policy also, you can first handle the normal template
support in this.
Thanks.
> Support Auto Queue Creation template configurations
> ---
>
> Key: YARN-10564
> URL: https://issues.apache.org/jira/browse/YARN-10564
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: zhuqi
>Priority: Major
> Attachments: YARN-10564.poc.001.patch
>
>
> Similar to how the template configuration works for ManagedParents, we need
> to support templates for the new auto queue creation logic. Proposition is to
> allow wildcards in template configs such as:
> {noformat}
> yarn.scheduler.capacity.root.*.*.weight 10{noformat}
> which would mean, that set weight to 10 of every leaf of every parent under
> root.
> We should possibly take an approach, that could support arbitrary depth of
> template configuration, because we might need to lift the limitation of auto
> queue nesting.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10564) Support Auto Queue Creation template configurations
[
https://issues.apache.org/jira/browse/YARN-10564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264919#comment-17264919
]
Andras Gyori commented on YARN-10564:
-
[~zhuqi] is it okay if I retake this issue, because I have already started
working on this?
> Support Auto Queue Creation template configurations
> ---
>
> Key: YARN-10564
> URL: https://issues.apache.org/jira/browse/YARN-10564
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: zhuqi
>Priority: Major
> Attachments: YARN-10564.poc.001.patch
>
>
> Similar to how the template configuration works for ManagedParents, we need
> to support templates for the new auto queue creation logic. Proposition is to
> allow wildcards in template configs such as:
> {noformat}
> yarn.scheduler.capacity.root.*.*.weight 10{noformat}
> which would mean, that set weight to 10 of every leaf of every parent under
> root.
> We should possibly take an approach, that could support arbitrary depth of
> template configuration, because we might need to lift the limitation of auto
> queue nesting.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264903#comment-17264903
]
Peter Bacsko edited comment on YARN-10506 at 1/14/21, 1:50 PM:
---
Hi guys. I haven't really contributed to this patch and there's some big
discussion going on here, but I'm adding my 2 cents.
To give you some background: I introduced the "create" flag for the new mapping
rules, simply because FS has the same - auto-creation depends on the rule and
users who migrate from FS will find this intuitive, even if the rule format has
changed drastically. Now, I was under the impression that with weight mode and
AQC, we're trying to approximate FS behaviour as much as possible, because the
whole "project" came up in the context on FS->CS migration.
So I thought that in this mode, users are allowed to create queues anywhere in
the hierarchy, eg. with {{}} rule, you can definitely do this in FS,
there is no restriction, at least I'm not aware of it. The FS property
"allow-undeclared-pools" is translated internally to {{}} rule.
Again, the was the idea.
On the other hand, it's also true that if we go this way, legacy percentage
mode and weight mode will diverge significantly, so the behavior will be
totally different. In pct/absolute mode, you can explicitly say where queues
can be created, but if a user switches to weight mode, they cannot do this,
which might be surprising for them (or maybe not - this is a pure speculation
at this point).
If we really want to limit users where they can create queues (as I can see
this is clearly the goal here) then this leads us to the problem what Gergely
just discussed above, have a "create" flag in the rule and have a similar
property which restricts whether a queue can be created, this can lead to some
unexpected behavior, especially if their priority is reversed. At the very
least, legacy FS users will be confused like hell IMO.
We also have to think about the following scenario: a user who just migrated
from FS, still wants the exact FS auto-queue semantics and uses
"allow-undeclared-pools" or a {{}} rule. So if that's the case, we
have to be able to set this up easily. That is, if they have 200 queues and
they have to set this property for _every single queue_, I can easily imagine
that their head will explode. So we should consider having the property
"auto-queue-creation-v2.enabled" as a setting which is automatically inherited
by all child queues, recursively. Or choose the default value to be "true"
which is probably the simplest thing to do.
My conclusion is:
1) I think we have to keep the "create" flag on a rule level if we want FS
rules to work the same way. Some rules might have create=true, some of them
might not, so we can't rely on having the queue properties alone.
2) If we want to introduce "auto-queue-creation-v2.enabled", it's important to
think about whether to make it inheritable or not. If it's not, then this can
potentially cause a huge pain for some users. Alternative approach: have a
separate property which applies to the whole hierarchy globally, which can be
overridden on a per-queue basis, this is a very common approach in CS.
3) I very much agree with this statement: _"Mapping rules will return a path
based on their configuration, but I don't think the placement rule should be
overriding the CS configuration"_. If we truly want both, then the queue
property should have its final say about the creation. The rule says "let's
create this queue if it doesn't exist", but then we must check
"auto-queue-creation-v2.enabled" for the target path to see if the queue can be
created at the given point in the hierarchy. It should not be the other way
around.
was (Author: pbacsko):
Hi guys. I haven't really contributed to this patch and there's some big
discussion going on here, but I'm adding my 2 cents.
To give you some background: I introduced the "create" flag for the new mapping
rules, simply because FS has the same - auto-creation depends on the rule and
users who migrate from FS will find this intuitive, even if the rule format has
changed drastically. Now, I was under the impression that with weight mode and
AQC, we're trying to approximate FS behaviour as much as possible, because the
whole "project" came up in the context on FS->CS migration.
So I thought that in this mode, users are allowed to create queues anywhere in
the hierarchy, eg. with {{}} rule, you can definitely do this in FS,
there is no restriction, at least I'm not aware of it. The FS property
"allow-undeclared-pools" is translated internally to {{}} rule.
Again, the was the idea.
On the other hand, it's also true that if we go this way, legacy percentage
mode and weight mode will diverge significantly, so the behavior will be
totally different. In pct/absolute mode, you can explicitly say where queues
can be
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264903#comment-17264903
]
Peter Bacsko commented on YARN-10506:
-
Hi guys. I haven't really contributed to this patch and there's some big
discussion going on here, but I'm adding my 2 cents.
To give you some background: I introduced the "create" flag for the new mapping
rules, simply because FS has the same - auto-creation depends on the rule and
users who migrate from FS will find this intuitive, even if the rule format has
changed drastically. Now, I was under the impression that with weight mode and
AQC, we're trying to approximate FS behaviour as much as possible, because the
whole "project" came up in the context on FS->CS migration.
So I thought that in this mode, users are allowed to create queues anywhere in
the hierarchy, eg. with {{}} rule, you can definitely do this in FS,
there is no restriction, at least I'm not aware of it. The FS property
"allow-undeclared-pools" is translated internally to {{}} rule.
Again, the was the idea.
On the other hand, it's also true that if we go this way, legacy percentage
mode and weight mode will diverge significantly, so the behavior will be
totally different. In pct/absolute mode, you can explicitly say where queues
can be created, but if a user switches to weight mode, they cannot do this,
which might be surprising for them (or maybe not - this is a pure speculation
at this point).
If we really want to limit users where they can create queues (as I can see
this is clearly the goal here) then this leads us to the problem what Gergely
just discussed above, have a "create" flag in the rule and have a similar
property which restricts whether a queue can be created, this can lead to some
unexpected behavior, especially if their priority is reversed. At the very
least, legacy FS users will be confused like hell IMO.
We also have to think about the following scenario: a user who just migrated
from FS, still wants the exact FS auto-queue semantics and uses
"allow-undeclared-pools" or a {{}} rule. So if that's the case, we
have to be able to set this up easily. That is, if they have 200 queues and
they have to set this property for _every single queue_, I can easily imagine
that their head will explode. So we should consider having the property
"auto-queue-creation-v2.enabled" as a setting which is automatically inherited
by all child queues, recursively. Or choose the default value to be "true"
which is probably the simplest thing to do.
My conclusion is:
1) I think we have to keep the "create" flag on a rule level if we want FS
rules to work the same way. Some rules might have create=true, some of them
might not, so we can't rely on having the queue properties alone.
2) If we want to introduce "auto-queue-creation-v2.enabled", it's important to
think about whether to make it inheritable or not. If it's not, then this can
potentially cause a huge pain for some users. Alternative approach: have a
separate property which applies to the whole hierarchy globally, which can be
overridden on a per-queue basis, this is a very common approach in CS.
3) I very much agree with this statement: _"Mapping rules will return a path
based on their configuration, but I don't think the placement rule should be
overriding the CS configuration"_. If we truly want both, then the queue
property should have its final say about the creation. The rule says "let's
create this queue if it doesn't exist", but then we must check
"auto-queue-creation-v2.enabled" for the target path to see if the queue can be
created at the given point in the hierarchy. I should not be the other way
around.
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10506-006-10504-010.patch,
> YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch,
> YARN-10506-012.patch, YARN-10506.001.patch, YARN-10506.002.patch,
> YARN-10506.003.patch, YARN-10506.004.patch, YARN-10506.005.patch,
> YARN-10506.006-combined.patch, YARN-10506.006.patch, YARN-10506.007.patch,
> YARN-10506.009.patch, YARN-10506.011.patch
>
>
> The queue creation logic should be updated to use weight mode and support the
> flexible creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail:
[jira] [Comment Edited] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264855#comment-17264855
]
Gergely Pollak edited comment on YARN-10506 at 1/14/21, 12:15 PM:
--
Hi [~gandras] [~wangda], [~zhuqi]
*1) How we deal with "create" flag of ApplicationPlacementContext?* **
I don't think we need this flag at all. The CapacityScheduler has the
information where can be dynamic queues created, it is supplied via the
configuration option {{queue-path.auto-queue-creation-v2.enabled}}
and
{\{queue-path.auto-queue-creation.enabled }}
So at any point the CS will be aware if a supplied queue path is creatable.
Mapping rules will return a path based on their configuration, but I don't
think the placement rule should be overriding the CS configuration. This can
cause some undesired issues, and we really don't get anything with it. It will
be a little more inconvenient for the user since they have to configure the
mapping rules AND the CS queues properly, but it doesn't look like a big deal.
On the other hand there are multiple places in the code which check if a queue
can be created and all these places use the queue hierarchy to determine
whether a queue is a managed parent, or not, however if the Mapping Rules are
allowed to circumvent this, it might lead to unforeseen inconsistencies. (As a
matter of fact the CSMappingPlacementRule currently uses the information from
CS to determine if a queue can be created or not, so it wouldn't set any of
these flags differently then CS is configured anyway)
So I would say let's omit the creation flags from the
ApoplicationPlacementContext, let the CSMappingPlacementRule rely on the
information provided by the CS about queue creation eligibility, and let it
pass back a "regular" ApplicationPlacementContext with the desired queue path,
then the CS will be able to create it if the configuration allows. This is a
much easier and stable solution, while we don't really put extra administration
overhead to the user, and also can be easily extended in the future.
As I see circumventing protections are never a good idea even if we are talking
about internal interfaces, and these flags would exactly do that, they would
force the CS to create queues which are against it's configuration.
was (Author: shuzirra):
Hi [~gandras] [~wangda],
*1) How we deal with "create" flag of ApplicationPlacementContext?* **
I don't think we need this flag at all. The CapacityScheduler has the
information where can be dynamic queues created, it is supplied via the
configuration option {{queue-path.auto-queue-creation-v2.enabled}}
and
{{queue-path.auto-queue-creation.enabled }}
So at any point the CS will be aware if a supplied queue path is creatable.
Mapping rules will return a path based on their configuration, but I don't
think the placement rule should be overriding the CS configuration. This can
cause some undesired issues, and we really don't get anything with it. It will
be a little more inconvenient for the user since they have to configure the
mapping rules AND the CS queues properly, but it doesn't look like a big deal.
On the other hand there are multiple places in the code which check if a queue
can be created and all these places use the queue hierarchy to determine
whether a queue is a managed parent, or not, however if the Mapping Rules are
allowed to circumvent this, it might lead to unforeseen inconsistencies. (As a
matter of fact the CSMappingPlacementRule currently uses the information from
CS to determine if a queue can be created or not, so it wouldn't set any of
these flags differently then CS is configured anyway)
So I would say let's omit the creation flags from the
ApoplicationPlacementContext, let the CSMappingPlacementRule rely on the
information provided by the CS about queue creation eligibility, and let it
pass back a "regular" ApplicationPlacementContext with the desired queue path,
then the CS will be able to create it if the configuration allows. This is a
much easier and stable solution, while we don't really put extra administration
overhead to the user, and also can be easily extended in the future.
As I see circumventing protections are never a good idea even if we are talking
about internal interfaces, and these flags would exactly do that, they would
force the CS to create queues which are against it's configuration.
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264855#comment-17264855
]
Gergely Pollak commented on YARN-10506:
---
Hi [~gandras] [~wangda],
*1) How we deal with "create" flag of ApplicationPlacementContext?* **
I don't think we need this flag at all. The CapacityScheduler has the
information where can be dynamic queues created, it is supplied via the
configuration option {{queue-path.auto-queue-creation-v2.enabled}}
and
{{queue-path.auto-queue-creation.enabled }}
So at any point the CS will be aware if a supplied queue path is creatable.
Mapping rules will return a path based on their configuration, but I don't
think the placement rule should be overriding the CS configuration. This can
cause some undesired issues, and we really don't get anything with it. It will
be a little more inconvenient for the user since they have to configure the
mapping rules AND the CS queues properly, but it doesn't look like a big deal.
On the other hand there are multiple places in the code which check if a queue
can be created and all these places use the queue hierarchy to determine
whether a queue is a managed parent, or not, however if the Mapping Rules are
allowed to circumvent this, it might lead to unforeseen inconsistencies. (As a
matter of fact the CSMappingPlacementRule currently uses the information from
CS to determine if a queue can be created or not, so it wouldn't set any of
these flags differently then CS is configured anyway)
So I would say let's omit the creation flags from the
ApoplicationPlacementContext, let the CSMappingPlacementRule rely on the
information provided by the CS about queue creation eligibility, and let it
pass back a "regular" ApplicationPlacementContext with the desired queue path,
then the CS will be able to create it if the configuration allows. This is a
much easier and stable solution, while we don't really put extra administration
overhead to the user, and also can be easily extended in the future.
As I see circumventing protections are never a good idea even if we are talking
about internal interfaces, and these flags would exactly do that, they would
force the CS to create queues which are against it's configuration.
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10506-006-10504-010.patch,
> YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch,
> YARN-10506-012.patch, YARN-10506.001.patch, YARN-10506.002.patch,
> YARN-10506.003.patch, YARN-10506.004.patch, YARN-10506.005.patch,
> YARN-10506.006-combined.patch, YARN-10506.006.patch, YARN-10506.007.patch,
> YARN-10506.009.patch, YARN-10506.011.patch
>
>
> The queue creation logic should be updated to use weight mode and support the
> flexible creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7200) SLS generates a realtimetrack.json file but that file is missing the closing ']'
[
https://issues.apache.org/jira/browse/YARN-7200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264853#comment-17264853
]
Agshin Kazimli edited comment on YARN-7200 at 1/14/21, 12:12 PM:
-
[~snemeth] Thanks for your thoughts.
I've changed the error messages to LOG.error().
Regarding the second thought, I believe metricsLogBW calls
org.apache.hadoop.yarn.sls.scheduler.SchedulerMetrics.MetricsLogRunnable is
invoked if the
{code:java}
(SchedulerWrapper) scheduler.getTracker().getQueueSet() != null{code}
{color:#cc7832}{color:#172b4d}in
org.apache.hadoop.yarn.sls.scheduler.SchedulerMetrics.MetricsLogRunnable{color},{color}
{color:#172b4d}which in scheduler is an instance of ResourceScheduler. So, if
there is a tracked application and the queue set is not null, then it'll be
invoked. In this case, if tearDown() is called, then there is no application
remaining, that means, it can't be invoked after that.{color}
But, just in case, I see that there is a boolean field 'running' which
indicates that metrics is running or not, I've added a new code line to the top
of the tearDown() method to set the value of running to false. It means, the
metrics has been stopped, and there is no need to write anything else.
was (Author: akshink):
[~snemeth] Thanks for your thoughts.
I've changed the error messages to LOG.error().
Regarding the second thought, I believe metricsLogBW calls
org.apache.hadoop.yarn.sls.scheduler.SchedulerMetrics.MetricsLogRunnable is
invoked if the
{code:java}
(SchedulerWrapper) scheduler.getTracker().getQueueSet() != null{code}
{color:#cc7832}{color:#172b4d}in
org.apache.hadoop.yarn.sls.scheduler.SchedulerMetrics.MetricsLogRunnable,{color}
{color:#172b4d}which in scheduler is an instance of ResourceScheduler. So, if
there is a tracked application and the queue set is not null, then it'll be
invoked. In this case, if tearDown() is called, then there is no application
remaining, that means, it can't be invoked after that.{color}
{color}
{color:#172b4d}But, just in case, I see that there is a boolean field 'running'
which indicates that metrics is running or not, I've added a new code line to
the top of the tearDown() method to set the value of running to false. It
means, the metrics has been stopped, and there is no need to write anything
else.{color}
> SLS generates a realtimetrack.json file but that file is missing the closing
> ']'
>
>
> Key: YARN-7200
> URL: https://issues.apache.org/jira/browse/YARN-7200
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler-load-simulator
>Reporter: Grant Sohn
>Assignee: Agshin Kazimli
>Priority: Minor
> Labels: newbie, newbie++
> Attachments: YARN-7200-branch-trunk.patch, YARN-7200.002.patch,
> YARN-7200.003.patch, YARN-7200.004.patch, YARN-7200.005.patch,
> snemeth-testing-20201113.zip
>
>
> File
> hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SchedulerMetrics.java
> shows:
> {noformat}
> void tearDown() throws Exception {
> if (metricsLogBW != null) {
> metricsLogBW.write("]");
> metricsLogBW.close();
> }
>
> {noformat}
> So the exit logic is flawed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7200) SLS generates a realtimetrack.json file but that file is missing the closing ']'
[
https://issues.apache.org/jira/browse/YARN-7200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264853#comment-17264853
]
Agshin Kazimli commented on YARN-7200:
--
[~snemeth] Thanks for your thoughts.
I've changed the error messages to LOG.error().
Regarding the second thought, I believe metricsLogBW calls
org.apache.hadoop.yarn.sls.scheduler.SchedulerMetrics.MetricsLogRunnable is
invoked if the
{code:java}
(SchedulerWrapper) scheduler.getTracker().getQueueSet() != null{code}
{color:#cc7832}{color:#172b4d}in
org.apache.hadoop.yarn.sls.scheduler.SchedulerMetrics.MetricsLogRunnable,{color}
{color:#172b4d}which in scheduler is an instance of ResourceScheduler. So, if
there is a tracked application and the queue set is not null, then it'll be
invoked. In this case, if tearDown() is called, then there is no application
remaining, that means, it can't be invoked after that.{color}
{color}
{color:#172b4d}But, just in case, I see that there is a boolean field 'running'
which indicates that metrics is running or not, I've added a new code line to
the top of the tearDown() method to set the value of running to false. It
means, the metrics has been stopped, and there is no need to write anything
else.{color}
> SLS generates a realtimetrack.json file but that file is missing the closing
> ']'
>
>
> Key: YARN-7200
> URL: https://issues.apache.org/jira/browse/YARN-7200
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler-load-simulator
>Reporter: Grant Sohn
>Assignee: Agshin Kazimli
>Priority: Minor
> Labels: newbie, newbie++
> Attachments: YARN-7200-branch-trunk.patch, YARN-7200.002.patch,
> YARN-7200.003.patch, YARN-7200.004.patch, YARN-7200.005.patch,
> snemeth-testing-20201113.zip
>
>
> File
> hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SchedulerMetrics.java
> shows:
> {noformat}
> void tearDown() throws Exception {
> if (metricsLogBW != null) {
> metricsLogBW.write("]");
> metricsLogBW.close();
> }
>
> {noformat}
> So the exit logic is flawed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10572) Merge YARN-8557 and YARN-10352, and rebase based YARN-10380.
[ https://issues.apache.org/jira/browse/YARN-10572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264155#comment-17264155 ] zhuqi edited comment on YARN-10572 at 1/14/21, 9:03 AM: [~wangda] [~bibinchundatt] [~prabhujoseph] I have updated a patch to rebase YARN-10352 and merge the difference in YARN-8557. Also refactor some method. If you could review and merge it? was (Author: zhuqi): [~wangda] [~bibinchundatt] [~dgsunil] I have updated a patch to rebase YARN-10352 and merge the difference in YARN-8557. Also refactor some method. If you could review and merge it? > Merge YARN-8557 and YARN-10352, and rebase based YARN-10380. > > > Key: YARN-10572 > URL: https://issues.apache.org/jira/browse/YARN-10572 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > Attachments: YARN-10572.001.patch > > > The work is : > 1. Because of YARN-10380, We should rebase YARN-10352 > 2. Also merge YARN-8557 for not running case skip. > 3. Refactor some method in YARN-10380 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10572) Merge YARN-8557 and YARN-10352, and rebase based YARN-10380.
[ https://issues.apache.org/jira/browse/YARN-10572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264155#comment-17264155 ] zhuqi edited comment on YARN-10572 at 1/14/21, 9:02 AM: [~wangda] [~bibinchundatt] [~dgsunil] I have updated a patch to rebase YARN-10352 and merge the difference in YARN-8557. Also refactor some method. If you could review and merge it? was (Author: zhuqi): [~wangda] [~bibinchundatt] I have updated a patch to rebase YARN-10352 and merge the difference in YARN-8557. Also refactor some method. If you could review and merge it? > Merge YARN-8557 and YARN-10352, and rebase based YARN-10380. > > > Key: YARN-10572 > URL: https://issues.apache.org/jira/browse/YARN-10572 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > Attachments: YARN-10572.001.patch > > > The work is : > 1. Because of YARN-10380, We should rebase YARN-10352 > 2. Also merge YARN-8557 for not running case skip. > 3. Refactor some method in YARN-10380 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10572) Merge YARN-8557 and YARN-10352, and rebase based YARN-10380.
[ https://issues.apache.org/jira/browse/YARN-10572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264155#comment-17264155 ] zhuqi edited comment on YARN-10572 at 1/14/21, 9:01 AM: [~wangda] [~bibinchundatt] I have updated a patch to rebase YARN-10352 and merge the difference in YARN-8557. Also refactor some method. If you could review and merge it? was (Author: zhuqi): [~leftnoteasy] [~bibinchundatt] I have updated a patch to rebase YARN-10352 and merge the difference in YARN-8557. Also refactor some method. If you could review and merge it? > Merge YARN-8557 and YARN-10352, and rebase based YARN-10380. > > > Key: YARN-10572 > URL: https://issues.apache.org/jira/browse/YARN-10572 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > Attachments: YARN-10572.001.patch > > > The work is : > 1. Because of YARN-10380, We should rebase YARN-10352 > 2. Also merge YARN-8557 for not running case skip. > 3. Refactor some method in YARN-10380 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10572) Merge YARN-8557 and YARN-10352, and rebase based YARN-10380.
[ https://issues.apache.org/jira/browse/YARN-10572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264155#comment-17264155 ] zhuqi edited comment on YARN-10572 at 1/14/21, 9:00 AM: [~leftnoteasy] [~bibinchundatt] I have updated a patch to rebase YARN-10352 and merge the difference in YARN-8557. Also refactor some method. If you could review and merge it? was (Author: zhuqi): [~bibinchundatt] I have updated a patch to rebase YARN-10352 and merge the difference in YARN-8557. Also refactor some method. If you could review and merge it? > Merge YARN-8557 and YARN-10352, and rebase based YARN-10380. > > > Key: YARN-10572 > URL: https://issues.apache.org/jira/browse/YARN-10572 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > Attachments: YARN-10572.001.patch > > > The work is : > 1. Because of YARN-10380, We should rebase YARN-10352 > 2. Also merge YARN-8557 for not running case skip. > 3. Refactor some method in YARN-10380 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264689#comment-17264689
]
Hadoop QA commented on YARN-10506:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
14s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
1s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 3
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
24s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 33s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m
4s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config;
considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
0s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 52s{color}
|
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/480/artifact/out/diff-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt{color}
| {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 generated 2 new + 40
unchanged - 2 fixed = 42 total (was 42) {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 43s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/480/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
| {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 10 new + 761 unchanged - 1 fixed = 771 total (was 762)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
47s{color} | {color:green}{color} |
[jira] [Comment Edited] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264683#comment-17264683 ] zhuqi edited comment on YARN-10506 at 1/14/21, 8:17 AM: Thanks for [~gandras] comment. When submit without enabled mapping rules(auto create leaf/parent ) , the logic will back to submit the normal leafqueue if the leafqueue already exists, i think this is reasonable. If there are other cases AQC will not used by mapping rules, we should use other function rather than autoCreateLeafQueue, and the APC flags is mainly used for mapping rules check. If you any other advice [~wangda] ? was (Author: zhuqi): [~gandras] When submit without enabled mapping rules(auto create leaf/parent ) , the logic will back to submit the normal leafqueue if the leafqueue already exists, i think this is reasonable. If there are other cases AQC will not used by mapping rules, we should use other function rather than autoCreateLeafQueue, and the APC flags is mainly used for mapping rules check. If you any other advice [~wangda] ? > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506-012.patch, YARN-10506.001.patch, YARN-10506.002.patch, > YARN-10506.003.patch, YARN-10506.004.patch, YARN-10506.005.patch, > YARN-10506.006-combined.patch, YARN-10506.006.patch, YARN-10506.007.patch, > YARN-10506.009.patch, YARN-10506.011.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264683#comment-17264683 ] zhuqi commented on YARN-10506: -- [~gandras] When submit without enabled mapping rules(auto create leaf/parent ) , the logic will back to submit the normal leafqueue if the leafqueue already exists, i think this is reasonable. If there are other cases AQC will not used by mapping rules, we should use other function rather than autoCreateLeafQueue, and the APC flags is mainly used for mapping rules check. If you any other advice [~wangda] ? > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506-012.patch, YARN-10506.001.patch, YARN-10506.002.patch, > YARN-10506.003.patch, YARN-10506.004.patch, YARN-10506.005.patch, > YARN-10506.006-combined.patch, YARN-10506.006.patch, YARN-10506.007.patch, > YARN-10506.009.patch, YARN-10506.011.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264673#comment-17264673 ] Andras Gyori commented on YARN-10506: - Thank you [~wangda] and [~zhuqi] for the reviews and fixes! I would still add my comments here, but if we do not want to address it, we can proceed with this patch (I will fix the checkstyle issues): * We are now completely reliant on mapping rules when it comes to AQC. If we turn off mapping rules, the new AQC will simply wont work. Is it desirable? * We are now using redundant checks, namely: ** APC flags ** Configuration flags > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506-012.patch, YARN-10506.001.patch, YARN-10506.002.patch, > YARN-10506.003.patch, YARN-10506.004.patch, YARN-10506.005.patch, > YARN-10506.006-combined.patch, YARN-10506.006.patch, YARN-10506.007.patch, > YARN-10506.009.patch, YARN-10506.011.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
