[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266385#comment-17266385 ] Wangda Tan commented on YARN-10506: --- Verified the test failure is an intermittent one which is not caused by the patch, but I still added a minor change to call mockRM.shutdown to cleanup mockRM before starting a new one (see ver.17). Committed the ver.17 patch to trunk. Thanks for the contribution from [~zhuqi] and [~gandras]! And really appreciate reviews from [~shuzirra], [~pbacsko], and [~bteke]! > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506-012.patch, YARN-10506-013.patch, YARN-10506.001.patch, > YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch, > YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch, > YARN-10506.007.patch, YARN-10506.009.patch, YARN-10506.011.patch, > YARN-10506.014.patch, YARN-10506.015.patch, YARN-10506.016.patch, > YARN-10506.017.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266384#comment-17266384
]
Hadoop QA commented on YARN-10506:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 11s{color}
| {color:red}{color} | {color:red} YARN-10506 does not apply to trunk. Rebase
required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for
help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-10506 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13018848/YARN-10506.017.patch |
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/491/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10506-006-10504-010.patch,
> YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch,
> YARN-10506-012.patch, YARN-10506-013.patch, YARN-10506.001.patch,
> YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch,
> YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch,
> YARN-10506.007.patch, YARN-10506.009.patch, YARN-10506.011.patch,
> YARN-10506.014.patch, YARN-10506.015.patch, YARN-10506.016.patch,
> YARN-10506.017.patch
>
>
> The queue creation logic should be updated to use weight mode and support the
> flexible creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266315#comment-17266315
]
Hadoop QA commented on YARN-10506:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
12s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 2
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
49s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 11s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
48s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 53s{color}
|
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/487/artifact/out/diff-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt{color}
| {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 generated 2 new + 40
unchanged - 2 fixed = 42 total (was 42) {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green}{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 0 new + 738 unchanged - 1 fixed = 738 total (was 739)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:gre
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266292#comment-17266292 ] Wangda Tan commented on YARN-10506: --- Thanks for the additional comments, I +1 to the latest patch, if no objections I will get it in by this afternoon. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506-012.patch, YARN-10506-013.patch, YARN-10506.001.patch, > YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch, > YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch, > YARN-10506.007.patch, YARN-10506.009.patch, YARN-10506.011.patch, > YARN-10506.014.patch, YARN-10506.015.patch, YARN-10506.016.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266265#comment-17266265
]
Hadoop QA commented on YARN-10506:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
25s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 2
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 32m
49s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 53s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
50s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 52s{color}
|
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/485/artifact/out/diff-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt{color}
| {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 generated 1 new + 41
unchanged - 1 fixed = 42 total (was 42) {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 0 new + 737 unchanged - 1 fixed = 737 total (was 738)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
46s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:gre
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266181#comment-17266181 ] Andras Gyori commented on YARN-10506: - Thanks [~zhuqi], you are right, but I realized the whole logic was flawed because of the parent == null check. I have removed this check and added your suggestion, also, fixed the corresponding test. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506-012.patch, YARN-10506-013.patch, YARN-10506.001.patch, > YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch, > YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch, > YARN-10506.007.patch, YARN-10506.009.patch, YARN-10506.011.patch, > YARN-10506.014.patch, YARN-10506.015.patch, YARN-10506.016.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266170#comment-17266170
]
zhuqi commented on YARN-10506:
--
[~gandras]
Here should be changed to :
{code:java}
if (parentQueue != null &&
conf.isAutoCreateChildQueueEnabled(parentQueue.getQueuePath())) {
// Case 1: Handle ManagedParentQueue
AutoCreatedLeafQueue autoCreatedLeafQueue = null;
ManagedParentQueue autoCreateEnabledParentQueue =
(ManagedParentQueue) parentQueue;
autoCreatedLeafQueue = new AutoCreatedLeafQueue(this, leafQueueName,
autoCreateEnabledParentQueue);
addQueue(autoCreatedLeafQueue);
return autoCreatedLeafQueue;
}
{code}
Because parentQueue be null, it should be handled in new auto created, we can
created parent then it may passed. if throw exception, the case will not be
handled.
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10506-006-10504-010.patch,
> YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch,
> YARN-10506-012.patch, YARN-10506-013.patch, YARN-10506.001.patch,
> YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch,
> YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch,
> YARN-10506.007.patch, YARN-10506.009.patch, YARN-10506.011.patch,
> YARN-10506.014.patch, YARN-10506.015.patch
>
>
> The queue creation logic should be updated to use weight mode and support the
> flexible creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266096#comment-17266096 ] Andras Gyori commented on YARN-10506: - Thank you [~shuzirra] for the feedback, I too find out troubling and fixed this concern, and also found an additional line where we used short name instead of full path. As for the rest, I will handle them in the followup refactoring jira, but in general I agree with each of them. Also thank you [~bteke] for the comments, I have removed all the unused variables. [~zhuqi] will reintroduce the fields that are needed in an other jira. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506-012.patch, YARN-10506-013.patch, YARN-10506.001.patch, > YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch, > YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch, > YARN-10506.007.patch, YARN-10506.009.patch, YARN-10506.011.patch, > YARN-10506.014.patch, YARN-10506.015.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266081#comment-17266081
]
Gergely Pollak commented on YARN-10506:
---
Thank you [~gandras] for the fix, here are a few comments, most of them can
wait for a followup JIRA:
*CapacitySchedulerAutoQueueHandler class*:
Why is it a separate class, this is the functionality of the actual queue, I
think it would have been better to extend the parentQueue and add this
functionality there, this is a class which automagicly changes the ParentQueue
class and enhances it with new functionality, but I think it would be better to
solve this via inheritance. Also that would make differentiating between
ParentQueue and the DynamicParentQueue much easier. Probably we should fix it
in a followup JIRA.
*CapacitySchedulerAutoQueueHandler#getQueue*
This method is unnecessary, since CSQueueStore.get method handles null paths,
and returns null. See CapacitySchedulerQueueManager#getQueue and
CSQueueStore#get
*CapacitySchedulerAutoQueueHandler#autoCreateQueue*
I might be wrong, but this is concerning:
{code:java}
CSQueue existingQueueCandidate =
getQueue(queueCandidateContext.getQueue());{code}
The getQueue will always return the leaf name of the queue, which might be
ambiguous so this might return null even if the queue exists on the path we are
checking, and might identify an already existing queue invalid.
As a rule of thumb, wherever possible use the full path of the queues please!
If we have the following structure:
{code:java}
root
+--alice
| +--test
| +--dev
+--bob
| +--test
| +--dev
{code}
Where both _dev_ and _test_ queues are dynamic parents, then when I try to
submit something to root.bob.dev.something, this part of the code
{code:java}
ApplicationPlacementContext queueCandidateContext = parentContext; //
(Parent context is root.bob.dev)
CSQueue existingQueueCandidate =
getQueue(queueCandidateContext.getQueue()); //queueCandidateContext.getQueue()
= 'dev'
getQueue(queueCandidateContext.getQueue()) == null //because dev is
ambiguous
{code}
Will identify root.bob.dev as a not existing queue.
So please use _ApplciationPlacementContext#getFullQueuePath_
*CapacitySchedulerAutoQueueHandler#isDynamicQueue*
Is not fool/future proof, while I'm aware currently only the AbstractCSQueue
class implements the interface, but still it would be better to check before
casting it to AbstractCSQueue. Or move the isDynamic down to the interface,
then you don't have to cast
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10506-006-10504-010.patch,
> YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch,
> YARN-10506-012.patch, YARN-10506-013.patch, YARN-10506.001.patch,
> YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch,
> YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch,
> YARN-10506.007.patch, YARN-10506.009.patch, YARN-10506.011.patch,
> YARN-10506.014.patch
>
>
> The queue creation logic should be updated to use weight mode and support the
> flexible creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266061#comment-17266061 ] zhuqi commented on YARN-10506: -- Thanks for [~gandras] fix. Thanks for [~bteke] review. The lastSubmittedTimestamp, i will use it for YARN-10532. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506-012.patch, YARN-10506-013.patch, YARN-10506.001.patch, > YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch, > YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch, > YARN-10506.007.patch, YARN-10506.009.patch, YARN-10506.011.patch, > YARN-10506.014.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266036#comment-17266036
]
Benjamin Teke commented on YARN-10506:
--
Thanks for working on this! Checked the latest patch, generally seems good to
me. I have two minor nits about it:
* In AbstractCSQueue the field {{lastSubmittedTimestamp}} is unused. It has a
"setter" and a getter, but neither of them are called. Shouldn't this be
removed to avoid confusion?
* The same goes for CapacitySchedulerAutoQueueHandler's conf field.
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10506-006-10504-010.patch,
> YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch,
> YARN-10506-012.patch, YARN-10506-013.patch, YARN-10506.001.patch,
> YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch,
> YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch,
> YARN-10506.007.patch, YARN-10506.009.patch, YARN-10506.011.patch,
> YARN-10506.014.patch
>
>
> The queue creation logic should be updated to use weight mode and support the
> flexible creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266018#comment-17266018
]
Hadoop QA commented on YARN-10506:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
23s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 2
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
49s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 28s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
50s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 53s{color}
|
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/484/artifact/out/diff-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt{color}
| {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 generated 2 new + 40
unchanged - 2 fixed = 42 total (was 42) {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green}{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 0 new + 738 unchanged - 1 fixed = 738 total (was 739)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:gre
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17265868#comment-17265868 ] Andras Gyori commented on YARN-10506: - Thank you [~shuzirra] [~pbacsko] [~zhuqi] [~wangda] for the patches, reviews and discussion details. I have fixed some minor things (comments and findbug fixes), also reintroduced the locking on in autoCreateLeafQueue at LoC 3403, I think it is needed here, but might be subject to change. Apart from this, I have no additional comments +1. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506-012.patch, YARN-10506-013.patch, YARN-10506.001.patch, > YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch, > YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch, > YARN-10506.007.patch, YARN-10506.009.patch, YARN-10506.011.patch, > YARN-10506.014.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17265713#comment-17265713
]
Hadoop QA commented on YARN-10506:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
13s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 2
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
53s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 29s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
51s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 53s{color}
|
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/483/artifact/out/diff-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt{color}
| {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 generated 2 new + 40
unchanged - 2 fixed = 42 total (was 42) {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green}{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 0 new + 738 unchanged - 1 fixed = 738 total (was 739)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:gre
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17265652#comment-17265652 ] Wangda Tan commented on YARN-10506: --- Thanks [~zhuqi], the latest patch looks good to me! I will wait for thoughts from others, if no additional opposite opinions I will get it in by tomorrow. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506-012.patch, YARN-10506-013.patch, YARN-10506.001.patch, > YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch, > YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch, > YARN-10506.007.patch, YARN-10506.009.patch, YARN-10506.011.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17265651#comment-17265651 ] zhuqi commented on YARN-10506: -- I update this and fix the check style in 013 patch. Thanks all for patient review. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506-012.patch, YARN-10506-013.patch, YARN-10506.001.patch, > YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch, > YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch, > YARN-10506.007.patch, YARN-10506.009.patch, YARN-10506.011.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17265618#comment-17265618 ] zhuqi commented on YARN-10506: -- It makes sense to me. Thanks to [~shuzirra] [~pbacsko] for your patient comment to make it clear. [~wangda] i will update according to the latest proposal. *1) yarn.scheduler.capacity..auto-queue-creation-v2.enabled* To make sure the flag is right set. *2) Removing "create" from the ApplicationPlacementContext:* Remove it, and change related test. Make the property inheritable or not, will be another jira. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506-012.patch, YARN-10506.001.patch, YARN-10506.002.patch, > YARN-10506.003.patch, YARN-10506.004.patch, YARN-10506.005.patch, > YARN-10506.006-combined.patch, YARN-10506.006.patch, YARN-10506.007.patch, > YARN-10506.009.patch, YARN-10506.011.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17265521#comment-17265521 ] Wangda Tan commented on YARN-10506: --- Sounds good, thanks [~shuzirra], [~pbacsko]. It makes sense to me, [~zhuqi] can you add your thoughts about the latest proposal? Thanks, > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506-012.patch, YARN-10506.001.patch, YARN-10506.002.patch, > YARN-10506.003.patch, YARN-10506.004.patch, YARN-10506.005.patch, > YARN-10506.006-combined.patch, YARN-10506.006.patch, YARN-10506.007.patch, > YARN-10506.009.patch, YARN-10506.011.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17265208#comment-17265208
]
Gergely Pollak commented on YARN-10506:
---
Actually the queue creation checks were present in the legacy engine
(https://github.com/apache/hadoop/blob/9d5144e66d71551ad6e1a8d3f1670e49ba181999/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java#L333-L340),
if I see correctly since 3.0. But the first commit seems much older:
[https://github.com/apache/hadoop/commit/0987a7b8c2c1e4c2095821d98a7db19644df]
.
I've just kept it during the refactor, and as Peter pointed out now we actually
build on it, since we need to be able to determine if a rule will be executed
successfully (ie. the queue can be created), and if it cannot we can move onto
the next rule. This fallback behaviour is part of the FS Placement engine as
well, so we need it in order to server FS customers, so it was somewhat lucky
that it was already in place. (In the original implementation we just used this
check to throw exceptions, which was quite pointless, since the CS would have
rejected the application if the queue was uncreatable anyway)
So this is why I say we can safely remove those flags from the application
placement context, this way we can reduce the code complexity.
This way we only need what [~pbacsko] mentioned:
{code:java}
1) "create" flag on the rule itself in the JSON - true/false
2) "yarh.scheduler.capacity..auto-queue-creation-v2.enabled" -
true/false{code}
Also I'm currently adopting this change in the CSMappingPlacementRule, and I
don't use these flags. What I'll need in the future a better, centralised way
to determine if a path can be created, and both CS and the MappingEngine should
use the same methods, to do it. Currently we are implementing the same logic
differently, and it's much harder to maintain the code this way, but during the
followup code cleanup jiras we'll take care of this.
About the race condition, I think (and I might be very wrong, because race
conditions can be really sneaky), we cannot do much about, theoretically it can
happen that the Mapping engine determines a rule can be executed, because the
target queue exists or there is a proper dynamic parent, then a config change
occurs, and by the time we get to the actual creation the queue structure
changes, at this point the rejection is the proper action to take since the
submission IS against the configuration. And I don't see a way around it since
at the time the MappingEngine makes the decision that decision is the correct
one, but we shouldn't enforce this decision if the configuration changes. But
this race condition should only occur when the user changes the queue
configuration, and in this case they should expect failing application
submissions for the queues which have been changed or removed.
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10506-006-10504-010.patch,
> YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch,
> YARN-10506-012.patch, YARN-10506.001.patch, YARN-10506.002.patch,
> YARN-10506.003.patch, YARN-10506.004.patch, YARN-10506.005.patch,
> YARN-10506.006-combined.patch, YARN-10506.006.patch, YARN-10506.007.patch,
> YARN-10506.009.patch, YARN-10506.011.patch
>
>
> The queue creation logic should be updated to use weight mode and support the
> flexible creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17265149#comment-17265149
]
Wangda Tan commented on YARN-10506:
---
[~pbacsko], I see, I think it makes sense.
{quote}The logic must be there, otherwise we can't proceed to the next rule if
the fallback is set to "skip". If you perform the validations (which are
currently done by {{CSMappingPlacementRule}}) outside of the placement engine,
you lose the ability to execute different actions (skip/reject/place to
default).
{quote}
Because it is different from the previous CS queue mapping: previously, when a
rule matches, it up to the scheduler to make a decision such as create queue,
accept or reject the app.
Now we need to make sure rule engine makes the right decision before sending
the request to scheduler. We need to be careful about possible racing
conditions. But I think you're right, when admin change
.auto-queue-creation-v2.enabled in the queue hierarchy, it is possible
to cause undesirable result. But syncing between the two systems (rule engine
and scheduler) will have delays in any case, so removing create flag from
ApplicationPlacementContext sounds not bad.
I will let [~zhuqi], [~gandras] and you guys to decide what is the best
approach to take, I'm OK with either approaches. But let's reach a conclusion
fast since this patch blocks a number of follow up tickets.
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10506-006-10504-010.patch,
> YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch,
> YARN-10506-012.patch, YARN-10506.001.patch, YARN-10506.002.patch,
> YARN-10506.003.patch, YARN-10506.004.patch, YARN-10506.005.patch,
> YARN-10506.006-combined.patch, YARN-10506.006.patch, YARN-10506.007.patch,
> YARN-10506.009.patch, YARN-10506.011.patch
>
>
> The queue creation logic should be updated to use weight mode and support the
> flexible creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17265144#comment-17265144
]
Peter Bacsko commented on YARN-10506:
-
"_It is possible when the rule checks, the queue allow creation, but when
ApplicationPlacementContext arrived at scheduler, the queue is refreshed"_
This indeed sounds like a race condition, but if the queue is removed in the
meantime, then the submission would fail anyway. The refresh itself is perfomed
while holding a write lock, so that in itself should not be a problem.
"RuleEngine can just pass ApplicationPLacementContext to Scheduler and let
scheduler making the decision"
Could you elaborate on what "decision making" you're referring to? I think
inside the engine, it's important that it has to determine whether the target
queue is available or not, if not, can we create it, etc. The logic must be
there, otherwise we can't proceed to the next rule if the fallback is set to
"skip". If you perform the validations (which are currently done by
{{CSMappingPlacementRule}}) outside of the placement engine, you lose the
ability to execute different actions (skip/reject/place to default).
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10506-006-10504-010.patch,
> YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch,
> YARN-10506-012.patch, YARN-10506.001.patch, YARN-10506.002.patch,
> YARN-10506.003.patch, YARN-10506.004.patch, YARN-10506.005.patch,
> YARN-10506.006-combined.patch, YARN-10506.006.patch, YARN-10506.007.patch,
> YARN-10506.009.patch, YARN-10506.011.patch
>
>
> The queue creation logic should be updated to use weight mode and support the
> flexible creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17265085#comment-17265085 ] Wangda Tan commented on YARN-10506: --- Thanks [~shuzirra], [~pbacsko], posting this comment while in the middle of something else, I hope I didn't miss important information. Here you shared several things: *1) Add a create flag on the rule itself in the JSON:* I agreed. *2) yarh.scheduler.capacity..auto-queue-creation-v2.enabled* I agreed. Also, I agree we should look at if we can make the property inheritable or not, I agree but I suggest to move the implementation to a separate Jira. *3) Removing "create" from the ApplicationPlacementContext:* I initially thought it is needed because the rule engine may not do exhaust checks and the concurrency issue. _(It is possible when the rule checks, the queue allow creation, but when ApplicationPlacementContext arrived at scheduler, the queue is refreshed)_. So I think it is still valuable to let the scheduler do the creation based on demand, and fail the App submission atomically. Also, if we relies on Scheduler to check creatable or not, it can reduce complexities of RuleEngine to do additional check. RuleEngine can just pass ApplicationPLacementContext to Scheduler and let scheduler making the decision. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506-012.patch, YARN-10506.001.patch, YARN-10506.002.patch, > YARN-10506.003.patch, YARN-10506.004.patch, YARN-10506.005.patch, > YARN-10506.006-combined.patch, YARN-10506.006.patch, YARN-10506.007.patch, > YARN-10506.009.patch, YARN-10506.011.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264977#comment-17264977 ] Peter Bacsko commented on YARN-10506: - *Clarification comment* I turned out I misunderstood [~shuzirra]'s comment above. I thought he was speaking about the "create" flag the comes from the rule itself. I should have read it more carefully. Anyway we just had an internal meeting so now I have a better understanding. _"How we deal with "create" flag of ApplicationPlacementContext?"_ I don't think we need _any_ flags in this class. We need two settings basically: 1) "create" flag on the rule itself in the JSON - true/false 2) "yarh.scheduler.capacity..auto-queue-creation-v2.enabled" - true/false The placement engine (even the old one) can access both to make a decision about whether the queue is allowed to be created or not. If it is, it returns the full path inside the "ApplicationPlacementContext". Otherwise it decides what to do by checking the "fallbackResult" setting of the rule: returns "root.default", proceeds to the next rule or rejects the application. In fact, Gergo told me that *the new engine already works the way I proposed*. We don't need any additional input or flag to make an explicit, valid decision about the placement. Also, [~gandras] + [~snemeth] reached the same conclusion. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506-012.patch, YARN-10506.001.patch, YARN-10506.002.patch, > YARN-10506.003.patch, YARN-10506.004.patch, YARN-10506.005.patch, > YARN-10506.006-combined.patch, YARN-10506.006.patch, YARN-10506.007.patch, > YARN-10506.009.patch, YARN-10506.011.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264903#comment-17264903
]
Peter Bacsko commented on YARN-10506:
-
Hi guys. I haven't really contributed to this patch and there's some big
discussion going on here, but I'm adding my 2 cents.
To give you some background: I introduced the "create" flag for the new mapping
rules, simply because FS has the same - auto-creation depends on the rule and
users who migrate from FS will find this intuitive, even if the rule format has
changed drastically. Now, I was under the impression that with weight mode and
AQC, we're trying to approximate FS behaviour as much as possible, because the
whole "project" came up in the context on FS->CS migration.
So I thought that in this mode, users are allowed to create queues anywhere in
the hierarchy, eg. with {{}} rule, you can definitely do this in FS,
there is no restriction, at least I'm not aware of it. The FS property
"allow-undeclared-pools" is translated internally to {{}} rule.
Again, the was the idea.
On the other hand, it's also true that if we go this way, legacy percentage
mode and weight mode will diverge significantly, so the behavior will be
totally different. In pct/absolute mode, you can explicitly say where queues
can be created, but if a user switches to weight mode, they cannot do this,
which might be surprising for them (or maybe not - this is a pure speculation
at this point).
If we really want to limit users where they can create queues (as I can see
this is clearly the goal here) then this leads us to the problem what Gergely
just discussed above, have a "create" flag in the rule and have a similar
property which restricts whether a queue can be created, this can lead to some
unexpected behavior, especially if their priority is reversed. At the very
least, legacy FS users will be confused like hell IMO.
We also have to think about the following scenario: a user who just migrated
from FS, still wants the exact FS auto-queue semantics and uses
"allow-undeclared-pools" or a {{}} rule. So if that's the case, we
have to be able to set this up easily. That is, if they have 200 queues and
they have to set this property for _every single queue_, I can easily imagine
that their head will explode. So we should consider having the property
"auto-queue-creation-v2.enabled" as a setting which is automatically inherited
by all child queues, recursively. Or choose the default value to be "true"
which is probably the simplest thing to do.
My conclusion is:
1) I think we have to keep the "create" flag on a rule level if we want FS
rules to work the same way. Some rules might have create=true, some of them
might not, so we can't rely on having the queue properties alone.
2) If we want to introduce "auto-queue-creation-v2.enabled", it's important to
think about whether to make it inheritable or not. If it's not, then this can
potentially cause a huge pain for some users. Alternative approach: have a
separate property which applies to the whole hierarchy globally, which can be
overridden on a per-queue basis, this is a very common approach in CS.
3) I very much agree with this statement: _"Mapping rules will return a path
based on their configuration, but I don't think the placement rule should be
overriding the CS configuration"_. If we truly want both, then the queue
property should have its final say about the creation. The rule says "let's
create this queue if it doesn't exist", but then we must check
"auto-queue-creation-v2.enabled" for the target path to see if the queue can be
created at the given point in the hierarchy. I should not be the other way
around.
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10506-006-10504-010.patch,
> YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch,
> YARN-10506-012.patch, YARN-10506.001.patch, YARN-10506.002.patch,
> YARN-10506.003.patch, YARN-10506.004.patch, YARN-10506.005.patch,
> YARN-10506.006-combined.patch, YARN-10506.006.patch, YARN-10506.007.patch,
> YARN-10506.009.patch, YARN-10506.011.patch
>
>
> The queue creation logic should be updated to use weight mode and support the
> flexible creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264855#comment-17264855
]
Gergely Pollak commented on YARN-10506:
---
Hi [~gandras] [~wangda],
*1) How we deal with "create" flag of ApplicationPlacementContext?* **
I don't think we need this flag at all. The CapacityScheduler has the
information where can be dynamic queues created, it is supplied via the
configuration option {{queue-path.auto-queue-creation-v2.enabled}}
and
{{queue-path.auto-queue-creation.enabled }}
So at any point the CS will be aware if a supplied queue path is creatable.
Mapping rules will return a path based on their configuration, but I don't
think the placement rule should be overriding the CS configuration. This can
cause some undesired issues, and we really don't get anything with it. It will
be a little more inconvenient for the user since they have to configure the
mapping rules AND the CS queues properly, but it doesn't look like a big deal.
On the other hand there are multiple places in the code which check if a queue
can be created and all these places use the queue hierarchy to determine
whether a queue is a managed parent, or not, however if the Mapping Rules are
allowed to circumvent this, it might lead to unforeseen inconsistencies. (As a
matter of fact the CSMappingPlacementRule currently uses the information from
CS to determine if a queue can be created or not, so it wouldn't set any of
these flags differently then CS is configured anyway)
So I would say let's omit the creation flags from the
ApoplicationPlacementContext, let the CSMappingPlacementRule rely on the
information provided by the CS about queue creation eligibility, and let it
pass back a "regular" ApplicationPlacementContext with the desired queue path,
then the CS will be able to create it if the configuration allows. This is a
much easier and stable solution, while we don't really put extra administration
overhead to the user, and also can be easily extended in the future.
As I see circumventing protections are never a good idea even if we are talking
about internal interfaces, and these flags would exactly do that, they would
force the CS to create queues which are against it's configuration.
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10506-006-10504-010.patch,
> YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch,
> YARN-10506-012.patch, YARN-10506.001.patch, YARN-10506.002.patch,
> YARN-10506.003.patch, YARN-10506.004.patch, YARN-10506.005.patch,
> YARN-10506.006-combined.patch, YARN-10506.006.patch, YARN-10506.007.patch,
> YARN-10506.009.patch, YARN-10506.011.patch
>
>
> The queue creation logic should be updated to use weight mode and support the
> flexible creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264689#comment-17264689
]
Hadoop QA commented on YARN-10506:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
14s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
1s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 3
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
24s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 33s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m
4s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config;
considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
0s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 52s{color}
|
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/480/artifact/out/diff-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt{color}
| {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 generated 2 new + 40
unchanged - 2 fixed = 42 total (was 42) {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 43s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/480/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
| {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 10 new + 761 unchanged - 1 fixed = 771 total (was 762)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
47s{color} | {color:
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264683#comment-17264683 ] zhuqi commented on YARN-10506: -- [~gandras] When submit without enabled mapping rules(auto create leaf/parent ) , the logic will back to submit the normal leafqueue if the leafqueue already exists, i think this is reasonable. If there are other cases AQC will not used by mapping rules, we should use other function rather than autoCreateLeafQueue, and the APC flags is mainly used for mapping rules check. If you any other advice [~wangda] ? > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506-012.patch, YARN-10506.001.patch, YARN-10506.002.patch, > YARN-10506.003.patch, YARN-10506.004.patch, YARN-10506.005.patch, > YARN-10506.006-combined.patch, YARN-10506.006.patch, YARN-10506.007.patch, > YARN-10506.009.patch, YARN-10506.011.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264673#comment-17264673 ] Andras Gyori commented on YARN-10506: - Thank you [~wangda] and [~zhuqi] for the reviews and fixes! I would still add my comments here, but if we do not want to address it, we can proceed with this patch (I will fix the checkstyle issues): * We are now completely reliant on mapping rules when it comes to AQC. If we turn off mapping rules, the new AQC will simply wont work. Is it desirable? * We are now using redundant checks, namely: ** APC flags ** Configuration flags > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506-012.patch, YARN-10506.001.patch, YARN-10506.002.patch, > YARN-10506.003.patch, YARN-10506.004.patch, YARN-10506.005.patch, > YARN-10506.006-combined.patch, YARN-10506.006.patch, YARN-10506.007.patch, > YARN-10506.009.patch, YARN-10506.011.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264666#comment-17264666 ] zhuqi commented on YARN-10506: -- Thanks for [~wangda] patient review. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506-012.patch, YARN-10506.001.patch, YARN-10506.002.patch, > YARN-10506.003.patch, YARN-10506.004.patch, YARN-10506.005.patch, > YARN-10506.006-combined.patch, YARN-10506.006.patch, YARN-10506.007.patch, > YARN-10506.009.patch, YARN-10506.011.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264661#comment-17264661 ] Wangda Tan commented on YARN-10506: --- Thanks [~zhuqi], I don't have further comments, +1. [~gandras] can you share your thoughts on the latest patch? If no further objections, I plan to get the patch committed by tomorrow. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506-012.patch, YARN-10506.001.patch, YARN-10506.002.patch, > YARN-10506.003.patch, YARN-10506.004.patch, YARN-10506.005.patch, > YARN-10506.006-combined.patch, YARN-10506.006.patch, YARN-10506.007.patch, > YARN-10506.009.patch, YARN-10506.011.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264616#comment-17264616
]
zhuqi commented on YARN-10506:
--
[~wangda] [~gandras]
I have updated a patch according to [~wangda] comment, i think it's more
reasonable in
*1) How we deal with "create" flag of ApplicationPlacementContext?*
1.autoCreateLeafQueue:
{code:java}
private LeafQueue autoCreateLeafQueue(
ApplicationPlacementContext placementContext)
throws IOException, YarnException {
String leafQueueName = placementContext.getQueue();
String parentQueueName = placementContext.getParentQueue();
ApplicationPlacementContext apc =
new ApplicationPlacementContext(placementContext);
if (!StringUtils.isEmpty(parentQueueName)) {
CSQueue parentQueue = getQueue(parentQueueName);
CSQueue leafQueue = getQueue(leafQueueName);
// Check if parent should not be null
// when cant't auto create parent.
if (parentQueue == null && !apc.isCreateParentQueue()) {
throw new SchedulerDynamicEditException(
"Could not auto-create leaf queue for " + leafQueueName
+ ". Queue mapping specifies an invalid parent queue "
+ "which does not exist or can't auto create" +
parentQueueName);
}
if (parentQueue != null &&
conf.isAutoCreateChildQueueEnabled(parentQueue.getQueuePath())) {
// Case 1: Handle ManagedParentQueue
AutoCreatedLeafQueue autoCreatedLeafQueue = null;
ManagedParentQueue autoCreateEnabledParentQueue =
(ManagedParentQueue) parentQueue;
autoCreatedLeafQueue = new AutoCreatedLeafQueue(this, leafQueueName,
autoCreateEnabledParentQueue);
addQueue(autoCreatedLeafQueue);
return autoCreatedLeafQueue;
} else {
// Now the parentQueue will not be null later in autoCreateQueue.
// Because that: if parentQueue null, apc.isCreateParentQueue() is
enabled.
// Otherwise the parentQueue will not be null.
// Just check the leafQueue will handle special case.
if (!apc.isCreateLeafQueue()) {
if (leafQueue != null && leafQueue instanceof LeafQueue) {
return (LeafQueue) leafQueue;
} else {
throw new SchedulerDynamicEditException(
"Could not auto-create leaf queue for " + leafQueueName
+ "which auto create leaf not enabed. " +
"Meanwhile it does not exist or not a leaf queue.");
}
}
// Now can create parent with leaf.
// Other check will be in autoCreateQueue.
return autoQueueHandler.autoCreateQueue(apc);
}
}
throw new SchedulerDynamicEditException(
"Could not auto-create leaf queue for " + leafQueueName
+ ". Queue mapping does not specify"
+ " which parent queue it needs to be created under.");
}
}
{code}
I handle all the case, may some corner case i don't mind it.
I also fill the test cases in
{code:java}
@Test
public void testAutoQueueCreationOnAppSubmission() throws Exception {
startScheduler();
createBasicQueueStructureAndValidate();
// Parent exists, allow create leaf will pass
submitApp(cs, USER0, USER0, "root.e-auto", true, false);
AbstractCSQueue e = (AbstractCSQueue) cs.getQueue("root.e-auto");
Assert.assertNotNull(e);
Assert.assertTrue(e.isDynamicQueue());
AbstractCSQueue user0 = (AbstractCSQueue) cs.getQueue(
"root.e-auto." + USER0);
Assert.assertNotNull(user0);
Assert.assertTrue(user0.isDynamicQueue());
// Parent not exists (null), allow create parent and allow create leaf will
pass
submitApp(cs, USER1, USER1, "root.e-auto2", true, true);
AbstractCSQueue e2 = (AbstractCSQueue) cs.getQueue("root.e-auto2");
Assert.assertNotNull(e2);
Assert.assertTrue(e2.isDynamicQueue());
AbstractCSQueue user1 = (AbstractCSQueue) cs.getQueue(
"root.e-auto2." + USER1);
Assert.assertNotNull(user0);
Assert.assertTrue(user1.isDynamicQueue());
// Parent exists, don't allow create leaf, but leaf queue exists will pass
submitApp(cs, USER0, USER0, "root.e-auto", false, false);
Assert.assertNotNull(e);
Assert.assertTrue(e.isDynamicQueue());
Assert.assertNotNull(user0);
Assert.assertTrue(user0.isDynamicQueue());
// Parent not exists, don't allow create parent will not pass
submitApp(cs, USER2, USER2, "root.e-auto3", true, false);
// e3 will be null
AbstractCSQueue e3 = (AbstractCSQueue) cs.getQueue("root.e-auto3");
Assert.assertNull(e3);
AbstractCSQueue user3 = (AbstractCSQueue) cs.getQueue(
"root.e-auto3." + USER1);
// user3 will be null
Assert.assertNull(user3);
}
{code}
Add a submit with auto create leaf/parent flag in
TestCapacitySchedulerAutoCreatedQueueBase.
Reg. *2) How we deal with the queue's auto-queue-creation con
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264397#comment-17264397
]
Wangda Tan commented on YARN-10506:
---
Reg. *2) How we deal with the queue's auto-queue-creation configuration flag?*
Can we rename the property to {{queue-path.auto-queue-creation-v2.enabled}} ?
I'm looking for an approach to more distinguished from the older one.
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10506-006-10504-010.patch,
> YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch,
> YARN-10506.001.patch, YARN-10506.002.patch, YARN-10506.003.patch,
> YARN-10506.004.patch, YARN-10506.005.patch, YARN-10506.006-combined.patch,
> YARN-10506.006.patch, YARN-10506.007.patch, YARN-10506.009.patch,
> YARN-10506.011.patch
>
>
> The queue creation logic should be updated to use weight mode and support the
> flexible creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264390#comment-17264390
]
Wangda Tan commented on YARN-10506:
---
[~zhuqi], [~gandras],
I just took a look at the latest patch, here's my comment:
I think we still need to make a conclusion for the following items:
*1) How we deal with "create" flag of ApplicationPlacementContext?*
Based on latest patch, we have two flags added to ApplicationPlacementContext.
But we only do one
{code:java}
if (apc.isCreateLeafQueue()
|| apc.isCreateParentQueue()) {
...
LeafQueue lq =
autoQueueHandler.autoCreateQueue(apc);
} {code}
And we hardcoded the two values:
{code:java}
apc.setCreateParentQueue(true);
apc.setCreateLeafQueue(true); {code}
To me, It is not sufficient, we need to check inside the handler:
{code:java}
if (apc.isCreateParentQueue()) {
createParentQueue()
}
if (apc.isCreatedLeafQueue()) {
createLeafQueue()
}
{code}
We should add tests for that because it is contract for future integration, we
should have the following test cases:
{code:java}
1) when createLeaf = false, createParent = false:
1.1 When both Leaf doesn't exist or Parent doesn't exist: Application will
be rejected.
1.2 When Parent exists but Leaf doesnt't exist: Application will be
rejected.
1.3 When both exists, application will be accepted
2) Other combinations ..{code}
If we can abstract common test functionality, we should be able to do the
testing without too much-duplicated code.
Can we do it with this patch? *I don't want to delay this (to a separate Jira)
because once another feature integration happens (such as from Queue placement
policy), we will face issues and will cause further delays.*
*2) How we deal with the queue's auto-queue-creation configuration flag?*
I think we can create a flag for c-s.xml to enable auto create queue for each
parent now, but I felt we need to change it later. As far as we get
functionality correct, I'm OK with pushing this to a follow-up patch.
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10506-006-10504-010.patch,
> YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch,
> YARN-10506.001.patch, YARN-10506.002.patch, YARN-10506.003.patch,
> YARN-10506.004.patch, YARN-10506.005.patch, YARN-10506.006-combined.patch,
> YARN-10506.006.patch, YARN-10506.007.patch, YARN-10506.009.patch,
> YARN-10506.011.patch
>
>
> The queue creation logic should be updated to use weight mode and support the
> flexible creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264332#comment-17264332 ] Andras Gyori commented on YARN-10506: - I think javac warnings are unrelated (we did not touch any of the lines listed there). > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506.001.patch, YARN-10506.002.patch, YARN-10506.003.patch, > YARN-10506.004.patch, YARN-10506.005.patch, YARN-10506.006-combined.patch, > YARN-10506.006.patch, YARN-10506.007.patch, YARN-10506.009.patch, > YARN-10506.011.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264238#comment-17264238 ] Wangda Tan commented on YARN-10506: --- Can we also take care of javac warnings? > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506.001.patch, YARN-10506.002.patch, YARN-10506.003.patch, > YARN-10506.004.patch, YARN-10506.005.patch, YARN-10506.006-combined.patch, > YARN-10506.006.patch, YARN-10506.007.patch, YARN-10506.009.patch, > YARN-10506.011.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264237#comment-17264237
]
Wangda Tan commented on YARN-10506:
---
Thanks [~zhuqi] and [~gandras] for quick updates!
*I will review the latest patch (in detail) during my day time*, and to the
question:
{quote} However, I think this method should not take into account which mode
the parent is in, this should be handled outside of this. I think it is safe to
assume, that for empty queues just return WEIGHT, because it is more
restrictive, than PERCENTAGE mode. Lets wait for the opinion of [~wangda] about
it as well
{quote}
I would agree with the statement. We can revisit how we can do better to
distinguish WEIGHT, PERCENTAGE, ABS configuration, which needs additional
cleanup and refactoring. So far, I think it is good for this patch.
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10506-006-10504-010.patch,
> YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch,
> YARN-10506.001.patch, YARN-10506.002.patch, YARN-10506.003.patch,
> YARN-10506.004.patch, YARN-10506.005.patch, YARN-10506.006-combined.patch,
> YARN-10506.006.patch, YARN-10506.007.patch, YARN-10506.009.patch,
> YARN-10506.011.patch
>
>
> The queue creation logic should be updated to use weight mode and support the
> flexible creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264210#comment-17264210
]
Hadoop QA commented on YARN-10506:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
30s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
1s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 2
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
34s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 33s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
58s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 58s{color}
|
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/476/artifact/out/diff-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt{color}
| {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 generated 2 new + 40
unchanged - 2 fixed = 42 total (was 42) {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
48s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 44s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/476/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
| {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 5 new + 738 unchanged - 1 fixed = 743 total (was 739)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:g
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264151#comment-17264151 ] Andras Gyori commented on YARN-10506: - Thanks [~zhuqi] for the insights, it is indeed a good question. However, I think this method should not take into account which mode the parent is in, this should be handled outside of this. I think it is safe to assume, that for empty queues just return WEIGHT, because it is more restrictive, than PERCENTAGE mode. Lets wait for the opinion of [~wangda] about it as well. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506.001.patch, YARN-10506.002.patch, YARN-10506.003.patch, > YARN-10506.004.patch, YARN-10506.005.patch, YARN-10506.006-combined.patch, > YARN-10506.006.patch, YARN-10506.007.patch, YARN-10506.009.patch, > YARN-10506.011.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264113#comment-17264113 ] zhuqi commented on YARN-10506: -- Thanks for [~gandras] for the fix. Another question is when parent is percentage, there are no child queues, we will always return weight in this change, if this is reasonable? If this case will happen in setChildQueues? The test case is about the above question. If you any advice? > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506.001.patch, YARN-10506.002.patch, YARN-10506.003.patch, > YARN-10506.004.patch, YARN-10506.005.patch, YARN-10506.006-combined.patch, > YARN-10506.006.patch, YARN-10506.007.patch, YARN-10506.009.patch, > YARN-10506.011.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264108#comment-17264108 ] Andras Gyori commented on YARN-10506: - Uploaded v11 revision of patch, in which the following changes were made: * Fixed the findbug issue with copy constructor instead of clone * Fixed numerous checkstyle issues * Simplified logic in getCapacityConfigurationTypeForQueues. This broke a lot of tests. The logic is the following: Weight mode if no child queues and if weight mode is used for siblings, abs mode when it is used, and percentage mode in every other cases. What we still need to do: * Property name and ApplicationPlacementContext behavior (let it drive the logic or not) * Also I could not think of a test case, where we have an empty static queue. I think it is not possible. For empty dynamic parentqueues, I think we already cover this case when we create a 2 levels queue (In case of root.a.a1-auto.a2-auto, a1-auto will be a ParentQueue with 0 children, when we assign a2-auto to it) > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506.001.patch, YARN-10506.002.patch, YARN-10506.003.patch, > YARN-10506.004.patch, YARN-10506.005.patch, YARN-10506.006-combined.patch, > YARN-10506.006.patch, YARN-10506.007.patch, YARN-10506.009.patch, > YARN-10506.011.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264071#comment-17264071
]
Hadoop QA commented on YARN-10506:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
3s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
1s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 2
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 27m
44s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
21m 8s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
55s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
52s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 59s{color}
|
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/475/artifact/out/diff-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt{color}
| {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 generated 2 new + 40
unchanged - 2 fixed = 42 total (was 42) {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 44s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/475/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
| {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 30 new + 738 unchanged - 1 fixed = 768 total (was 739)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264048#comment-17264048 ] zhuqi commented on YARN-10506: -- [~gandras] Regarding 2.2 atomicity: The last instanceof Parent check should have never occurred, because then we have already made every necessary check. I have only included that step because I was reluctant to use a casting without an explicit check. I find it too, so just change to the one class. Copy constructor is a good idea. about : Regarding CSQueueUtils#extractQueuePath. I think we should keep this here, because I have seen the same thing repeated over and over across the code base, so I will instead address this in the cleanup jira. I think this is the minor things, we my should deal with the major things first. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506.001.patch, YARN-10506.002.patch, YARN-10506.003.patch, > YARN-10506.004.patch, YARN-10506.005.patch, YARN-10506.006-combined.patch, > YARN-10506.006.patch, YARN-10506.007.patch, YARN-10506.009.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264036#comment-17264036 ] Andras Gyori commented on YARN-10506: - Regarding CSQueueUtils#extractQueuePath. I think we should keep this here, because I have seen the same thing repeated over and over across the code base, so I will instead address this in the cleanup jira. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506.001.patch, YARN-10506.002.patch, YARN-10506.003.patch, > YARN-10506.004.patch, YARN-10506.005.patch, YARN-10506.006-combined.patch, > YARN-10506.006.patch, YARN-10506.007.patch, YARN-10506.009.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264027#comment-17264027 ] Andras Gyori commented on YARN-10506: - Regarding 2.2 atomicity: The last instanceof Parent check should have never occurred, because then we have already made every necessary check. I have only included that step because I was reluctant to use a casting without an explicit check. This is rendered obsolete, however, due to the fix [~zhuqi] introduced. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506.001.patch, YARN-10506.002.patch, YARN-10506.003.patch, > YARN-10506.004.patch, YARN-10506.005.patch, YARN-10506.006-combined.patch, > YARN-10506.006.patch, YARN-10506.007.patch, YARN-10506.009.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264018#comment-17264018 ] zhuqi commented on YARN-10506: -- Thank you [~gandras] for the comment, about the ApplicationPlacementContext part: I agree with : The auto queue creation should not be used when mapping rules are turned off, therefore placement context driven AQC would be a limitation. But the original AQC with capacity, use the conf.isAutoCreateChildQueueEnabled for check. If we should auto create for queue mapping, we don't use ApplicationPlacementContext to check, we can't consistent with original logic, and autoCreateLeafQueue will always go to else when conf.isAutoCreateChildQueueEnabled is false. W'd better to add a flag also if we don't want to use ApplicationPlacementContext check, or just let something like isEligibleForAutoQueueCreation ahead to autoCreateLeafQueue, which meets the consistent auto create check for (ApplicationPlacementContext auto creation, or other based auto creation). And i forget to change this in update patch: The same follow up patch should also clean up addQueue() method of ResourceScheduler. It is only used by CapacitySchedulerPlanFollow, we don't need to add it to the abstract class. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506.001.patch, YARN-10506.002.patch, YARN-10506.003.patch, > YARN-10506.004.patch, YARN-10506.005.patch, YARN-10506.006-combined.patch, > YARN-10506.006.patch, YARN-10506.007.patch, YARN-10506.009.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264000#comment-17264000 ] Andras Gyori commented on YARN-10506: - Thank you [~zhuqi] for the patch, I am going to build the remaining solutions on your approach. However, check out my comments regarding the ApplicationPlacementContext part, I am interested in your opinion/feedback about it. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506.001.patch, YARN-10506.002.patch, YARN-10506.003.patch, > YARN-10506.004.patch, YARN-10506.005.patch, YARN-10506.006-combined.patch, > YARN-10506.006.patch, YARN-10506.007.patch, YARN-10506.009.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263999#comment-17263999 ] Andras Gyori commented on YARN-10506: - Thank you [~wangda] for the review. My comments: # My initial approach was exactly this, however, did not want to extend the scope of this patch. Will refactor the mentioned parts in a followup jira. # This was the approach at the beginning, however, we talked about this with [~shuzirra], and came to the conclusion, that: ## The auto queue creation should be used when mapping rules are turned off, therefore placement context driven AQC would be a limitation We also concluded, that its not the responsibility of the mapping rule/PlacementContext to decide whether AQC is enabled or not, the create flag will only be used to do a preliminary check of the possibility The ParentQueue isEligibleForAutoQueueCreation method was made to support this mapping rule check ## I will check it out, thank you for noticing! ## I am not sure what you mean by that, but I will check it out. However, I have distinguished the parent hierarchy in its own method to be more flexible (we might need to support deeper hierarchy, more rigorous checks etc.) # We have argued about this problem, and I agree, that the auto queue/dynamic queue terminology is overused, and wanted to make a proposal, to distinguish this new AQC from the ManagedParent old logic. However, I think it is still a good idea to use a property to drive this behaviour due to security reasons: if a customer wants to use weight mode, the AQC will implicitly be turned on, and it might be not desirable (rogue users could create queues anywhere). I prefer explicit allowance in this case. That said, we should come up with a distinguishable terminology for the new auto queue creation (perhaps saying auto-queue-creation-v2 or extended). # I too found this problematic, and will address this issue. # I will add the testcase. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506.001.patch, YARN-10506.002.patch, YARN-10506.003.patch, > YARN-10506.004.patch, YARN-10506.005.patch, YARN-10506.006-combined.patch, > YARN-10506.006.patch, YARN-10506.007.patch, YARN-10506.009.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263961#comment-17263961
]
zhuqi commented on YARN-10506:
--
[~wangda] [~gandras]
Thanks [~wangda] for review.
I update a patch for:
According the comment by [~wangda] above:
1. CapacityScheduler
1.1 Fix the not return lq in Capacity.
1.2 Add 008 patch remaining include to fix testautocreateleaf:
As above in 008 patch, Fix autoCreateLeafQueue to use path , the logic change ,
the 009 not include this:
{code:java}
@VisibleForTesting
protected LeafQueue autoCreateLeafQueue(
ApplicationPlacementContext placementContext)
throws IOException, YarnException {
String leafQueueName = placementContext.getQueue();
String parentQueueName = placementContext.getParentQueue();
if (!StringUtils.isEmpty(parentQueueName)) {
CSQueue parentQueue = getQueue(parentQueueName);
if (parentQueue == null) {
throw new SchedulerDynamicEditException(
"Could not auto-create leaf queue for " + leafQueueName
+ ". Queue mapping specifies an invalid parent queue "
+ "which does not exist " + parentQueueName);
}
if (conf.isAutoCreateChildQueueEnabled(parentQueue.getQueuePath())) {
// Case 1: Handle ManagedParentQueue
AutoCreatedLeafQueue autoCreatedLeafQueue = null;
ManagedParentQueue autoCreateEnabledParentQueue =
(ManagedParentQueue) parentQueue;
autoCreatedLeafQueue = new AutoCreatedLeafQueue(this, leafQueueName,
autoCreateEnabledParentQueue);
addQueue(autoCreatedLeafQueue);
return autoCreatedLeafQueue;
} ...
}
{code}
2. CapacitySchedulerAutoQueueHandler
2.1 Back create flag in ApplicationPlacementContext, and use it in
autoCreateLeafQueue placement check :
{code:java}
private LeafQueue autoCreateLeafQueue(
ApplicationPlacementContext placementContext)
throws IOException, YarnException {
...
if (!StringUtils.isEmpty(parentQueueName)) {
...
if (conf.isAutoCreateChildQueueEnabled(parentQueue.getQueuePath())) {
...
} else {
ApplicationPlacementContext apc = placementContext.clone();
// Set default true now
// TODO get by queue placement policy
apc.setCreateParentQueue(true);
apc.setCreateLeafQueue(true);
if (placementContext.isCreateLeafQueue()
|| placementContext.isCreateParentQueue()) {
} else {
...
}
}
}
{code}
2.2 -2.3
Remove autoCreateParentHierarchy, the logic change to autoCreateQueuePath. And
the autoCreateQueuePath to autoCreateQueue, and make it atomic, and fix related
test.
{code:java}
public LeafQueue autoCreateQueue(ApplicationPlacementContext queue)
throws YarnException {
...
// The existingParentQueue is already ParentQueue
LeafQueue leafQueue = existingParentQueue.addDynamicLeafQueue(
queue.getFullQueuePath());
queueManager.addQueue(leafQueue.getQueuePath(), leafQueue);
return leafQueue;
}
{code}
3. CapacitySchedulerConfiguration change:
Remaining todo.
4. ParentQueue#addDynamicChildQueue, change the logic handle static queue with
empty childs:
{code:java}
private QueueCapacityType getCapacityConfigurationTypeForQueues(
Collection queues) throws IOException {
...
else {
if (isDynamicQueue()) {
// If it is an empty dynamic queue, consider weight mode instead
return QueueCapacityType.WEIGHT;
} else {
// Double check queues should be empty
// When static queue with queues empty, return parent ConfigurationType
if (queues.isEmpty()) {
return getCapacityConfigurationTypeForQueues(ImmutableList.of(this));
} else {
throw new
IOException("Unknown case when child queues not empty! " +
"But no child config type!");
}
}
}
{code}
5. Change the submit test to trigger the placement:
{code:java}
@Test
public void testAutoQueueCreationOnAppSubmission() throws Exception {
startScheduler();
createBasicQueueStructureAndValidate();
submitApp(cs, USER0, USER0, "root.e-auto");
AbstractCSQueue e = (AbstractCSQueue) cs.getQueue("root.e-auto");
Assert.assertNotNull(e);
Assert.assertTrue(e.isDynamicQueue());
AbstractCSQueue user0 = (AbstractCSQueue) cs.getQueue("root.e-auto." + USER0);
Assert.assertNotNull(user0);
Assert.assertTrue(user0.isDynamicQueue());
}
{code}
6. Fix clone finding bugs.
Remaining to do
minor things:
# AutoCreateLeafQueue should be moved to autoQueueHandler.
# CSQueueUtils#extractQueuePath should be moved to CSAutoQueueHandler. (It
won't be used by other classes). And rename extractQueuePath to
extractApplicationPlacementContext (we don't just extract path).
# Add test case that static queues with empty childs.
Major things:
CapacitySchedulerConfiguration change.
T
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263672#comment-17263672
]
Wangda Tan commented on YARN-10506:
---
Thanks for all updates, here's my comments:
1) CapacityScheduler:
Minor:
autoCreateLeafQueue should be moved to autoQueueHandler.
We can do this in a follow up patch, because now we have two places to handle
the auto queue creation, and ideally "autoQueueHandler" should be responsible
for that.
The same follow up patch should also clean up addQueue() method of
ResourceScheduler. It is only used by CapacitySchedulerPlanFollow, we don't
need to add it to the abstract class.
{code}
LeafQueue lq = autoQueueHandler.autoCreateQueuePath(placementContext);
{code}
Return value is not used.
2) CapacitySchedulerAutoQueueHandler:
Major:
2.1) I think the existing logic is to create auto queue based on
ApplicationPlacementContext, but when we do queue mapping, we need to indicate
if auto creation is allowed or not.
In the mapping rule, we have "create" flag, I think CSAutoQueueHandler should
consider the "create" flag.
Previously, in my patch, I added createLeafQueue and createParentQueue flag,
the latest patch removed the logic.
Can you share more thoughts on this? Any please let me know if I missed
anything.
Even though this logic can be done in a separate patch, but I still suggest to
handle it within this one for completeness.
(A side note, I noticed there's a method:
ParentQueue#isEligibleForAutoQueueCreation, it only handles "if a parent queue
can allow auto creation underneath or not", but cannot handle "if a creation
itself is allowed by placement policy or not").
2.2) autoCreateQueuePath doesn't look like atomic, it could create parent
first, but later found LeafQueue cannot be created:
{code}
if (parent instanceof ParentQueue) {
...
} else {
throw new SchedulerDynamicEditException(
"Could not auto-create leaf queue for " + queue.getQueue()
+ ". Queue mapping specifies an invalid parent queue "
+ "which does not exist"
+ queue.getParentQueue());
}
{code}
Can we make it atomic?
2.3) I think autoCreateParentHierarchy itself should be able to handle
LeafQueue creation, because itself can handle multiple leaves, we don't have to
maintain a separate LeafQueue creation logic:
{code}
ParentQueue parentQueue = (ParentQueue) parent;
LeafQueue leafQueue = parentQueue.addDynamicLeafQueue(
queue.getFullQueuePath());
queueManager.addQueue(leafQueue.getQueuePath(), leafQueue);
return leafQueue;
{code}
Minor:
- Rename autoCreateQueuePath to autoCreateQueue (we will never create a "queue
path")
- CSQueueUtils#extractQueuePath should be moved to CSAutoQueueHandler. (It
won't be used by other classes). And rename extractQueuePath to
extractApplicationPlacementContext (we don't just extract path)
3) CapacitySchedulerConfiguration change:
Major:
- Now we added queue-path.auto-queue-creation.enabled flag, which is in
parallel of auto-create-child-queue.enabled flag.
First of all, it is very confusing because there're two parameters looks
similar.
Second, We still need to check the weight mode is configured or not.
I'm actually thinking to get rid of the flag, and completely relying on
ParentQueue#addDynamicChildQueue to do the weights check: If a parent queue has
children use weight, we can proceed with queue creation.
We can improve the flag later, thoughts?
4) ParentQueue#addDynamicChildQueue:
Major:
- The logic ParentQueue#getCapacityConfigurationTypeForQueues returns PERCENT
when there's no children under the parent. For that case, I think we should
specially handle it inside getCapacityConfigurationTypeForQueues:
{code}
Which should return WEIGHT when Collection queues isEmpty
{code}
And we should add an unit test to add queue under a static parent queue which
doesn't have children, because it will be a common case when user use the
feature.
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10506-006-10504-010.patch,
> YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506.001.patch,
> YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch,
> YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch,
> YARN-10506.007.patch, YARN-10506.009.patch
>
>
> The queue creation logic should be updated to use weight mode and support the
> flexible creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263550#comment-17263550
]
Hadoop QA commented on YARN-10506:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
51s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
1s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 2
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
22s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 15s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
48s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 48s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/471/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
| {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 34 new + 737 unchanged - 1 fixed = 771 total (was 738)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 39s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {c
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263457#comment-17263457 ] Wangda Tan commented on YARN-10506: --- [~zhuqi], [~gandras] can you check the findbugs warning? > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506.001.patch, > YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch, > YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch, > YARN-10506.007.patch, YARN-10506.009.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263399#comment-17263399 ] zhuqi commented on YARN-10506: -- [~gandras] Thanks for including the fix. LGTM +1 Waiting for the CI. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506.001.patch, > YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch, > YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch, > YARN-10506.007.patch, YARN-10506.009.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263393#comment-17263393 ] Andras Gyori commented on YARN-10506: - I have uploaded a new patch, which is based on trunk with the freshly committed YARN-10504. Added an additional test case for queue creation with application submission. Thank you [~zhuqi] for the fixes, I have included them in the patch. At this point, nothing comes to my mind as to what else is needed, therefore this could be reviewed now. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506.001.patch, > YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch, > YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch, > YARN-10506.007.patch, YARN-10506.009.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263211#comment-17263211
]
Hadoop QA commented on YARN-10506:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
12s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 2
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
54s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 58s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
53s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 42s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/467/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
| {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 34 new + 738 unchanged - 1 fixed = 772 total (was 739)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 45s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {c
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263118#comment-17263118
]
zhuqi commented on YARN-10506:
--
[~wangda] [~gandras]
Update a patch to fix two problem:
# Fix autoCreateLeafQueue to use path , the logic change to
{code:java}
@VisibleForTesting
protected LeafQueue autoCreateLeafQueue(
ApplicationPlacementContext placementContext)
throws IOException, YarnException {
String leafQueueName = placementContext.getQueue();
String parentQueueName = placementContext.getParentQueue();
if (!StringUtils.isEmpty(parentQueueName)) {
CSQueue parentQueue = getQueue(parentQueueName);
if (parentQueue == null) {
throw new SchedulerDynamicEditException(
"Could not auto-create leaf queue for " + leafQueueName
+ ". Queue mapping specifies an invalid parent queue "
+ "which does not exist " + parentQueueName);
}
if (conf.isAutoCreateChildQueueEnabled(parentQueue.getQueuePath())) {
// Case 1: Handle ManagedParentQueue
AutoCreatedLeafQueue autoCreatedLeafQueue = null;
ManagedParentQueue autoCreateEnabledParentQueue =
(ManagedParentQueue) parentQueue;
autoCreatedLeafQueue = new AutoCreatedLeafQueue(this, leafQueueName,
autoCreateEnabledParentQueue);
addQueue(autoCreatedLeafQueue);
return autoCreatedLeafQueue;
} else {
try {
writeLock.lock();
LeafQueue lq = autoQueueHandler.autoCreateQueuePath(placementContext);
} finally {
writeLock.unlock();
}
}
}
throw new SchedulerDynamicEditException(
"Could not auto-create leaf queue for " + leafQueueName
+ ". Queue mapping does not specify"
+ " which parent queue it needs to be created under.");
}
{code}
# Fix testActivateApplicationAfterQueueRefresh in testLeafQueue to update
active apps use update:
{code:java}
// This will not update active apps
root.reinitialize(newRoot, csContext.getClusterResource());
// Cause this to update active apps
root.updateClusterResource(csContext.getClusterResource(),
new ResourceLimits(csContext.getClusterResource()));
{code}
Thanks.
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10506-006-10504-010.patch,
> YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506.001.patch,
> YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch,
> YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch,
> YARN-10506.007.patch
>
>
> The queue creation logic should be updated to use weight mode and support the
> flexible creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17262789#comment-17262789
]
Hadoop QA commented on YARN-10506:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
14s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
1s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include
13 new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
56s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
20m 27s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m
5s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config;
considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 53s{color}
|
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/461/artifact/out/diff-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt{color}
| {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 generated 2 new + 40
unchanged - 2 fixed = 42 total (was 42) {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 47s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/461/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
| {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 104 new + 936 unchanged - 14 fixed = 1040 total (was 950)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {co
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17262685#comment-17262685 ] Andras Gyori commented on YARN-10506: - Thank you [~zhuqi] it was indeed the issue. I have fixed it in an other method, where we assume, that if the queue is a dynamic queue and has no child queues, it is in weight mode (it should be the case). Also, after a discussion, we agreed on that the property must be always set in order to be eligible for auto queue creation, therefore I removed the corresponding code from ApplicationPlacementContext. In addition to this, some part of the validation logic is moved to ParentQueue because the mapping rules will need it. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506.001.patch, YARN-10506.002.patch, > YARN-10506.003.patch, YARN-10506.004.patch, YARN-10506.005.patch, > YARN-10506.006-combined.patch, YARN-10506.006.patch, YARN-10506.007.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17262673#comment-17262673
]
Hadoop QA commented on YARN-10506:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 12s{color}
| {color:red}{color} | {color:red} YARN-10506 does not apply to trunk. Rebase
required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for
help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-10506 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13018528/YARN-10506.007.patch |
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/462/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10506-006-10504-010.patch,
> YARN-10506-007-10504-010.patch, YARN-10506.001.patch, YARN-10506.002.patch,
> YARN-10506.003.patch, YARN-10506.004.patch, YARN-10506.005.patch,
> YARN-10506.006-combined.patch, YARN-10506.006.patch, YARN-10506.007.patch
>
>
> The queue creation logic should be updated to use weight mode and support the
> flexible creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17262660#comment-17262660
]
zhuqi commented on YARN-10506:
--
[~wangda] [~gandras]
Add an updated patch for fix all test in
TestCapacitySchedulerNewQueueAutoCreation.
In ParentQueue's addDynamicChildQueue
{code:java}
// New method to add child queue
public CSQueue addDynamicChildQueue(String childQueuePath, boolean isLeaf)
throws SchedulerDynamicEditException {
writeLock.lock();
try {
// Changed here
boolean weightsAreUsed = false;
try {
if (isLeaf && childQueues.isEmpty()) {
weightsAreUsed = true;
} else {
weightsAreUsed = getCapacityConfigurationTypeForQueues(childQueues)
== QueueCapacityType.WEIGHT;
}
} catch (IOException e) {
LOG.warn("Caught Exception during auto queue creation", e);
}
...
}{code}
Reason:
When add a dynamic leaf queue, but we still not update the weight it, also the
childQueues is empty, when original getCapacityConfigurationTypeForQueues will
return percentage, we should handle this case to weightsAreUsed = true, avoid
the error.
Any other test fail, may be related queueplacement.
Thanks.
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10506-006-10504-010.patch,
> YARN-10506-007-10504-010.patch, YARN-10506.001.patch, YARN-10506.002.patch,
> YARN-10506.003.patch, YARN-10506.004.patch, YARN-10506.005.patch,
> YARN-10506.006-combined.patch, YARN-10506.006.patch
>
>
> The queue creation logic should be updated to use weight mode and support the
> flexible creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17262619#comment-17262619
]
Hadoop QA commented on YARN-10506:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
17s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include
13 new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
28s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 36s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m
7s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config;
considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
5s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 5s{color}
|
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/458/artifact/out/diff-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt{color}
| {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 generated 2 new + 40
unchanged - 2 fixed = 42 total (was 42) {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 50s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/458/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
| {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 104 new + 936 unchanged - 14 fixed = 1040 total (was 950)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {co
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17262528#comment-17262528 ] Andras Gyori commented on YARN-10506: - Combined YARN-10504-010 and YARN-10506-006 to see test failures. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, YARN-10506.001.patch, > YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch, > YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17262364#comment-17262364
]
Hadoop QA commented on YARN-10506:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
43s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include
13 new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
0s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 0s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
45s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
43s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 56s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/455/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
| {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 104 new + 935 unchanged - 14 fixed = 1039 total (was 949)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 51s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17262323#comment-17262323 ] Wangda Tan commented on YARN-10506: --- Attached ver.6 combined patch (which includes YARN-10506.005 patch, plus YARN-10504.009 patch), which will trigger Jenkins and see how the run looks like. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506.001.patch, YARN-10506.002.patch, > YARN-10506.003.patch, YARN-10506.004.patch, YARN-10506.005.patch, > YARN-10506.006-combined.patch, YARN-10506.006.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17262305#comment-17262305
]
Hadoop QA commented on YARN-10506:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 10s{color}
| {color:red}{color} | {color:red} YARN-10506 does not apply to trunk. Rebase
required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for
help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-10506 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13018452/YARN-10506.006.patch |
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/453/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10506.001.patch, YARN-10506.002.patch,
> YARN-10506.003.patch, YARN-10506.004.patch, YARN-10506.005.patch,
> YARN-10506.006.patch
>
>
> The queue creation logic should be updated to use weight mode and support the
> flexible creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17262302#comment-17262302 ] Wangda Tan commented on YARN-10506: --- Manually rebased patch on top of the latest YARN-10504 (008). Haven't run any tests, just wanted to see if there are any significant conflicts (so far no). > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506.001.patch, YARN-10506.002.patch, > YARN-10506.003.patch, YARN-10506.004.patch, YARN-10506.005.patch, > YARN-10506.006.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17261259#comment-17261259
]
Andras Gyori commented on YARN-10506:
-
Added support for placementContexts. As of now, there are two ways to enable
auto queue creation under a specific ParentQueue:
1. Set the isCreateParentQueue and isCreateLeafQueue of
ApplicationPlacementContext (which will be done by the mapping rules)
2. If they are not set, the logic will fall back to check if the option enabled
for the ParentQueue:
{noformat}
yarn.scheduler.capacity.root..auto-queue-creation.enabled{noformat}
Also, by its nature, dynamically created ParentQueues allow auto queue creation
by default. Therefore it is possible to create a new queue under a previously
auto created ParentQueue without setting anything.
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10506.001.patch, YARN-10506.002.patch,
> YARN-10506.003.patch, YARN-10506.004.patch, YARN-10506.005.patch
>
>
> The queue creation logic should be updated to use weight mode and support the
> flexible creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17261258#comment-17261258
]
Hadoop QA commented on YARN-10506:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 11s{color}
| {color:red}{color} | {color:red} YARN-10506 does not apply to trunk. Rebase
required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for
help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-10506 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13018367/YARN-10506.005.patch |
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/443/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10506.001.patch, YARN-10506.002.patch,
> YARN-10506.003.patch, YARN-10506.004.patch, YARN-10506.005.patch
>
>
> The queue creation logic should be updated to use weight mode and support the
> flexible creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259764#comment-17259764 ] Andras Gyori commented on YARN-10506: - Thank you [~zhuqi] for your feedback! I understand your point, and you are right, we need to extend the template policy logic to support multiple auto created levels, and will update the patch later. Regarding your last point, the queue auto deletion should be handled by https://issues.apache.org/jira/browse/YARN-10532. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506.001.patch, YARN-10506.002.patch, > YARN-10506.003.patch, YARN-10506.004.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259445#comment-17259445 ] zhuqi commented on YARN-10506: -- [~gandras] The new patch, LGTM. And we also should support template settings for dynamic created queue, such as: The default weight setting, now the default weight is always 1. The max dynamic queues for creation for some parent etc. Also since we don't use ManagedParent/AutoCreatedLeafQueue , but some policy we may should fill in parent/leaf queue, such as expired inactive queues to 0 capacity / or deletion (two option) etc. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506.001.patch, YARN-10506.002.patch, > YARN-10506.003.patch, YARN-10506.004.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17258220#comment-17258220
]
Hadoop QA commented on YARN-10506:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 13s{color}
| {color:red}{color} | {color:red} YARN-10506 does not apply to trunk. Rebase
required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for
help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-10506 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13018122/YARN-10506.004.patch |
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/424/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10506.001.patch, YARN-10506.002.patch,
> YARN-10506.003.patch, YARN-10506.004.patch
>
>
> The queue creation logic should be updated to use weight mode and support the
> flexible creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17258209#comment-17258209
]
Hadoop QA commented on YARN-10506:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 9s{color}
| {color:red}{color} | {color:red} YARN-10506 does not apply to trunk. Rebase
required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for
help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-10506 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13018120/YARN-10506.004.patch |
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/422/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10506.001.patch, YARN-10506.002.patch,
> YARN-10506.003.patch, YARN-10506.004.patch
>
>
> The queue creation logic should be updated to use weight mode and support the
> flexible creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17258204#comment-17258204 ] Andras Gyori commented on YARN-10506: - Fixed the test regarding mixed weight mode and capacity mode. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506.001.patch, YARN-10506.002.patch, > YARN-10506.003.patch, YARN-10506.004.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17257451#comment-17257451 ] Andras Gyori commented on YARN-10506: - Uploaded a new revision. Since we already have a test of converting dynamic parent to static parent, I have kept this logic for now and extended the related code part to handle this corner case. However, we might need to change this to be an illegal queue declaration, as [~zhuqi] suggested, because it could impose difficulties we are not aware now. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506.001.patch, YARN-10506.002.patch, > YARN-10506.003.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17257449#comment-17257449
]
Hadoop QA commented on YARN-10506:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 10s{color}
| {color:red}{color} | {color:red} YARN-10506 does not apply to trunk. Rebase
required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for
help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-10506 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13018037/YARN-10506.003.patch |
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/421/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10506.001.patch, YARN-10506.002.patch,
> YARN-10506.003.patch
>
>
> The queue creation logic should be updated to use weight mode and support the
> flexible creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17253943#comment-17253943 ] zhuqi commented on YARN-10506: -- [~gandras] Thanks for your reply. 1. What i said is related policy management, we should fixed it later, and may be we need a new sub-task. 2. I think we should disallow it by throwing an exception, but we can add a admin api to realize destroying all its underlying children for optional, it will be more reasonable. Thanks. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506.001.patch, YARN-10506.002.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17253687#comment-17253687
]
Hadoop QA commented on YARN-10506:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 9s{color}
| {color:red}{color} | {color:red} YARN-10506 does not apply to trunk. Rebase
required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for
help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-10506 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13017530/YARN-10506.002.patch |
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/412/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10506.001.patch, YARN-10506.002.patch
>
>
> The queue creation logic should be updated to use weight mode and support the
> flexible creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17253682#comment-17253682 ] Andras Gyori commented on YARN-10506: - I have taken your advise into account, however, I would need more details about from you about the problem at hand. You mentioned ManagedParent, but I need to emphasise, that we have taken an approach that does not involve ManagedParentQueue or AutoCreatedLeafQueue at all. We are reusing the original ParentQueue and LeafQueue classes. I have fixed the failing tests, but I have also identified a possible issue, which needs to be addressed. If a dynamic queue is a ParentQueue, and it has child queues as well, and we make it a static queue, it will be converted to a LeafQueue, basically destroying all its underlying children. Either we disallow it by throwing an exception, or we convert it successfully. What is your opinion about it? > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506.001.patch, YARN-10506.002.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17252934#comment-17252934 ] Andras Gyori commented on YARN-10506: - Thank you [~zhuqi] for the feedback! I intend to upload a new revision tomorrow and will keep in mind your suggestions. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506.001.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17252919#comment-17252919
]
zhuqi commented on YARN-10506:
--
[~wangda] [~gandras]
When i deep into some test in TestCapacitySchedulerAutoQueueCreation, because
of the update resources now independent of queue init according to
[YARN-10504|https://issues.apache.org/jira/browse/YARN-10504], we should finish
the policy related absolute update in ManagedParentQueue's addChildQueue such
as :
{code:java}
// In ManagedParentQueue's addChildQueue
for (String label : policy.leafQueueTemplateNodeLabels) {
// Weight will be normalized to queue.weight =
// queue.weight(sum({sibling-queues.weight}))
// When weight is set, capacity will be set to 0;
// When capacity is set, weight will be normalized to 0,
// So get larger from normalized_weight and capacity will make sure we do
// calculation correct
float capacity =
Math.max(policy.leafQueueTemplate.getQueueCapacities().getCapacity(label),
policy.leafQueueTemplate.getQueueCapacities().getNormalizedWeight(label));
if (capacity > 0f) {
policy.leafQueueTemplate.getQueueCapacities()
.setAbsoluteCapacity(label, capacity * (
parentQueueCapacities == null ?
1 :
parentQueueCapacities.getAbsoluteCapacity(label)));
policy.leafQueueTemplateCapacities =
policy.leafQueueTemplate.getQueueCapacities();
}
float maxCapacity =
policy.leafQueueTemplate.getQueueCapacities().getMaximumCapacity(label);
if (maxCapacity > 0f) {
policy.leafQueueTemplate.getQueueCapacities()
.setAbsoluteMaximumCapacity(label, maxCapacity * (
parentQueueCapacities == null ?
1 :
parentQueueCapacities.getAbsoluteMaximumCapacity(label)));
policy.leafQueueTemplateCapacities =
policy.leafQueueTemplate.getQueueCapacities();
}
}
{code}
and policy should update template :
{code:java}
private void updateCapacityFromTemplate(QueueCapacities capacities,
String nodeLabel) {
capacities.setCapacity(nodeLabel,
leafQueueTemplateCapacities.getCapacity(nodeLabel));
capacities.setMaximumCapacity(nodeLabel,
leafQueueTemplateCapacities.getMaximumCapacity(nodeLabel));
capacities.setAbsoluteCapacity(nodeLabel,
leafQueueTemplateCapacities.getAbsoluteCapacity(nodeLabel));
}
{code}
Then the test will pass, but the reinitialize related will block.
Also when we reinitialize the queue, we should think how to bring up this to
reinitialize and update resource progress.
So i am thinking , if it is worth to make update resources independent of
queue init.
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10506.001.patch
>
>
> The queue creation logic should be updated to use weight mode and support the
> flexible creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17248989#comment-17248989
]
zhuqi commented on YARN-10506:
--
[~wangda]
{quote}Handle convert of dynamic queue to static queue (still see some test
failures).
{quote}
The test failed because some mistakes queue name , should change to this:
{code:java}
// Do change 2nd level queue from dynamic to static
csConf.setQueues("root", new String[] { "a", "b", "c-auto", "e" });
csConf.setNonLabeledQueueWeight("root.e", 6f);
csConf.setQueues("root.e", new String[] { "e1-auto" });
csConf.setNonLabeledQueueWeight("root.e.e1-auto", 6f);
cs.reinitialize(csConf, mockRM.getRMContext());
// Get queue c
CSQueue e1 = cs.getQueue("root.e.e1-auto");
// e's abs resource should be 6/20 * (6/7), (since a/c/e.weight=6, all other 2
peers
// have weight=1, and e1's weight is 6, e2's weight is 1).
Assert.assertEquals((6 / 20f) * (6 / 7f), e1.getAbsoluteCapacity(), 1e-6);
Assert.assertEquals(360 * GB,
c.getQueueResourceQuotas().getEffectiveMinResource().getMemorySize());
Assert.assertEquals(6f, c.getQueueCapacities().getWeight(), 1e-6);
{code}
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Priority: Major
> Attachments: YARN-10506.001.patch
>
>
> The queue creation logic should be updated to use weight mode and support the
> flexible creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17248263#comment-17248263
]
Hadoop QA commented on YARN-10506:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 10s{color}
| {color:red}{color} | {color:red} YARN-10506 does not apply to trunk. Rebase
required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for
help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-10506 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13016968/YARN-10506.001.patch |
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/382/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Priority: Major
> Attachments: YARN-10506.001.patch
>
>
> The queue creation logic should be updated to use weight mode and support the
> flexible creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17248261#comment-17248261 ] Wangda Tan commented on YARN-10506: --- Uploaded a patch, which is based on YARN-10504. It has done the following: - Handle create of leaf and parent. - Added control parameter to ApplicationPlacementContext. - Handle dynamic adjust weights for queues. Partically complete: - Handle convert of dynamic queue to static queue (still see some test failures). Not started: - Integrate with Queue Placement Policy and related tests. Have done unit tests for some part of logics, details see TestCapacitySchedulerNewQueueAutoCreation > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Priority: Major > Attachments: YARN-10506.001.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17248048#comment-17248048 ] Wangda Tan commented on YARN-10506: --- I'm looking at PoC of the patch now, and will keep the JIRA updated in a day or two. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Priority: Major > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
