[jira] [Commented] (YARN-10124) Remove restriction of ParentQueue capacity zero when childCapacities > 0
[
https://issues.apache.org/jira/browse/YARN-10124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17354393#comment-17354393
]
Hadoop QA commented on YARN-10124:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 13s{color}
| {color:red}{color} | {color:red} YARN-10124 does not apply to trunk. Rebase
required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for
help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-10124 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12994612/YARN-10124-002.patch |
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1029/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
> Remove restriction of ParentQueue capacity zero when childCapacities > 0
>
>
> Key: YARN-10124
> URL: https://issues.apache.org/jira/browse/YARN-10124
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-10124-001.patch, YARN-10124-002.patch
>
>
> ParentQueue capacity cannot be set to 0 when child capacities > 0. To disable
> a parent queue temporarily, user can only STOP the queue but the capacity of
> the queue cannot be used for other queues. Allowing 0 capacity for parent
> queue will allow user to use the capacity for other queues and also to retain
> the child queue capacity values. (else user has to set all child queue
> capacities to 0)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10124) Remove restriction of ParentQueue capacity zero when childCapacities > 0
[
https://issues.apache.org/jira/browse/YARN-10124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17050927#comment-17050927
]
Hadoop QA commented on YARN-10124:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
5s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 16s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 3s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 23m 0s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch passed. {color} |
| {color:blue}0{color} | {color:blue} asflicense {color} | {color:blue} 0m
44s{color} | {color:blue} ASF License check generated no output? {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 84m 9s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.6 Server=19.03.6 Image:yetus/hadoop:c44943d1fc3 |
| JIRA Issue | YARN-10124 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12994612/YARN-10124-002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 81e622dc1776 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / bbd704b |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_242 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/25622/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/25622/testReport/ |
| Max. process+thread count | 803 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/h
[jira] [Commented] (YARN-10124) Remove restriction of ParentQueue capacity zero when childCapacities > 0
[
https://issues.apache.org/jira/browse/YARN-10124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17045262#comment-17045262
]
Hadoop QA commented on YARN-10124:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 6s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 19s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 89m 14s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}154m 10s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.6 Server=19.03.6 Image:yetus/hadoop:c44943d1fc3 |
| JIRA Issue | YARN-10124 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12994612/YARN-10124-002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux b1394a15799e 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 900430b |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_242 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/25583/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/25583/testReport/ |
| Max. process+thread count | 822 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-projec
[jira] [Commented] (YARN-10124) Remove restriction of ParentQueue capacity zero when childCapacities > 0
[ https://issues.apache.org/jira/browse/YARN-10124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17045183#comment-17045183 ] Prabhu Joseph commented on YARN-10124: -- Thanks [~wilfreds] for reviewing. Have addressed above comments in [^YARN-10124-002.patch] . > Remove restriction of ParentQueue capacity zero when childCapacities > 0 > > > Key: YARN-10124 > URL: https://issues.apache.org/jira/browse/YARN-10124 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Attachments: YARN-10124-001.patch, YARN-10124-002.patch > > > ParentQueue capacity cannot be set to 0 when child capacities > 0. To disable > a parent queue temporarily, user can only STOP the queue but the capacity of > the queue cannot be used for other queues. Allowing 0 capacity for parent > queue will allow user to use the capacity for other queues and also to retain > the child queue capacity values. (else user has to set all child queue > capacities to 0) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10124) Remove restriction of ParentQueue capacity zero when childCapacities > 0
[
https://issues.apache.org/jira/browse/YARN-10124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17044996#comment-17044996
]
Wilfred Spiegelenburg commented on YARN-10124:
--
Ah ok, that makes sense now.
Based on the testing you have done it looks like we're all set and do not have
any side effects of the change.
Two little nits that need to be fixed before we can commit this: can we clean
up the message generation for the strings and remove the surplus addition in
the text for the string in these two spots:
{code}
196 throw new IllegalArgumentException("Illegal" + " capacity of "
{code}
and
{code}
229 "Illegal" + " capacity of " + sum + " for children of queue "
{code}
Beside that: +1
> Remove restriction of ParentQueue capacity zero when childCapacities > 0
>
>
> Key: YARN-10124
> URL: https://issues.apache.org/jira/browse/YARN-10124
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-10124-001.patch
>
>
> ParentQueue capacity cannot be set to 0 when child capacities > 0. To disable
> a parent queue temporarily, user can only STOP the queue but the capacity of
> the queue cannot be used for other queues. Allowing 0 capacity for parent
> queue will allow user to use the capacity for other queues and also to retain
> the child queue capacity values. (else user has to set all child queue
> capacities to 0)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10124) Remove restriction of ParentQueue capacity zero when childCapacities > 0
[ https://issues.apache.org/jira/browse/YARN-10124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17044366#comment-17044366 ] Prabhu Joseph commented on YARN-10124: -- [~wilfreds] Sorry for confusion between soft (capacity) and hard limit (maxCapacity) of a queue. Below is the answer to your previous question bq. How does it affect the capacity calculation, i.e. is the queue now always below or above its (hard)limits? 2. Any queue irrespective of capacity 0 or any other value will be always below or equal to its maxCapacity (hard limit). i.e, it can expand up to maxCapacity when the cluster is idle. If the queue capacity (soft limit) is 0, then any used resources by the queue are preemptible at any time. > Remove restriction of ParentQueue capacity zero when childCapacities > 0 > > > Key: YARN-10124 > URL: https://issues.apache.org/jira/browse/YARN-10124 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Attachments: YARN-10124-001.patch > > > ParentQueue capacity cannot be set to 0 when child capacities > 0. To disable > a parent queue temporarily, user can only STOP the queue but the capacity of > the queue cannot be used for other queues. Allowing 0 capacity for parent > queue will allow user to use the capacity for other queues and also to retain > the child queue capacity values. (else user has to set all child queue > capacities to 0) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10124) Remove restriction of ParentQueue capacity zero when childCapacities > 0
[ https://issues.apache.org/jira/browse/YARN-10124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17044286#comment-17044286 ] Wilfred Spiegelenburg commented on YARN-10124: -- 1: this point by itself sounds good, we want the child queues to get resources and run the apps until they finish 2: if when 0 is set for the parent it is always above the hard limit then how does 1 work? I would have expected that the limit is enforced in the hierarchy. So can you please explain a bit more on how this works as this is not what I would expect. 2: (second time) any resource above the limit should be preemptable so it lines up with the first point 2. 3: is expected I would consider setting any on the root (min max etc) as a bug: the root is the whole cluster and shoulkd thus mirror what is available in the cluster without limits. Otherwise a cluster could never grow or shrink. > Remove restriction of ParentQueue capacity zero when childCapacities > 0 > > > Key: YARN-10124 > URL: https://issues.apache.org/jira/browse/YARN-10124 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Attachments: YARN-10124-001.patch > > > ParentQueue capacity cannot be set to 0 when child capacities > 0. To disable > a parent queue temporarily, user can only STOP the queue but the capacity of > the queue cannot be used for other queues. Allowing 0 capacity for parent > queue will allow user to use the capacity for other queues and also to retain > the child queue capacity values. (else user has to set all child queue > capacities to 0) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10124) Remove restriction of ParentQueue capacity zero when childCapacities > 0
[ https://issues.apache.org/jira/browse/YARN-10124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17040271#comment-17040271 ] Prabhu Joseph commented on YARN-10124: -- Below are the observations after testing the patch with setting a parent queue to 0% capacity. 1. Leaf queues under the parent queue which is set with 0% capacity won't accept any new apps. The already running apps will get resources based on the queue max capacity. 2. When there is a used resource in the queue, then it is above its hard limit 0%. 2. The used resources is preemptible when the other queue has demand and under the configured capacity, cluster is full. 3. Root queue cannot be set to 0%, it will be always 100%. > Remove restriction of ParentQueue capacity zero when childCapacities > 0 > > > Key: YARN-10124 > URL: https://issues.apache.org/jira/browse/YARN-10124 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Attachments: YARN-10124-001.patch > > > ParentQueue capacity cannot be set to 0 when child capacities > 0. To disable > a parent queue temporarily, user can only STOP the queue but the capacity of > the queue cannot be used for other queues. Allowing 0 capacity for parent > queue will allow user to use the capacity for other queues and also to retain > the child queue capacity values. (else user has to set all child queue > capacities to 0) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10124) Remove restriction of ParentQueue capacity zero when childCapacities > 0
[ https://issues.apache.org/jira/browse/YARN-10124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17039817#comment-17039817 ] Wilfred Spiegelenburg commented on YARN-10124: -- When you set a parent queue to a 0 capacity could that not leave the apps in the leaf queues below it in a state that they are not scheduled? How does it affect the capacity calculation, i.e. is the queue now always below or above its (hard)limits? > Remove restriction of ParentQueue capacity zero when childCapacities > 0 > > > Key: YARN-10124 > URL: https://issues.apache.org/jira/browse/YARN-10124 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Attachments: YARN-10124-001.patch > > > ParentQueue capacity cannot be set to 0 when child capacities > 0. To disable > a parent queue temporarily, user can only STOP the queue but the capacity of > the queue cannot be used for other queues. Allowing 0 capacity for parent > queue will allow user to use the capacity for other queues and also to retain > the child queue capacity values. (else user has to set all child queue > capacities to 0) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10124) Remove restriction of ParentQueue capacity zero when childCapacities > 0
[ https://issues.apache.org/jira/browse/YARN-10124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17034337#comment-17034337 ] Prabhu Joseph commented on YARN-10124: -- Interesting, this will be helpful. There is a difference in usability between both cases A) STOP a queue and then utilize the capacity of STOPPED queue for sibling queues and B) set parent queue capacity to 0 and utilize the capacity for sibling queues. In case A, user has to wait for queue to be STOPPED completely from DRAINING state and then increase sibling queue capacities. Setting capacity to 0 when queue is in DRAINING state can cause preemption of containers of running apps. In case B, user need not wait, both set parent queue capacity to 0 and update sibling queue capacities can be done at a time. Would like to have both cases. If you are fine with above, will handle case B as part of this Jira and create another Jira for case A. > Remove restriction of ParentQueue capacity zero when childCapacities > 0 > > > Key: YARN-10124 > URL: https://issues.apache.org/jira/browse/YARN-10124 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Attachments: YARN-10124-001.patch > > > ParentQueue capacity cannot be set to 0 when child capacities > 0. To disable > a parent queue temporarily, user can only STOP the queue but the capacity of > the queue cannot be used for other queues. Allowing 0 capacity for parent > queue will allow user to use the capacity for other queues and also to retain > the child queue capacity values. (else user has to set all child queue > capacities to 0) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10124) Remove restriction of ParentQueue capacity zero when childCapacities > 0
[ https://issues.apache.org/jira/browse/YARN-10124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17034266#comment-17034266 ] Wilfred Spiegelenburg commented on YARN-10124: -- Based on what I read here is that you expect a stopped queue to not have a capacity and thus return 0 when calculating the correct distributing. If that is the use case then why can't we implement that. In other words: why not turn this around, only return or take into account the capacity of a queue when it is not in a stopped state? So you return 0 for all stopped queues. You do not have to go further than that. No need to (re)calculate below the parent that is stopped 0 (as that is all ignored) and turning the queue back on will trigger the existing settings to be applied again without further changes. > Remove restriction of ParentQueue capacity zero when childCapacities > 0 > > > Key: YARN-10124 > URL: https://issues.apache.org/jira/browse/YARN-10124 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Attachments: YARN-10124-001.patch > > > ParentQueue capacity cannot be set to 0 when child capacities > 0. To disable > a parent queue temporarily, user can only STOP the queue but the capacity of > the queue cannot be used for other queues. Allowing 0 capacity for parent > queue will allow user to use the capacity for other queues and also to retain > the child queue capacity values. (else user has to set all child queue > capacities to 0) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10124) Remove restriction of ParentQueue capacity zero when childCapacities > 0
[
https://issues.apache.org/jira/browse/YARN-10124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17033559#comment-17033559
]
Hadoop QA commented on YARN-10124:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
53s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 30s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 17s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 88m
14s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}148m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 |
| JIRA Issue | YARN-10124 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12993018/YARN-10124-001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 11730e960044 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d5467d2 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_242 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/25510/testReport/ |
| Max. process+thread count | 829 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/25510/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Remove restriction of ParentQueu
[jira] [Commented] (YARN-10124) Remove restriction of ParentQueue capacity zero when childCapacities > 0
[ https://issues.apache.org/jira/browse/YARN-10124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17033478#comment-17033478 ] Prabhu Joseph commented on YARN-10124: -- Thanks [~sunilg], submitting a patch to check the failing testcases to know if any impact. > Remove restriction of ParentQueue capacity zero when childCapacities > 0 > > > Key: YARN-10124 > URL: https://issues.apache.org/jira/browse/YARN-10124 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Attachments: YARN-10124-001.patch > > > ParentQueue capacity cannot be set to 0 when child capacities > 0. To disable > a parent queue temporarily, user can only STOP the queue but the capacity of > the queue cannot be used for other queues. Allowing 0 capacity for parent > queue will allow user to use the capacity for other queues and also to retain > the child queue capacity values. (else user has to set all child queue > capacities to 0) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10124) Remove restriction of ParentQueue capacity zero when childCapacities > 0
[ https://issues.apache.org/jira/browse/YARN-10124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17033472#comment-17033472 ] Sunil G commented on YARN-10124: Ideally parent queue can be of "Zero" capacity, while its children may still have some +ve value for capacities. This gives a good flexibility in terms of queue management. I am +ve about relaxing current constraint and make parent queue capacity to 0. [~leftnoteasy] [~cheersyang] [~wilfreds] [~jhung] thoughts? > Remove restriction of ParentQueue capacity zero when childCapacities > 0 > > > Key: YARN-10124 > URL: https://issues.apache.org/jira/browse/YARN-10124 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > > ParentQueue capacity cannot be set to 0 when child capacities > 0. To disable > a parent queue temporarily, user can only STOP the queue but the capacity of > the queue cannot be used for other queues. Allowing 0 capacity for parent > queue will allow user to use the capacity for other queues and also to retain > the child queue capacity values. (else user has to set all child queue > capacities to 0) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
