[jira] [Updated] (YARN-10628) Add node usage metrics in SLS
[ https://issues.apache.org/jira/browse/YARN-10628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] VADAGA ANANYO RAO updated YARN-10628: - Attachment: YARN-10628.0001.patch > Add node usage metrics in SLS > - > > Key: YARN-10628 > URL: https://issues.apache.org/jira/browse/YARN-10628 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler-load-simulator >Affects Versions: 3.3.1 >Reporter: VADAGA ANANYO RAO >Assignee: VADAGA ANANYO RAO >Priority: Major > Attachments: Nodes_memory_usage.png, Nodes_vcores_usage.png, > YARN-10628.0001.patch > > Original Estimate: 336h > Remaining Estimate: 336h > > Given the work around container packing going on in YARN schedulers, it would > be beneficial to have charts showing the usage per node in SLS. This will > help to improve container packing algorithms for more efficient packings. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10628) Add node usage metrics in SLS
VADAGA ANANYO RAO created YARN-10628: Summary: Add node usage metrics in SLS Key: YARN-10628 URL: https://issues.apache.org/jira/browse/YARN-10628 Project: Hadoop YARN Issue Type: Improvement Components: scheduler-load-simulator Affects Versions: 3.3.1 Reporter: VADAGA ANANYO RAO Assignee: VADAGA ANANYO RAO Attachments: Nodes_memory_usage.png, Nodes_vcores_usage.png Given the work around container packing going on in YARN schedulers, it would be beneficial to have charts showing the usage per node in SLS. This will help to improve container packing algorithms for more efficient packings. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10626) Log resource allocation in NM log at container start time
[ https://issues.apache.org/jira/browse/YARN-10626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284957#comment-17284957 ] Eric Badger commented on YARN-10626: [~Jim_Brennan], could you review this when you get a chance? Due to the nature of a log change, I don't believe any unit tests are necessary for this patch > Log resource allocation in NM log at container start time > - > > Key: YARN-10626 > URL: https://issues.apache.org/jira/browse/YARN-10626 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-10626.001.patch, YARN-10626.002.patch > > > As far as I can tell, there are no resource allocation logs in the NM log for > the various containers that are scheduled. These can be useful when trying to > debug what resources were requested vs what resources were actually > allocated. This is especially useful when debugging upstream technology > changes to make sure that they are correctly interpreting and passing down > resource parameters -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10626) Log resource allocation in NM log at container start time
[
https://issues.apache.org/jira/browse/YARN-10626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284948#comment-17284948
]
Hadoop QA commented on YARN-10626:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
7s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red}{color} | {color:red} The patch doesn't appear to
include any new or modified tests. Please justify why no new tests are needed
for this patch. Also please list what manual steps were performed to verify
this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
20s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
21s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
40s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 18s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
20s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
21s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
21s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
15s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 1s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {col
[jira] [Created] (YARN-10627) Extend logging to give more information about weight mode
Benjamin Teke created YARN-10627:
Summary: Extend logging to give more information about weight mode
Key: YARN-10627
URL: https://issues.apache.org/jira/browse/YARN-10627
Project: Hadoop YARN
Issue Type: Sub-task
Components: yarn
Reporter: Benjamin Teke
Assignee: Benjamin Teke
In YARN-10504 weight mode was added, however the logged information about the
created queues or the toString methods weren't updated accordingly. Some
examples:
ParentQueue#setupQueueConfigs:
{code:java}
LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
+ ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
+ ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
+ ", absoluteMaxCapacity=" + this.queueCapacities
.getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
+ aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
+ ", reservationsContinueLooking=" + reservationsContinueLooking
+ ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
+ ", priority=" + priority
+ ", allowZeroCapacitySum=" + allowZeroCapacitySum);
{code}
ParentQueue#toString:
{code:java}
public String toString() {
return queueName + ": " +
"numChildQueue= " + childQueues.size() + ", " +
"capacity=" + queueCapacities.getCapacity() + ", " +
"absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
"usedResources=" + queueUsage.getUsed() +
"usedCapacity=" + getUsedCapacity() + ", " +
"numApps=" + getNumApplications() + ", " +
"numContainers=" + getNumContainers();
}
{code}
LeafQueue#setupQueueConfigs:
{code:java}
LOG.info(
"Initializing " + getQueuePath() + "\n" + "capacity = "
+ queueCapacities.getCapacity()
+ " [= (float) configuredCapacity / 100 ]" + "\n"
+ "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
+ " [= parentAbsoluteCapacity * capacity ]" + "\n"
+ "maxCapacity = " + queueCapacities.getMaximumCapacity()
+ " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity = "
+ queueCapacities.getAbsoluteMaximumCapacity()
+ " [= 1.0 maximumCapacity undefined, "
+ "(parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise
]"
+ "\n" + "effectiveMinResource=" +
getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
+ " , effectiveMaxResource=" +
getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
+ "\n" + "userLimit = " + usersManager.getUserLimit()
+ " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
+ usersManager.getUserLimitFactor()
+ " [= configuredUserLimitFactor ]" + "\n" + "maxApplications = "
+ maxApplications
+ " [= configuredMaximumSystemApplicationsPerQueue or"
+ " (int)(configuredMaximumSystemApplications *
absoluteCapacity)]"
+ "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
+ " [= (int)(maxApplications * (userLimit / 100.0f) * "
+ "userLimitFactor) ]" + "\n"
+ "maxParallelApps = " + getMaxParallelApps() + "\n"
+ "usedCapacity = " +
+ queueCapacities.getUsedCapacity() + " [= usedResourcesMemory / "
+ "(clusterResourceMemory * absoluteCapacity)]" + "\n"
+ "absoluteUsedCapacity = " + absoluteUsedCapacity
+ " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
+ "maxAMResourcePerQueuePercent = " + maxAMResourcePerQueuePercent
+ " [= configuredMaximumAMResourcePercent ]" + "\n"
+ "minimumAllocationFactor = " + minimumAllocationFactor
+ " [= (float)(maximumAllocationMemory - minimumAllocationMemory)
/ "
+ "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
+ maximumAllocation + " [= configuredMaxAllocation ]" + "\n"
+ "numContainers = " + numContainers
+ " [= currentNumContainers ]" + "\n" + "state = " + getState()
+ " [= configuredState ]" + "\n" + "acls = " + aclsString
+ " [= configuredAcls ]" + "\n"
+ "nodeLocalityDelay = " + nodeLocalityDelay + "\n"
+ "rackLocalityAdditionalDelay = "
+ rackLocalityAdditionalDelay + "\n"
+ "labels=" + labelStrBuilder.toString() + "\n"
+ "reservationsContinueLooking = "
+ reservationsContinueLooking + "\n" + "preemptionDisabled = "
+ getPreemptionDisabled() + "\n" + "defaultAppPriorityPerQueue = "
+ defaultAppPriorit
[jira] [Updated] (YARN-10626) Log resource allocation in NM log at container start time
[ https://issues.apache.org/jira/browse/YARN-10626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Badger updated YARN-10626: --- Attachment: YARN-10626.002.patch > Log resource allocation in NM log at container start time > - > > Key: YARN-10626 > URL: https://issues.apache.org/jira/browse/YARN-10626 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-10626.001.patch, YARN-10626.002.patch > > > As far as I can tell, there are no resource allocation logs in the NM log for > the various containers that are scheduled. These can be useful when trying to > debug what resources were requested vs what resources were actually > allocated. This is especially useful when debugging upstream technology > changes to make sure that they are correctly interpreting and passing down > resource parameters -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10626) Log resource allocation in NM log at container start time
[
https://issues.apache.org/jira/browse/YARN-10626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284922#comment-17284922
]
Hadoop QA commented on YARN-10626:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 10s{color}
| {color:red}{color} | {color:red} YARN-10626 does not apply to trunk. Rebase
required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for
help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-10626 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13020492/YARN-10626.001.patch |
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/617/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
> Log resource allocation in NM log at container start time
> -
>
> Key: YARN-10626
> URL: https://issues.apache.org/jira/browse/YARN-10626
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-10626.001.patch
>
>
> As far as I can tell, there are no resource allocation logs in the NM log for
> the various containers that are scheduled. These can be useful when trying to
> debug what resources were requested vs what resources were actually
> allocated. This is especially useful when debugging upstream technology
> changes to make sure that they are correctly interpreting and passing down
> resource parameters
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10626) Log resource allocation in NM log at container start time
[ https://issues.apache.org/jira/browse/YARN-10626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Badger updated YARN-10626: --- Attachment: YARN-10626.001.patch > Log resource allocation in NM log at container start time > - > > Key: YARN-10626 > URL: https://issues.apache.org/jira/browse/YARN-10626 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-10626.001.patch > > > As far as I can tell, there are no resource allocation logs in the NM log for > the various containers that are scheduled. These can be useful when trying to > debug what resources were requested vs what resources were actually > allocated. This is especially useful when debugging upstream technology > changes to make sure that they are correctly interpreting and passing down > resource parameters -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10626) Log resource allocation in NM log at container start time
Eric Badger created YARN-10626: -- Summary: Log resource allocation in NM log at container start time Key: YARN-10626 URL: https://issues.apache.org/jira/browse/YARN-10626 Project: Hadoop YARN Issue Type: Improvement Reporter: Eric Badger Assignee: Eric Badger As far as I can tell, there are no resource allocation logs in the NM log for the various containers that are scheduled. These can be useful when trying to debug what resources were requested vs what resources were actually allocated. This is especially useful when debugging upstream technology changes to make sure that they are correctly interpreting and passing down resource parameters -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10625) FairScheduler: add global flag to disable AM-preemption
[
https://issues.apache.org/jira/browse/YARN-10625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284823#comment-17284823
]
Hadoop QA commented on YARN-10625:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
23s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
59s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 51s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
50s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 12s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color
[jira] [Commented] (YARN-10625) FairScheduler: add global flag to disable AM-preemption
[ https://issues.apache.org/jira/browse/YARN-10625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284777#comment-17284777 ] Benjamin Teke commented on YARN-10625: -- [~pbacsko] thanks for the patch. LGTM +1 (non-binding). I agree with the doc update, as the global property doesn't overwrite the queue level ones, which can be non-trivial. > FairScheduler: add global flag to disable AM-preemption > --- > > Key: YARN-10625 > URL: https://issues.apache.org/jira/browse/YARN-10625 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 3.3.0 >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-10625-001.patch > > > YARN-9537 added a feature to disable AM preemption on a per queue basis. > This is a nice enhancement, but it's very inconvenient if the cluster has a > lot of queues or queues dynamically created/deleted regularly (static queue > configuration changes). > It's a legitimate use-case to have AM preemption turned off completely. To > make it easier, add property which acts as a global flag for this feature. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10625) FairScheduler: add global flag to disable AM-preemption
[ https://issues.apache.org/jira/browse/YARN-10625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284749#comment-17284749 ] Peter Bacsko commented on YARN-10625: - [~bteke] / [~snemeth] please review this patch if you have some free cycles. I think a doc update is also desirable, I'd do that in a follow-up JIRA. > FairScheduler: add global flag to disable AM-preemption > --- > > Key: YARN-10625 > URL: https://issues.apache.org/jira/browse/YARN-10625 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 3.3.0 >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-10625-001.patch > > > YARN-9537 added a feature to disable AM preemption on a per queue basis. > This is a nice enhancement, but it's very inconvenient if the cluster has a > lot of queues or queues dynamically created/deleted regularly (static queue > configuration changes). > It's a legitimate use-case to have AM preemption turned off completely. To > make it easier, add property which acts as a global flag for this feature. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10622) Fix preemption policy to exclude childless ParentQueues
[
https://issues.apache.org/jira/browse/YARN-10622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szilard Nemeth updated YARN-10622:
--
Fix Version/s: 3.4.0
> Fix preemption policy to exclude childless ParentQueues
> ---
>
> Key: YARN-10622
> URL: https://issues.apache.org/jira/browse/YARN-10622
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: YARN-10622.001.patch
>
>
> ProportionalCapacityPreemptionPolicy selects the potential LeafQueues to be
> preempted by this logic:
> {code:java}
> private Set getLeafQueueNames(TempQueuePerPartition q) {
> // If its a ManagedParentQueue, it might not have any children
> if ((q.children == null || q.children.isEmpty())
> && !(q.parentQueue instanceof ManagedParentQueue)) {
> return ImmutableSet.of(q.queueName);
> }
> Set leafQueueNames = new HashSet<>();
> for (TempQueuePerPartition child : q.children) {
> leafQueueNames.addAll(getLeafQueueNames(child));
> }
> return leafQueueNames;
> }
> {code}
> This, however does not take childless ParentQueues (which was introduced in
> YARN-10596) into account.
> A childless ParentQueue will throw a NPE in
> FifoCandidatesSelector#selectCandidates:
> {code:java}
> LeafQueue leafQueue = preemptionContext.getQueueByPartition(queueName,
> RMNodeLabelsManager.NO_LABEL).leafQueue;
> {code}
> TempQueuePerPartition has a leafQueue member variable, which is null, if the
> queue is not a LeafQueue. In case of childless ParentQueue, it is null, but
> its name is present in the leafQueueNames as stated before.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10622) Fix preemption policy to exclude childless ParentQueues
[
https://issues.apache.org/jira/browse/YARN-10622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284739#comment-17284739
]
Szilard Nemeth commented on YARN-10622:
---
Thanks [~gandras] for working on this,
Patch LGTM, committed to trunk.
> Fix preemption policy to exclude childless ParentQueues
> ---
>
> Key: YARN-10622
> URL: https://issues.apache.org/jira/browse/YARN-10622
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: YARN-10622.001.patch
>
>
> ProportionalCapacityPreemptionPolicy selects the potential LeafQueues to be
> preempted by this logic:
> {code:java}
> private Set getLeafQueueNames(TempQueuePerPartition q) {
> // If its a ManagedParentQueue, it might not have any children
> if ((q.children == null || q.children.isEmpty())
> && !(q.parentQueue instanceof ManagedParentQueue)) {
> return ImmutableSet.of(q.queueName);
> }
> Set leafQueueNames = new HashSet<>();
> for (TempQueuePerPartition child : q.children) {
> leafQueueNames.addAll(getLeafQueueNames(child));
> }
> return leafQueueNames;
> }
> {code}
> This, however does not take childless ParentQueues (which was introduced in
> YARN-10596) into account.
> A childless ParentQueue will throw a NPE in
> FifoCandidatesSelector#selectCandidates:
> {code:java}
> LeafQueue leafQueue = preemptionContext.getQueueByPartition(queueName,
> RMNodeLabelsManager.NO_LABEL).leafQueue;
> {code}
> TempQueuePerPartition has a leafQueue member variable, which is null, if the
> queue is not a LeafQueue. In case of childless ParentQueue, it is null, but
> its name is present in the leafQueueNames as stated before.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10625) FairScheduler: add global flag to disable AM-preemption
[ https://issues.apache.org/jira/browse/YARN-10625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-10625: Attachment: YARN-10625-001.patch > FairScheduler: add global flag to disable AM-preemption > --- > > Key: YARN-10625 > URL: https://issues.apache.org/jira/browse/YARN-10625 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 3.3.0 >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-10625-001.patch > > > YARN-9537 added a feature to disable AM preemption on a per queue basis. > This is a nice enhancement, but it's very inconvenient if the cluster has a > lot of queues or queues dynamically created/deleted regularly (static queue > configuration changes). > It's a legitimate use-case to have AM preemption turned off completely. To > make it easier, add property which acts as a global flag for this feature. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10624) Support max queues limit configuration in new auto created queue, consistent with old auto created.
[ https://issues.apache.org/jira/browse/YARN-10624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-10624: -- Fix Version/s: 3.4.0 > Support max queues limit configuration in new auto created queue, consistent > with old auto created. > --- > > Key: YARN-10624 > URL: https://issues.apache.org/jira/browse/YARN-10624 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10624.001.patch, YARN-10624.002.patch > > > Since old created leaf queue has the max leaf queues limit, i think we also > should support this in new auto created queue, both the auto created leaf and > the auto created parent need limits. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10624) Support max queues limit configuration in new auto created queue, consistent with old auto created.
[ https://issues.apache.org/jira/browse/YARN-10624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284709#comment-17284709 ] Szilard Nemeth commented on YARN-10624: --- Thanks [~zhuqi] for working on this. Patch LGTM, committed to trunk. Thanks [~gandras] and [~bteke] for the reviews. > Support max queues limit configuration in new auto created queue, consistent > with old auto created. > --- > > Key: YARN-10624 > URL: https://issues.apache.org/jira/browse/YARN-10624 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10624.001.patch, YARN-10624.002.patch > > > Since old created leaf queue has the max leaf queues limit, i think we also > should support this in new auto created queue, both the auto created leaf and > the auto created parent need limits. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used
[ https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284706#comment-17284706 ] Qi Zhu commented on YARN-10532: --- Thanks [~gandras] for your patient review and valid suggestions, i will add more logging info in next patch. > Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is > not being used > > > Key: YARN-10532 > URL: https://issues.apache.org/jira/browse/YARN-10532 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Qi Zhu >Priority: Major > Attachments: YARN-10532.001.patch, YARN-10532.002.patch, > YARN-10532.003.patch, YARN-10532.004.patch, YARN-10532.005.patch, > YARN-10532.006.patch, YARN-10532.007.patch, YARN-10532.008.patch, > YARN-10532.009.patch, YARN-10532.010.patch, YARN-10532.011.patch, > YARN-10532.012.patch, YARN-10532.013.patch, YARN-10532.014.patch, > YARN-10532.015.patch, YARN-10532.016.patch, YARN-10532.017.patch, > YARN-10532.018.patch, YARN-10532.019.patch, YARN-10532.020.patch, > image-2021-02-12-21-32-02-267.png > > > It's better if we can delete auto-created queues when they are not in use for > a period of time (like 5 mins). It will be helpful when we have a large > number of auto-created queues (e.g. from 500 users), but only a small subset > of queues are actively used. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10624) Support max queues limit configuration in new auto created queue, consistent with old auto created.
[ https://issues.apache.org/jira/browse/YARN-10624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284704#comment-17284704 ] Qi Zhu commented on YARN-10624: --- Thanks for [~gandras] for your review. [~ztang] [~snemeth] If you could help commit it, when you are free, thanks. > Support max queues limit configuration in new auto created queue, consistent > with old auto created. > --- > > Key: YARN-10624 > URL: https://issues.apache.org/jira/browse/YARN-10624 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Attachments: YARN-10624.001.patch, YARN-10624.002.patch > > > Since old created leaf queue has the max leaf queues limit, i think we also > should support this in new auto created queue, both the auto created leaf and > the auto created parent need limits. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10622) Fix preemption policy to exclude childless ParentQueues
[
https://issues.apache.org/jira/browse/YARN-10622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284671#comment-17284671
]
Hadoop QA commented on YARN-10622:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
12s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
41s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 5s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
50s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 55s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green}{color} | {color
[jira] [Commented] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used
[ https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284614#comment-17284614 ] Andras Gyori commented on YARN-10532: - Thank you [~zhuqi], the latest patch looks good to me. If someone else has feedbacks, can you make sure to add some additional logging info in policy? In this current state, I only saw Removing queue entry, which could be elaborated a bit. My suggestion is to add a logging info when you send a deletion event to CS in the policy. > Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is > not being used > > > Key: YARN-10532 > URL: https://issues.apache.org/jira/browse/YARN-10532 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Qi Zhu >Priority: Major > Attachments: YARN-10532.001.patch, YARN-10532.002.patch, > YARN-10532.003.patch, YARN-10532.004.patch, YARN-10532.005.patch, > YARN-10532.006.patch, YARN-10532.007.patch, YARN-10532.008.patch, > YARN-10532.009.patch, YARN-10532.010.patch, YARN-10532.011.patch, > YARN-10532.012.patch, YARN-10532.013.patch, YARN-10532.014.patch, > YARN-10532.015.patch, YARN-10532.016.patch, YARN-10532.017.patch, > YARN-10532.018.patch, YARN-10532.019.patch, YARN-10532.020.patch, > image-2021-02-12-21-32-02-267.png > > > It's better if we can delete auto-created queues when they are not in use for > a period of time (like 5 mins). It will be helpful when we have a large > number of auto-created queues (e.g. from 500 users), but only a small subset > of queues are actively used. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10622) Fix preemption policy to exclude childless ParentQueues
[
https://issues.apache.org/jira/browse/YARN-10622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Gyori updated YARN-10622:
Attachment: YARN-10622.001.patch
> Fix preemption policy to exclude childless ParentQueues
> ---
>
> Key: YARN-10622
> URL: https://issues.apache.org/jira/browse/YARN-10622
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10622.001.patch
>
>
> ProportionalCapacityPreemptionPolicy selects the potential LeafQueues to be
> preempted by this logic:
> {code:java}
> private Set getLeafQueueNames(TempQueuePerPartition q) {
> // If its a ManagedParentQueue, it might not have any children
> if ((q.children == null || q.children.isEmpty())
> && !(q.parentQueue instanceof ManagedParentQueue)) {
> return ImmutableSet.of(q.queueName);
> }
> Set leafQueueNames = new HashSet<>();
> for (TempQueuePerPartition child : q.children) {
> leafQueueNames.addAll(getLeafQueueNames(child));
> }
> return leafQueueNames;
> }
> {code}
> This, however does not take childless ParentQueues (which was introduced in
> YARN-10596) into account.
> A childless ParentQueue will throw a NPE in
> FifoCandidatesSelector#selectCandidates:
> {code:java}
> LeafQueue leafQueue = preemptionContext.getQueueByPartition(queueName,
> RMNodeLabelsManager.NO_LABEL).leafQueue;
> {code}
> TempQueuePerPartition has a leafQueue member variable, which is null, if the
> queue is not a LeafQueue. In case of childless ParentQueue, it is null, but
> its name is present in the leafQueueNames as stated before.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10622) Fix preemption policy to exclude childless ParentQueues
[
https://issues.apache.org/jira/browse/YARN-10622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Gyori updated YARN-10622:
Attachment: YARN-10622.001.patch
> Fix preemption policy to exclude childless ParentQueues
> ---
>
> Key: YARN-10622
> URL: https://issues.apache.org/jira/browse/YARN-10622
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10622.001.patch
>
>
> ProportionalCapacityPreemptionPolicy selects the potential LeafQueues to be
> preempted by this logic:
> {code:java}
> private Set getLeafQueueNames(TempQueuePerPartition q) {
> // If its a ManagedParentQueue, it might not have any children
> if ((q.children == null || q.children.isEmpty())
> && !(q.parentQueue instanceof ManagedParentQueue)) {
> return ImmutableSet.of(q.queueName);
> }
> Set leafQueueNames = new HashSet<>();
> for (TempQueuePerPartition child : q.children) {
> leafQueueNames.addAll(getLeafQueueNames(child));
> }
> return leafQueueNames;
> }
> {code}
> This, however does not take childless ParentQueues (which was introduced in
> YARN-10596) into account.
> A childless ParentQueue will throw a NPE in
> FifoCandidatesSelector#selectCandidates:
> {code:java}
> LeafQueue leafQueue = preemptionContext.getQueueByPartition(queueName,
> RMNodeLabelsManager.NO_LABEL).leafQueue;
> {code}
> TempQueuePerPartition has a leafQueue member variable, which is null, if the
> queue is not a LeafQueue. In case of childless ParentQueue, it is null, but
> its name is present in the leafQueueNames as stated before.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10622) Fix preemption policy to exclude childless ParentQueues
[
https://issues.apache.org/jira/browse/YARN-10622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Gyori updated YARN-10622:
Attachment: (was: YARN-10622.001.patch)
> Fix preemption policy to exclude childless ParentQueues
> ---
>
> Key: YARN-10622
> URL: https://issues.apache.org/jira/browse/YARN-10622
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10622.001.patch
>
>
> ProportionalCapacityPreemptionPolicy selects the potential LeafQueues to be
> preempted by this logic:
> {code:java}
> private Set getLeafQueueNames(TempQueuePerPartition q) {
> // If its a ManagedParentQueue, it might not have any children
> if ((q.children == null || q.children.isEmpty())
> && !(q.parentQueue instanceof ManagedParentQueue)) {
> return ImmutableSet.of(q.queueName);
> }
> Set leafQueueNames = new HashSet<>();
> for (TempQueuePerPartition child : q.children) {
> leafQueueNames.addAll(getLeafQueueNames(child));
> }
> return leafQueueNames;
> }
> {code}
> This, however does not take childless ParentQueues (which was introduced in
> YARN-10596) into account.
> A childless ParentQueue will throw a NPE in
> FifoCandidatesSelector#selectCandidates:
> {code:java}
> LeafQueue leafQueue = preemptionContext.getQueueByPartition(queueName,
> RMNodeLabelsManager.NO_LABEL).leafQueue;
> {code}
> TempQueuePerPartition has a leafQueue member variable, which is null, if the
> queue is not a LeafQueue. In case of childless ParentQueue, it is null, but
> its name is present in the leafQueueNames as stated before.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10622) Fix preemption policy to exclude childless ParentQueues
[
https://issues.apache.org/jira/browse/YARN-10622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Gyori updated YARN-10622:
Attachment: (was: YARN-10622.001.patch)
> Fix preemption policy to exclude childless ParentQueues
> ---
>
> Key: YARN-10622
> URL: https://issues.apache.org/jira/browse/YARN-10622
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10622.001.patch
>
>
> ProportionalCapacityPreemptionPolicy selects the potential LeafQueues to be
> preempted by this logic:
> {code:java}
> private Set getLeafQueueNames(TempQueuePerPartition q) {
> // If its a ManagedParentQueue, it might not have any children
> if ((q.children == null || q.children.isEmpty())
> && !(q.parentQueue instanceof ManagedParentQueue)) {
> return ImmutableSet.of(q.queueName);
> }
> Set leafQueueNames = new HashSet<>();
> for (TempQueuePerPartition child : q.children) {
> leafQueueNames.addAll(getLeafQueueNames(child));
> }
> return leafQueueNames;
> }
> {code}
> This, however does not take childless ParentQueues (which was introduced in
> YARN-10596) into account.
> A childless ParentQueue will throw a NPE in
> FifoCandidatesSelector#selectCandidates:
> {code:java}
> LeafQueue leafQueue = preemptionContext.getQueueByPartition(queueName,
> RMNodeLabelsManager.NO_LABEL).leafQueue;
> {code}
> TempQueuePerPartition has a leafQueue member variable, which is null, if the
> queue is not a LeafQueue. In case of childless ParentQueue, it is null, but
> its name is present in the leafQueueNames as stated before.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10622) Fix preemption policy to exclude childless ParentQueues
[
https://issues.apache.org/jira/browse/YARN-10622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Gyori updated YARN-10622:
Attachment: YARN-10622.001.patch
> Fix preemption policy to exclude childless ParentQueues
> ---
>
> Key: YARN-10622
> URL: https://issues.apache.org/jira/browse/YARN-10622
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10622.001.patch
>
>
> ProportionalCapacityPreemptionPolicy selects the potential LeafQueues to be
> preempted by this logic:
> {code:java}
> private Set getLeafQueueNames(TempQueuePerPartition q) {
> // If its a ManagedParentQueue, it might not have any children
> if ((q.children == null || q.children.isEmpty())
> && !(q.parentQueue instanceof ManagedParentQueue)) {
> return ImmutableSet.of(q.queueName);
> }
> Set leafQueueNames = new HashSet<>();
> for (TempQueuePerPartition child : q.children) {
> leafQueueNames.addAll(getLeafQueueNames(child));
> }
> return leafQueueNames;
> }
> {code}
> This, however does not take childless ParentQueues (which was introduced in
> YARN-10596) into account.
> A childless ParentQueue will throw a NPE in
> FifoCandidatesSelector#selectCandidates:
> {code:java}
> LeafQueue leafQueue = preemptionContext.getQueueByPartition(queueName,
> RMNodeLabelsManager.NO_LABEL).leafQueue;
> {code}
> TempQueuePerPartition has a leafQueue member variable, which is null, if the
> queue is not a LeafQueue. In case of childless ParentQueue, it is null, but
> its name is present in the leafQueueNames as stated before.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10622) Fix preemption policy to exclude childless ParentQueues
[
https://issues.apache.org/jira/browse/YARN-10622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Gyori updated YARN-10622:
Attachment: (was: YARN-10622.001.patch)
> Fix preemption policy to exclude childless ParentQueues
> ---
>
> Key: YARN-10622
> URL: https://issues.apache.org/jira/browse/YARN-10622
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10622.001.patch
>
>
> ProportionalCapacityPreemptionPolicy selects the potential LeafQueues to be
> preempted by this logic:
> {code:java}
> private Set getLeafQueueNames(TempQueuePerPartition q) {
> // If its a ManagedParentQueue, it might not have any children
> if ((q.children == null || q.children.isEmpty())
> && !(q.parentQueue instanceof ManagedParentQueue)) {
> return ImmutableSet.of(q.queueName);
> }
> Set leafQueueNames = new HashSet<>();
> for (TempQueuePerPartition child : q.children) {
> leafQueueNames.addAll(getLeafQueueNames(child));
> }
> return leafQueueNames;
> }
> {code}
> This, however does not take childless ParentQueues (which was introduced in
> YARN-10596) into account.
> A childless ParentQueue will throw a NPE in
> FifoCandidatesSelector#selectCandidates:
> {code:java}
> LeafQueue leafQueue = preemptionContext.getQueueByPartition(queueName,
> RMNodeLabelsManager.NO_LABEL).leafQueue;
> {code}
> TempQueuePerPartition has a leafQueue member variable, which is null, if the
> queue is not a LeafQueue. In case of childless ParentQueue, it is null, but
> its name is present in the leafQueueNames as stated before.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10622) Fix preemption policy to exclude childless ParentQueues
[
https://issues.apache.org/jira/browse/YARN-10622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Gyori updated YARN-10622:
Attachment: YARN-10622.001.patch
> Fix preemption policy to exclude childless ParentQueues
> ---
>
> Key: YARN-10622
> URL: https://issues.apache.org/jira/browse/YARN-10622
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10622.001.patch
>
>
> ProportionalCapacityPreemptionPolicy selects the potential LeafQueues to be
> preempted by this logic:
> {code:java}
> private Set getLeafQueueNames(TempQueuePerPartition q) {
> // If its a ManagedParentQueue, it might not have any children
> if ((q.children == null || q.children.isEmpty())
> && !(q.parentQueue instanceof ManagedParentQueue)) {
> return ImmutableSet.of(q.queueName);
> }
> Set leafQueueNames = new HashSet<>();
> for (TempQueuePerPartition child : q.children) {
> leafQueueNames.addAll(getLeafQueueNames(child));
> }
> return leafQueueNames;
> }
> {code}
> This, however does not take childless ParentQueues (which was introduced in
> YARN-10596) into account.
> A childless ParentQueue will throw a NPE in
> FifoCandidatesSelector#selectCandidates:
> {code:java}
> LeafQueue leafQueue = preemptionContext.getQueueByPartition(queueName,
> RMNodeLabelsManager.NO_LABEL).leafQueue;
> {code}
> TempQueuePerPartition has a leafQueue member variable, which is null, if the
> queue is not a LeafQueue. In case of childless ParentQueue, it is null, but
> its name is present in the leafQueueNames as stated before.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
