[jira] [Commented] (YARN-10622) Fix preemption policy to exclude childless ParentQueues
[ https://issues.apache.org/jira/browse/YARN-10622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17284739#comment-17284739 ] Szilard Nemeth commented on YARN-10622: --- Thanks [~gandras] for working on this, Patch LGTM, committed to trunk. > Fix preemption policy to exclude childless ParentQueues > --- > > Key: YARN-10622 > URL: https://issues.apache.org/jira/browse/YARN-10622 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Andras Gyori >Assignee: Andras Gyori >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10622.001.patch > > > ProportionalCapacityPreemptionPolicy selects the potential LeafQueues to be > preempted by this logic: > {code:java} > private Set getLeafQueueNames(TempQueuePerPartition q) { > // If its a ManagedParentQueue, it might not have any children > if ((q.children == null || q.children.isEmpty()) > && !(q.parentQueue instanceof ManagedParentQueue)) { > return ImmutableSet.of(q.queueName); > } > Set leafQueueNames = new HashSet<>(); > for (TempQueuePerPartition child : q.children) { > leafQueueNames.addAll(getLeafQueueNames(child)); > } > return leafQueueNames; > } > {code} > This, however does not take childless ParentQueues (which was introduced in > YARN-10596) into account. > A childless ParentQueue will throw a NPE in > FifoCandidatesSelector#selectCandidates: > {code:java} > LeafQueue leafQueue = preemptionContext.getQueueByPartition(queueName, > RMNodeLabelsManager.NO_LABEL).leafQueue; > {code} > TempQueuePerPartition has a leafQueue member variable, which is null, if the > queue is not a LeafQueue. In case of childless ParentQueue, it is null, but > its name is present in the leafQueueNames as stated before. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10622) Fix preemption policy to exclude childless ParentQueues
[ https://issues.apache.org/jira/browse/YARN-10622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17284671#comment-17284671 ] Hadoop QA commented on YARN-10622: -- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m 12s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m 41s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 51s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 53s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 17m 5s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m 50s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 46s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 49s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 54s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 46s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 46s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 48s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 55s{color} | {color:green}{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s{color} | {color:green}{color} | {color:green} the
[jira] [Commented] (YARN-10622) Fix preemption policy to exclude childless ParentQueues
[ https://issues.apache.org/jira/browse/YARN-10622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17282473#comment-17282473 ] Qi Zhu commented on YARN-10622: --- Thanks [~gandras] for good finding. > Fix preemption policy to exclude childless ParentQueues > --- > > Key: YARN-10622 > URL: https://issues.apache.org/jira/browse/YARN-10622 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Andras Gyori >Assignee: Andras Gyori >Priority: Major > > ProportionalCapacityPreemptionPolicy selects the potential LeafQueues to be > preempted by this logic: > {code:java} > private Set getLeafQueueNames(TempQueuePerPartition q) { > // If its a ManagedParentQueue, it might not have any children > if ((q.children == null || q.children.isEmpty()) > && !(q.parentQueue instanceof ManagedParentQueue)) { > return ImmutableSet.of(q.queueName); > } > Set leafQueueNames = new HashSet<>(); > for (TempQueuePerPartition child : q.children) { > leafQueueNames.addAll(getLeafQueueNames(child)); > } > return leafQueueNames; > } > {code} > This, however does not take childless ParentQueues (which was introduced in > YARN-10596) into account. > A childless ParentQueue will throw a NPE in > FifoCandidatesSelector#selectCandidates: > {code:java} > LeafQueue leafQueue = preemptionContext.getQueueByPartition(queueName, > RMNodeLabelsManager.NO_LABEL).leafQueue; > {code} > TempQueuePerPartition has a leafQueue member variable, which is null, if the > queue is not a LeafQueue. In case of childless ParentQueue, it is null, but > its name is present in the leafQueueNames as stated before. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org