[jira] [Updated] (YARN-10599) fs2cs should generate new "auto-queue-creation-v2.enabled" properties for all parents
[
https://issues.apache.org/jira/browse/YARN-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10599:
Description:
The property
{{yarn.scheduler.capacity..auto-queue-creation-v2.enabled}} is not
enabled by default for parent queues. However, users who migrate from FS need
this property enabled for all parents queues, because FS allows them to have
dynamic queues.
Note that this is only relevant if we convert directly to weights, it's not
needed in percentage mode.
was:
The property
{{yarn.scheduler.capacity..auto-queue-creation-v2.enabled}} is not
enabled by default for parent queues. However, users who migrate from FS need
this property enabled for all parents, because FS allows all parents to have
dynamic queues underneath them.
Note that this is only relevant if we convert directly to weights, it's not
needed in percentage mode.
> fs2cs should generate new "auto-queue-creation-v2.enabled" properties for all
> parents
> -
>
> Key: YARN-10599
> URL: https://issues.apache.org/jira/browse/YARN-10599
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Labels: fs2cs
>
> The property
> {{yarn.scheduler.capacity..auto-queue-creation-v2.enabled}} is
> not enabled by default for parent queues. However, users who migrate from FS
> need this property enabled for all parents queues, because FS allows them to
> have dynamic queues.
> Note that this is only relevant if we convert directly to weights, it's not
> needed in percentage mode.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10598) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information
[ https://issues.apache.org/jira/browse/YARN-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272546#comment-17272546 ] Sunil G commented on YARN-10598: [~bteke] [~snemeth] queueType cannot be changed as It will be incompatible. It's better we add only a new variable. > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the > creation type with additional information > -- > > Key: YARN-10598 > URL: https://issues.apache.org/jira/browse/YARN-10598 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Attachments: YARN-10598.001.patch > > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > Auto-queue creation has been also implemented with YARN-10506. > Connected to this effort, we would like to expose the type of the queue with > the RM's /scheduler REST endpoint. > To extend/modify the values added in YARN-10581 these 3 fields will describe > a queue: > * queueType : *parent/leaf* > * creationMethod : *static/dynamicLegacy/dynamicFlexible* > * autoCreationEligibility : *off/legacy/flexible* > After this change here are some example cases: > * Static parent queue which has the auto-creation-enabled-v2 false: > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *off* > * Static managed parent (can have dynamic children): > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *legacy* > * Legacy auto-created leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicLegacy* > ** autoCreationEligibility : *off* > * Auto-created (v2) parent queue, (implicitly) auto-creation-enabled-v2 > true: > ** queueType : *parent* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *flexible* > * Auto-created (v2) leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *off* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10598) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information
[
https://issues.apache.org/jira/browse/YARN-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272378#comment-17272378
]
Hadoop QA commented on YARN-10598:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
15s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 3
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
3s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 58s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
48s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 41s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/550/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
| {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 9 new + 116 unchanged - 1 fixed = 125 total (was 117)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 4s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
|
[jira] [Commented] (YARN-10585) Create a class which can convert from legacy mapping rule format to the new JSON format
[
https://issues.apache.org/jira/browse/YARN-10585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272281#comment-17272281
]
Hadoop QA commented on YARN-10585:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 25m
49s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 3
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
59s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
19s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
21m 25s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m
43s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
39s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
12s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
24s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
24s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
16s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 48s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/547/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
| {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 2 new + 5 unchanged - 0 fixed = 7 total (was 5) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 5s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
|
[jira] [Updated] (YARN-10585) Create a class which can convert from legacy mapping rule format to the new JSON format
[ https://issues.apache.org/jira/browse/YARN-10585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-10585: -- Fix Version/s: 3.4.0 > Create a class which can convert from legacy mapping rule format to the new > JSON format > --- > > Key: YARN-10585 > URL: https://issues.apache.org/jira/browse/YARN-10585 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10585.001.patch, YARN-10585.002.patch, > YARN-10585.003.patch > > > To make transition easier we need to create tooling to support the migration > effort. The first step is to create a class which can migrate from legacy to > the new JSON format. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10585) Create a class which can convert from legacy mapping rule format to the new JSON format
[ https://issues.apache.org/jira/browse/YARN-10585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272274#comment-17272274 ] Szilard Nemeth commented on YARN-10585: --- Thanks [~shuzirra] for this, good job! Latest patch LGTM, committed to trunk. > Create a class which can convert from legacy mapping rule format to the new > JSON format > --- > > Key: YARN-10585 > URL: https://issues.apache.org/jira/browse/YARN-10585 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10585.001.patch, YARN-10585.002.patch, > YARN-10585.003.patch > > > To make transition easier we need to create tooling to support the migration > effort. The first step is to create a class which can migrate from legacy to > the new JSON format. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10598) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information
[ https://issues.apache.org/jira/browse/YARN-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke updated YARN-10598: - Attachment: YARN-10598.001.patch > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the > creation type with additional information > -- > > Key: YARN-10598 > URL: https://issues.apache.org/jira/browse/YARN-10598 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Attachments: YARN-10598.001.patch > > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > Auto-queue creation has been also implemented with YARN-10506. > Connected to this effort, we would like to expose the type of the queue with > the RM's /scheduler REST endpoint. > To extend/modify the values added in YARN-10581 these 3 fields will describe > a queue: > * queueType : *parent/leaf* > * creationMethod : *static/dynamicLegacy/dynamicFlexible* > * autoCreationEligibility : *off/legacy/flexible* > After this change here are some example cases: > * Static parent queue which has the auto-creation-enabled-v2 false: > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *off* > * Static managed parent (can have dynamic children): > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *legacy* > * Legacy auto-created leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicLegacy* > ** autoCreationEligibility : *off* > * Auto-created (v2) parent queue, (implicitly) auto-creation-enabled-v2 > true: > ** queueType : *parent* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *flexible* > * Auto-created (v2) leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *off* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10596) Allow static definition of childless ParentQueues with auto-queue-creation-v2 enabled
[
https://issues.apache.org/jira/browse/YARN-10596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272247#comment-17272247
]
Hadoop QA commented on YARN-10596:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
42s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 2
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
52s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 34s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
46s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 50s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green}{color} | {color:green} the
[jira] [Updated] (YARN-10587) Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode.
[ https://issues.apache.org/jira/browse/YARN-10587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-10587: -- Fix Version/s: 3.4.0 > Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode. > - > > Key: YARN-10587 > URL: https://issues.apache.org/jira/browse/YARN-10587 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10587.001.patch, YARN-10587.002.patch > > > When introduced YARN-10504. > The logic related to auto created leaf queue changed. > The test in testAutoCreateLeafQueueCreation failed, we should fix the Error. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10531) Be able to disable user limit factor for CapacityScheduler Leaf Queue
[
https://issues.apache.org/jira/browse/YARN-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-10531:
--
Fix Version/s: 3.4.0
> Be able to disable user limit factor for CapacityScheduler Leaf Queue
> -
>
> Key: YARN-10531
> URL: https://issues.apache.org/jira/browse/YARN-10531
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: zhuqi
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: YARN-10531.001.patch, YARN-10531.002.patch,
> YARN-10531.003.patch, YARN-10531.004.patch, YARN-10531.005.patch,
> YARN-10531.006.patch, YARN-10531.007.patch
>
>
> User limit factor is used to define max cap of how much resource can be
> consumed by single user.
> Under Auto Queue Creation context, it doesn't make much sense to set user
> limit factor, because initially every queue will set weight to 1.0, we want
> user can consume more resource if possible. It is hard to pre-determine how
> to set up user limit factor. So it makes more sense to add a new value (like
> -1) to indicate we will disable user limit factor
> Logic need to be changed is below:
> (Inside LeafQueue.java)
> {code}
> Resource maxUserLimit = Resources.none();
> if (schedulingMode == SchedulingMode.RESPECT_PARTITION_EXCLUSIVITY) {
> maxUserLimit = Resources.multiplyAndRoundDown(queueCapacity,
> getUserLimitFactor());
> } else if (schedulingMode == SchedulingMode.IGNORE_PARTITION_EXCLUSIVITY)
> {
> maxUserLimit = partitionResource;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10428) Zombie applications in the YARN queue using FAIR + sizebasedweight
[ https://issues.apache.org/jira/browse/YARN-10428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272213#comment-17272213 ] Guang Yang commented on YARN-10428: --- Hi [~gandras] : I'm pretty sure the root cause is the divide-by-zero error as shown in my patch. I have also included a text case which showed a reproduce procedure. Feel free to take it over and push it forward. > Zombie applications in the YARN queue using FAIR + sizebasedweight > -- > > Key: YARN-10428 > URL: https://issues.apache.org/jira/browse/YARN-10428 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Affects Versions: 2.8.5 >Reporter: Guang Yang >Priority: Major > Attachments: YARN-10428.001.patch, YARN-10428.002.patch > > > Seeing zombie jobs in the YARN queue that uses FAIR and size based weight > ordering policy . > *Detection:* > The YARN UI shows incorrect number of "Num Schedulable Applications". > *Impact:* > The queue has an upper limit of number of running applications, with zombie > job, it hits the limit even though the number of running applications is far > less than the limit. > *Workaround:* > **Fail-over and restart Resource Manager process. > *Analysis:* > **In the heap dump, we can find the zombie jobs in the `FairOderingPolicy# > schedulableEntities` (see attachment). Take application > "application_1599157165858_29429" for example, it is still in the > `FairOderingPolicy#schedulableEntities` set, however, if we check the log of > resource manager, we can see RM already tried to remove the application: > > ./yarn-yarn-resourcemanager-ip-172-21-153-252.log.2020-09-04-04:2020-09-04 > 04:32:19,730 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue > (ResourceManager Event Processor): Application removed - appId: > application_1599157165858_29429 user: svc_di_data_eng queue: core-data > #user-pending-applications: -3 #user-active-applications: 7 > #queue-pending-applications: 0 #queue-active-applications: 21 > > So it appears RM failed to removed the application from the set. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10599) fs2cs should generate new "auto-queue-creation-v2.enabled" properties for all parents
[
https://issues.apache.org/jira/browse/YARN-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10599:
Description:
The property
{{yarn.scheduler.capacity..auto-queue-creation-v2.enabled}} is not
enabled by default for parent queues. However, users who migrate from FS need
this property enabled for all parents, because FS allows all parents to have
dynamic queues underneath them.
Note that this is only relevant if we convert directly to weights, it's not
needed in percentage mode.
was:The property
{{yarn.scheduler.capacity..auto-queue-creation-v2.enabled}} is not
enabled by default for parent queues. However, users who migrate from FS need
this property enabled for all parents, because FS allows all parents to have
dynamic queues underneath them.
> fs2cs should generate new "auto-queue-creation-v2.enabled" properties for all
> parents
> -
>
> Key: YARN-10599
> URL: https://issues.apache.org/jira/browse/YARN-10599
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Labels: fs2cs
>
> The property
> {{yarn.scheduler.capacity..auto-queue-creation-v2.enabled}} is
> not enabled by default for parent queues. However, users who migrate from FS
> need this property enabled for all parents, because FS allows all parents to
> have dynamic queues underneath them.
> Note that this is only relevant if we convert directly to weights, it's not
> needed in percentage mode.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10576) Update Capacity Scheduler documentation about JSON-based placement mapping
[
https://issues.apache.org/jira/browse/YARN-10576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272173#comment-17272173
]
Hadoop QA commented on YARN-10576:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
41s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:blue}0{color} | {color:blue} markdownlint {color} | {color:blue} 0m
0s{color} | {color:blue}{color} | {color:blue} markdownlint was not available.
{color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
49s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
34m 31s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
13s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
18s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 16s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} || ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green}{color} | {color:green} The patch does not generate
ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 50m 6s{color} |
{color:black}{color} | {color:black}{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/549/artifact/out/Dockerfile
|
| JIRA Issue | YARN-10576 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13019421/YARN-10576-001.patch |
| Optional Tests | dupname asflicense mvnsite markdownlint |
| uname | Linux 9ec3cb938d69 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / a9ff726e42b |
| Max. process+thread count | 701 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site |
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/549/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
> Update Capacity Scheduler documentation about JSON-based placement mapping
> --
>
> Key: YARN-10576
> URL: https://issues.apache.org/jira/browse/YARN-10576
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10576-001.patch
>
>
> The weight mode and AQC also affects how the new placement engine in CS works
> and the documentation has to reflect that.
> Certain statements in the documentation are no longer valid, for example:
> * create flag: "Only applies to managed queue parents" - there is no
> ManagedParentQueue in weight mode.
> * "The nested rules primaryGroupUser and secondaryGroupUser expects the
> parent queues to exist, ie. they cannot be created automatically". This only
> applies to the legacy absolute/percentage mode.
> Find all statements that mentions possible limitations and fix them if
> necessary.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (YARN-10599) fs2cs should generate new "auto-queue-creation-v2.enabled" properties for all parents
[
https://issues.apache.org/jira/browse/YARN-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10599:
Description: The property
{{yarn.scheduler.capacity..auto-queue-creation-v2.enabled}} is not
enabled by default for parent queues. However, users who migrate from FS need
this property enabled for all parents, because FS allows all parents to have
dynamic queues underneath them.
> fs2cs should generate new "auto-queue-creation-v2.enabled" properties for all
> parents
> -
>
> Key: YARN-10599
> URL: https://issues.apache.org/jira/browse/YARN-10599
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Labels: fs2cs
>
> The property
> {{yarn.scheduler.capacity..auto-queue-creation-v2.enabled}} is
> not enabled by default for parent queues. However, users who migrate from FS
> need this property enabled for all parents, because FS allows all parents to
> have dynamic queues underneath them.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10584) Move the functionality of MappingRuleValidationHelper to CSQueueManager
[ https://issues.apache.org/jira/browse/YARN-10584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272168#comment-17272168 ] Andras Gyori commented on YARN-10584: - I think this should be done in tandem with YARN-10571. I have not looked in it yet, but I have a feeling that there could be some shared logic between CSAutoQueueHandler and MappingRuleValidationHelper. My idea was to move the handling logic to the QueueManager as well. > Move the functionality of MappingRuleValidationHelper to CSQueueManager > --- > > Key: YARN-10584 > URL: https://issues.apache.org/jira/browse/YARN-10584 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > > The only reason we created the helper class was to make the features > introduced in YARN-10506 accessible as soon as possible. Due to our current > test helpers and frameworks, integrating these changes into the > CapacitySchedulerQueueManager would have been more time consuming. > However we need to remove this standalone class and merge the functionality > into either the QueueManager or into CapacityScheduler. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10581) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include queue creation type for queues
[ https://issues.apache.org/jira/browse/YARN-10581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272169#comment-17272169 ] Szilard Nemeth commented on YARN-10581: --- Resolving this Jira as [~pbacsko] committed it last week. Thanks for the commit. > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include > queue creation type for queues > > > Key: YARN-10581 > URL: https://issues.apache.org/jira/browse/YARN-10581 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-10581.001.patch, YARN-10581.002.patch, > YARN-10581.003.patch > > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > Auto-queue creation has been also imlemented with YARN-10506. > Connected to this effort, we would like to expose the type of the queue with > the RM's /scheduler REST endpoint. > The queue type should hold these values: > * Auto-created parent queue: *autoCreatedParent* > * Auto-created leaf queue: *autoCreatedLeaf* > * Static parent: *staticParent* > * Static leaf: *staticLeaf* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10599) fs2cs should generate new "auto-queue-creation-v2.enabled" properties for all parents
Peter Bacsko created YARN-10599: --- Summary: fs2cs should generate new "auto-queue-creation-v2.enabled" properties for all parents Key: YARN-10599 URL: https://issues.apache.org/jira/browse/YARN-10599 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Peter Bacsko Assignee: Peter Bacsko -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10596) Allow static definition of childless ParentQueues with auto-queue-creation-v2 enabled
[ https://issues.apache.org/jira/browse/YARN-10596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272161#comment-17272161 ] Szilard Nemeth commented on YARN-10596: --- Hi [~gandras], Thanks for working on this. Latest patch looks good to me, committed to trunk. Thanks [~pbacsko] and [~bteke] for the reviews. > Allow static definition of childless ParentQueues with auto-queue-creation-v2 > enabled > - > > Key: YARN-10596 > URL: https://issues.apache.org/jira/browse/YARN-10596 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Andras Gyori >Assignee: Andras Gyori >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10596.001.patch, YARN-10596.002.patch > > > The old auto queue creation/managed queue logic allowed the definition of > childless parents to be created statically, if the auto-create-child-queue > flag was turned on the parent (thus making it a ManagedParentQueue). > Since it is not an edge case, we also need to support the creation of a > ParentQueue instead of a LeafQueue, if auto-queue-creation-v2 is enabled, > even when no child queue is defined under the parent. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10596) Allow static definition of childless ParentQueues with auto-queue-creation-v2 enabled
[ https://issues.apache.org/jira/browse/YARN-10596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-10596: -- Fix Version/s: 3.4.0 > Allow static definition of childless ParentQueues with auto-queue-creation-v2 > enabled > - > > Key: YARN-10596 > URL: https://issues.apache.org/jira/browse/YARN-10596 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Andras Gyori >Assignee: Andras Gyori >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10596.001.patch, YARN-10596.002.patch > > > The old auto queue creation/managed queue logic allowed the definition of > childless parents to be created statically, if the auto-create-child-queue > flag was turned on the parent (thus making it a ManagedParentQueue). > Since it is not an edge case, we also need to support the creation of a > ParentQueue instead of a LeafQueue, if auto-queue-creation-v2 is enabled, > even when no child queue is defined under the parent. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10576) Update Capacity Scheduler documentation about JSON-based placement mapping
[ https://issues.apache.org/jira/browse/YARN-10576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-10576: Attachment: YARN-10576-001.patch > Update Capacity Scheduler documentation about JSON-based placement mapping > -- > > Key: YARN-10576 > URL: https://issues.apache.org/jira/browse/YARN-10576 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-10576-001.patch > > > The weight mode and AQC also affects how the new placement engine in CS works > and the documentation has to reflect that. > Certain statements in the documentation are no longer valid, for example: > * create flag: "Only applies to managed queue parents" - there is no > ManagedParentQueue in weight mode. > * "The nested rules primaryGroupUser and secondaryGroupUser expects the > parent queues to exist, ie. they cannot be created automatically". This only > applies to the legacy absolute/percentage mode. > Find all statements that mentions possible limitations and fix them if > necessary. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10598) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information
[ https://issues.apache.org/jira/browse/YARN-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke updated YARN-10598: - Description: Under this umbrella (YARN-10496), weight-mode has been implemented for CS with YARN-10504. Auto-queue creation has been also implemented with YARN-10506. Connected to this effort, we would like to expose the type of the queue with the RM's /scheduler REST endpoint. To extend/modify the values added in YARN-10581 these 3 fields will describe a queue: * queueType : *parent/leaf* * creationMethod : *static/dynamicLegacy/dynamicFlexible* * autoCreationEligibility : *off/legacy/flexible* After this change here are some example cases: * Static parent queue which has the auto-creation-enabled-v2 false: ** queueType : *parent* ** creationMethod : *static* ** autoCreationEligibility : *off* * Static managed parent (can have dynamic children): ** queueType : *parent* ** creationMethod : *static* ** autoCreationEligibility : *legacy* * Legacy auto-created leaf queue (cannot have children): ** queueType : *leaf* ** creationMethod : *dynamicLegacy* ** autoCreationEligibility : *off* * Auto-created (v2) parent queue, (implicitly) auto-creation-enabled-v2 true: ** queueType : *parent* ** creationMethod : *dynamicFlexible* ** autoCreationEligibility : *flexible* * Auto-created (v2) leaf queue (cannot have children): ** queueType : *leaf* ** creationMethod : *dynamicFlexible* ** autoCreationEligibility : *off* was: Under this umbrella (YARN-10496), weight-mode has been implemented for CS with YARN-10504. Auto-queue creation has been also implemented with YARN-10506. Connected to this effort, we would like to expose the type of the queue with the RM's /scheduler REST endpoint. To extend/modify the values added in YARN-10581 these 3 fields will describe a queue: * queueType : *parent/leaf* * creationMethod : *static/dynamicLegacy/dynamicFlexible* * autoCreationEligibility : *off/legacy/flexible* After this change here are some example cases: * Static parent queue which has the auto-creation-enabled-v2 false: ** queueType : *parent* ** creationMethod : *static* ** autoCreationEligibility : *off* * Static managed parent (can have dynamic children): ** queueType : *parent* ** creationMethod : *static* ** autoCreationEligibility : *legacy* * Legacy auto-created leaf queue (cannot have children): ** queueType : *leaf* ** creationMethod : *dynamicLegacy* ** autoCreationEligibility : *off* * Auto-created (v2) parent queue, auto-creation-enabled-v2 true: ** queueType : *parent* ** creationMethod : *dynamicFlexible* ** autoCreationEligibility : *flexible* * Auto-created (v2) leaf queue (cannot have children): ** queueType : *leaf* ** creationMethod : *dynamicFlexible* ** autoCreationEligibility : *off* > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the > creation type with additional information > -- > > Key: YARN-10598 > URL: https://issues.apache.org/jira/browse/YARN-10598 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > Auto-queue creation has been also implemented with YARN-10506. > Connected to this effort, we would like to expose the type of the queue with > the RM's /scheduler REST endpoint. > To extend/modify the values added in YARN-10581 these 3 fields will describe > a queue: > * queueType : *parent/leaf* > * creationMethod : *static/dynamicLegacy/dynamicFlexible* > * autoCreationEligibility : *off/legacy/flexible* > After this change here are some example cases: > * Static parent queue which has the auto-creation-enabled-v2 false: > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *off* > * Static managed parent (can have dynamic children): > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *legacy* > * Legacy auto-created leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicLegacy* > ** autoCreationEligibility : *off* > * Auto-created (v2) parent queue, (implicitly) auto-creation-enabled-v2 > true: > ** queueType : *parent* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *flexible* > * Auto-created (v2) leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *off* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail:
[jira] [Updated] (YARN-10598) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information
[ https://issues.apache.org/jira/browse/YARN-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke updated YARN-10598: - Description: Under this umbrella (YARN-10496), weight-mode has been implemented for CS with YARN-10504. Auto-queue creation has been also implemented with YARN-10506. Connected to this effort, we would like to expose the type of the queue with the RM's /scheduler REST endpoint. To extend/modify the values added in YARN-10581 these 3 fields will describe a queue: * queueType : *parent/leaf* * creationMethod : *static/dynamicLegacy/dynamicFlexible* * autoCreationEligibility : *off/legacy/flexible* After this change here are some example cases: * Static parent queue which has the auto-creation-enabled-v2 false: ** queueType : *parent* ** creationMethod : *static* ** autoCreationEligibility : *off* * **Static managed parent (can have dynamic children): ** queueType : *parent* ** creationMethod : *static* ** autoCreationEligibility : *legacy* * Legacy auto-created leaf queue (cannot have children): ** queueType : *leaf* ** creationMethod : *dynamicLegacy* ** autoCreationEligibility : *off* * Auto-created (v2) parent queue, auto-creation-enabled-v2 true: ** queueType : *parent* ** creationMethod : *dynamicFlexible* ** autoCreationEligibility : *flexible* * Auto-created (v2) leaf queue (cannot have children): ** queueType : *leaf* ** creationMethod : *dynamicFlexible* ** autoCreationEligibility : *off* was: Under this umbrella (YARN-10496), weight-mode has been implemented for CS with YARN-10504. Auto-queue creation has been also imlemented with YARN-10506. Connected to this effort, we would like to expose the type of the queue with the RM's /scheduler REST endpoint. The queue type should hold these values: * Auto-created parent queue: *autoCreatedParent* * Auto-created leaf queue: *autoCreatedLeaf* * Static parent: *staticParent* * Static leaf: *staticLeaf* > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the > creation type with additional information > -- > > Key: YARN-10598 > URL: https://issues.apache.org/jira/browse/YARN-10598 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > Auto-queue creation has been also implemented with YARN-10506. > Connected to this effort, we would like to expose the type of the queue with > the RM's /scheduler REST endpoint. > To extend/modify the values added in YARN-10581 these 3 fields will describe > a queue: > * queueType : *parent/leaf* > * creationMethod : *static/dynamicLegacy/dynamicFlexible* > * autoCreationEligibility : *off/legacy/flexible* > After this change here are some example cases: > * Static parent queue which has the auto-creation-enabled-v2 false: > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *off* > * **Static managed parent (can have dynamic children): > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *legacy* > * Legacy auto-created leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicLegacy* > ** autoCreationEligibility : *off* > * Auto-created (v2) parent queue, auto-creation-enabled-v2 true: > ** queueType : *parent* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *flexible* > * Auto-created (v2) leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *off* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10598) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information
[ https://issues.apache.org/jira/browse/YARN-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke updated YARN-10598: - Description: Under this umbrella (YARN-10496), weight-mode has been implemented for CS with YARN-10504. Auto-queue creation has been also implemented with YARN-10506. Connected to this effort, we would like to expose the type of the queue with the RM's /scheduler REST endpoint. To extend/modify the values added in YARN-10581 these 3 fields will describe a queue: * queueType : *parent/leaf* * creationMethod : *static/dynamicLegacy/dynamicFlexible* * autoCreationEligibility : *off/legacy/flexible* After this change here are some example cases: * Static parent queue which has the auto-creation-enabled-v2 false: ** queueType : *parent* ** creationMethod : *static* ** autoCreationEligibility : *off* * Static managed parent (can have dynamic children): ** queueType : *parent* ** creationMethod : *static* ** autoCreationEligibility : *legacy* * Legacy auto-created leaf queue (cannot have children): ** queueType : *leaf* ** creationMethod : *dynamicLegacy* ** autoCreationEligibility : *off* * Auto-created (v2) parent queue, auto-creation-enabled-v2 true: ** queueType : *parent* ** creationMethod : *dynamicFlexible* ** autoCreationEligibility : *flexible* * Auto-created (v2) leaf queue (cannot have children): ** queueType : *leaf* ** creationMethod : *dynamicFlexible* ** autoCreationEligibility : *off* was: Under this umbrella (YARN-10496), weight-mode has been implemented for CS with YARN-10504. Auto-queue creation has been also implemented with YARN-10506. Connected to this effort, we would like to expose the type of the queue with the RM's /scheduler REST endpoint. To extend/modify the values added in YARN-10581 these 3 fields will describe a queue: * queueType : *parent/leaf* * creationMethod : *static/dynamicLegacy/dynamicFlexible* * autoCreationEligibility : *off/legacy/flexible* After this change here are some example cases: * Static parent queue which has the auto-creation-enabled-v2 false: ** queueType : *parent* ** creationMethod : *static* ** autoCreationEligibility : *off* * **Static managed parent (can have dynamic children): ** queueType : *parent* ** creationMethod : *static* ** autoCreationEligibility : *legacy* * Legacy auto-created leaf queue (cannot have children): ** queueType : *leaf* ** creationMethod : *dynamicLegacy* ** autoCreationEligibility : *off* * Auto-created (v2) parent queue, auto-creation-enabled-v2 true: ** queueType : *parent* ** creationMethod : *dynamicFlexible* ** autoCreationEligibility : *flexible* * Auto-created (v2) leaf queue (cannot have children): ** queueType : *leaf* ** creationMethod : *dynamicFlexible* ** autoCreationEligibility : *off* > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the > creation type with additional information > -- > > Key: YARN-10598 > URL: https://issues.apache.org/jira/browse/YARN-10598 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > Auto-queue creation has been also implemented with YARN-10506. > Connected to this effort, we would like to expose the type of the queue with > the RM's /scheduler REST endpoint. > To extend/modify the values added in YARN-10581 these 3 fields will describe > a queue: > * queueType : *parent/leaf* > * creationMethod : *static/dynamicLegacy/dynamicFlexible* > * autoCreationEligibility : *off/legacy/flexible* > After this change here are some example cases: > * Static parent queue which has the auto-creation-enabled-v2 false: > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *off* > * Static managed parent (can have dynamic children): > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *legacy* > * Legacy auto-created leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicLegacy* > ** autoCreationEligibility : *off* > * Auto-created (v2) parent queue, auto-creation-enabled-v2 true: > ** queueType : *parent* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *flexible* > * Auto-created (v2) leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *off* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail:
[jira] [Commented] (YARN-10596) Allow static definition of childless ParentQueues with auto-queue-creation-v2 enabled
[ https://issues.apache.org/jira/browse/YARN-10596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272123#comment-17272123 ] Andras Gyori commented on YARN-10596: - Thank you [~bteke] and [~pbacsko] for the review. Fixed the test and checkstyle errors, also incorporated [~bteke]'s suggestion. We have decided to defer the introduction of a global property until we have enough feedback that it is indeed a popular request among users. > Allow static definition of childless ParentQueues with auto-queue-creation-v2 > enabled > - > > Key: YARN-10596 > URL: https://issues.apache.org/jira/browse/YARN-10596 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Andras Gyori >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10596.001.patch, YARN-10596.002.patch > > > The old auto queue creation/managed queue logic allowed the definition of > childless parents to be created statically, if the auto-create-child-queue > flag was turned on the parent (thus making it a ManagedParentQueue). > Since it is not an edge case, we also need to support the creation of a > ParentQueue instead of a LeafQueue, if auto-queue-creation-v2 is enabled, > even when no child queue is defined under the parent. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10596) Allow static definition of childless ParentQueues with auto-queue-creation-v2 enabled
[ https://issues.apache.org/jira/browse/YARN-10596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Gyori updated YARN-10596: Attachment: YARN-10596.002.patch > Allow static definition of childless ParentQueues with auto-queue-creation-v2 > enabled > - > > Key: YARN-10596 > URL: https://issues.apache.org/jira/browse/YARN-10596 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Andras Gyori >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10596.001.patch, YARN-10596.002.patch > > > The old auto queue creation/managed queue logic allowed the definition of > childless parents to be created statically, if the auto-create-child-queue > flag was turned on the parent (thus making it a ManagedParentQueue). > Since it is not an edge case, we also need to support the creation of a > ParentQueue instead of a LeafQueue, if auto-queue-creation-v2 is enabled, > even when no child queue is defined under the parent. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10598) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information
[ https://issues.apache.org/jira/browse/YARN-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke reassigned YARN-10598: Assignee: Benjamin Teke (was: Szilard Nemeth) > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the > creation type with additional information > -- > > Key: YARN-10598 > URL: https://issues.apache.org/jira/browse/YARN-10598 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > Auto-queue creation has been also imlemented with YARN-10506. > Connected to this effort, we would like to expose the type of the queue with > the RM's /scheduler REST endpoint. > The queue type should hold these values: > * Auto-created parent queue: *autoCreatedParent* > * Auto-created leaf queue: *autoCreatedLeaf* > * Static parent: *staticParent* > * Static leaf: *staticLeaf* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10598) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information
Benjamin Teke created YARN-10598: Summary: CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information Key: YARN-10598 URL: https://issues.apache.org/jira/browse/YARN-10598 Project: Hadoop YARN Issue Type: Sub-task Reporter: Benjamin Teke Assignee: Szilard Nemeth Under this umbrella (YARN-10496), weight-mode has been implemented for CS with YARN-10504. Auto-queue creation has been also imlemented with YARN-10506. Connected to this effort, we would like to expose the type of the queue with the RM's /scheduler REST endpoint. The queue type should hold these values: * Auto-created parent queue: *autoCreatedParent* * Auto-created leaf queue: *autoCreatedLeaf* * Static parent: *staticParent* * Static leaf: *staticLeaf* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10564) Support Auto Queue Creation template configurations
[
https://issues.apache.org/jira/browse/YARN-10564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272094#comment-17272094
]
Hadoop QA commented on YARN-10564:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m
5s{color} | {color:red}{color} | {color:red} Docker failed to build
yetus/hadoop:b441ca86995. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-10564 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13019415/YARN-10564.002.patch |
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/546/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
> Support Auto Queue Creation template configurations
> ---
>
> Key: YARN-10564
> URL: https://issues.apache.org/jira/browse/YARN-10564
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10564.001.patch, YARN-10564.002.patch,
> YARN-10564.poc.001.patch
>
>
> Similar to how the template configuration works for ManagedParents, we need
> to support templates for the new auto queue creation logic. Proposition is to
> allow wildcards in template configs such as:
> {noformat}
> yarn.scheduler.capacity.root.*.*.weight 10{noformat}
> which would mean, that set weight to 10 of every leaf of every parent under
> root.
> We should possibly take an approach, that could support arbitrary depth of
> template configuration, because we might need to lift the limitation of auto
> queue nesting.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10564) Support Auto Queue Creation template configurations
[
https://issues.apache.org/jira/browse/YARN-10564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Gyori updated YARN-10564:
Attachment: YARN-10564.002.patch
> Support Auto Queue Creation template configurations
> ---
>
> Key: YARN-10564
> URL: https://issues.apache.org/jira/browse/YARN-10564
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10564.001.patch, YARN-10564.002.patch,
> YARN-10564.poc.001.patch
>
>
> Similar to how the template configuration works for ManagedParents, we need
> to support templates for the new auto queue creation logic. Proposition is to
> allow wildcards in template configs such as:
> {noformat}
> yarn.scheduler.capacity.root.*.*.weight 10{noformat}
> which would mean, that set weight to 10 of every leaf of every parent under
> root.
> We should possibly take an approach, that could support arbitrary depth of
> template configuration, because we might need to lift the limitation of auto
> queue nesting.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10585) Create a class which can convert from legacy mapping rule format to the new JSON format
[
https://issues.apache.org/jira/browse/YARN-10585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272089#comment-17272089
]
Hadoop QA commented on YARN-10585:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m
5s{color} | {color:red}{color} | {color:red} Docker failed to build
yetus/hadoop:b441ca86995. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-10585 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13019410/YARN-10585.003.patch |
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/545/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
> Create a class which can convert from legacy mapping rule format to the new
> JSON format
> ---
>
> Key: YARN-10585
> URL: https://issues.apache.org/jira/browse/YARN-10585
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Gergely Pollak
>Assignee: Gergely Pollak
>Priority: Major
> Attachments: YARN-10585.001.patch, YARN-10585.002.patch,
> YARN-10585.003.patch
>
>
> To make transition easier we need to create tooling to support the migration
> effort. The first step is to create a class which can migrate from legacy to
> the new JSON format.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10428) Zombie applications in the YARN queue using FAIR + sizebasedweight
[ https://issues.apache.org/jira/browse/YARN-10428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272081#comment-17272081 ] Andras Gyori commented on YARN-10428: - [~yguang11] if you do not mind, I will take it over, as we are in need of this fix. > Zombie applications in the YARN queue using FAIR + sizebasedweight > -- > > Key: YARN-10428 > URL: https://issues.apache.org/jira/browse/YARN-10428 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Affects Versions: 2.8.5 >Reporter: Guang Yang >Priority: Major > Attachments: YARN-10428.001.patch, YARN-10428.002.patch > > > Seeing zombie jobs in the YARN queue that uses FAIR and size based weight > ordering policy . > *Detection:* > The YARN UI shows incorrect number of "Num Schedulable Applications". > *Impact:* > The queue has an upper limit of number of running applications, with zombie > job, it hits the limit even though the number of running applications is far > less than the limit. > *Workaround:* > **Fail-over and restart Resource Manager process. > *Analysis:* > **In the heap dump, we can find the zombie jobs in the `FairOderingPolicy# > schedulableEntities` (see attachment). Take application > "application_1599157165858_29429" for example, it is still in the > `FairOderingPolicy#schedulableEntities` set, however, if we check the log of > resource manager, we can see RM already tried to remove the application: > > ./yarn-yarn-resourcemanager-ip-172-21-153-252.log.2020-09-04-04:2020-09-04 > 04:32:19,730 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue > (ResourceManager Event Processor): Application removed - appId: > application_1599157165858_29429 user: svc_di_data_eng queue: core-data > #user-pending-applications: -3 #user-active-applications: 7 > #queue-pending-applications: 0 #queue-active-applications: 21 > > So it appears RM failed to removed the application from the set. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10585) Create a class which can convert from legacy mapping rule format to the new JSON format
[ https://issues.apache.org/jira/browse/YARN-10585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272072#comment-17272072 ] Gergely Pollak commented on YARN-10585: --- Latest patch fixed [~gandras] "userGroupMappingRules and applicationNameMappingRules might be initialised as empty sets" concern, now the null check is centralised and moved to the setters. Also fixed ASF warnings, and added a testcase for the "u:%user:root.%user.QUEUE" case. > Create a class which can convert from legacy mapping rule format to the new > JSON format > --- > > Key: YARN-10585 > URL: https://issues.apache.org/jira/browse/YARN-10585 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-10585.001.patch, YARN-10585.002.patch, > YARN-10585.003.patch > > > To make transition easier we need to create tooling to support the migration > effort. The first step is to create a class which can migrate from legacy to > the new JSON format. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10585) Create a class which can convert from legacy mapping rule format to the new JSON format
[
https://issues.apache.org/jira/browse/YARN-10585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272073#comment-17272073
]
Hadoop QA commented on YARN-10585:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m
6s{color} | {color:red}{color} | {color:red} Docker failed to build
yetus/hadoop:b441ca86995. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-10585 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13019410/YARN-10585.003.patch |
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/544/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
> Create a class which can convert from legacy mapping rule format to the new
> JSON format
> ---
>
> Key: YARN-10585
> URL: https://issues.apache.org/jira/browse/YARN-10585
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Gergely Pollak
>Assignee: Gergely Pollak
>Priority: Major
> Attachments: YARN-10585.001.patch, YARN-10585.002.patch,
> YARN-10585.003.patch
>
>
> To make transition easier we need to create tooling to support the migration
> effort. The first step is to create a class which can migrate from legacy to
> the new JSON format.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10585) Create a class which can convert from legacy mapping rule format to the new JSON format
[ https://issues.apache.org/jira/browse/YARN-10585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergely Pollak updated YARN-10585: -- Attachment: YARN-10585.003.patch > Create a class which can convert from legacy mapping rule format to the new > JSON format > --- > > Key: YARN-10585 > URL: https://issues.apache.org/jira/browse/YARN-10585 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-10585.001.patch, YARN-10585.002.patch, > YARN-10585.003.patch > > > To make transition easier we need to create tooling to support the migration > effort. The first step is to create a class which can migrate from legacy to > the new JSON format. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10592) Support QueueCapacities to use vector based multi resource types, and update absolute related to use first.
[
https://issues.apache.org/jira/browse/YARN-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272001#comment-17272001
]
Hadoop QA commented on YARN-10592:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 16m
59s{color} | {color:red}{color} | {color:red} Docker failed to build
yetus/hadoop:b441ca86995. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-10592 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13019407/YARN-10592.002.patch |
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/543/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
> Support QueueCapacities to use vector based multi resource types, and update
> absolute related to use first.
> ---
>
> Key: YARN-10592
> URL: https://issues.apache.org/jira/browse/YARN-10592
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: zhuqi
>Assignee: zhuqi
>Priority: Major
> Attachments: YARN-10592.001.patch, YARN-10592.002.patch
>
>
> Because we need to support YARN-10524 and related other multi type support.
> Support QueueCapacities to use vector.
> Also this patch has to update absolute related to use first, to make sure no
> loss of multi type resource fixed by YARN-10590.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10592) Support QueueCapacities to use vector based multi resource types, and update absolute related to use first.
[ https://issues.apache.org/jira/browse/YARN-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271183#comment-17271183 ] zhuqi edited comment on YARN-10592 at 1/26/21, 9:20 AM: cc [~wangda] [~sunilg] [~epayne] [~bteke] [~gandras] Any thoughts about this, we use vector update in QueueCapacities for future need in YARN-10524. And the patch has checked the absolute related, which can fix YARN-10590. And i think the multi resource type based cap/abs/weight , and related change is huge, if we should move this related to a new issue? was (Author: zhuqi): cc [~wangda] [~sunilg] [~epayne] [~bteke] [~gandras] Any thoughts about this, we use vector update in QueueCapacities for future need in YARN-10524. And the patch has checked the absolute related, which can fix YARN-10590. > Support QueueCapacities to use vector based multi resource types, and update > absolute related to use first. > --- > > Key: YARN-10592 > URL: https://issues.apache.org/jira/browse/YARN-10592 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > Attachments: YARN-10592.001.patch, YARN-10592.002.patch > > > Because we need to support YARN-10524 and related other multi type support. > Support QueueCapacities to use vector. > Also this patch has to update absolute related to use first, to make sure no > loss of multi type resource fixed by YARN-10590. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10592) Support QueueCapacities to use vector based multi resource types, and update absolute related to use first.
[ https://issues.apache.org/jira/browse/YARN-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuqi updated YARN-10592: - Attachment: YARN-10592.002.patch > Support QueueCapacities to use vector based multi resource types, and update > absolute related to use first. > --- > > Key: YARN-10592 > URL: https://issues.apache.org/jira/browse/YARN-10592 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > Attachments: YARN-10592.001.patch, YARN-10592.002.patch > > > Because we need to support YARN-10524 and related other multi type support. > Support QueueCapacities to use vector. > Also this patch has to update absolute related to use first, to make sure no > loss of multi type resource fixed by YARN-10590. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
