[jira] [Created] (YARN-10580) Fix some issues in TestRMWebServicesCapacitySchedDynamicConfig
Szilard Nemeth created YARN-10580:
-
Summary: Fix some issues in
TestRMWebServicesCapacitySchedDynamicConfig
Key: YARN-10580
URL: https://issues.apache.org/jira/browse/YARN-10580
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Szilard Nemeth
YARN-10512 introduced some changes that could be fixed, [~pbacsko] highlighted
those issues with this comment.
Pasting the contents of the comment as a reference
#1 In TestRMWebServicesCapacitySchedDynamicConfig:
{noformat}
config.set(YarnConfiguration.SCHEDULER_CONFIGURATION_STORE_CLASS,
YarnConfiguration.MEMORY_CONFIGURATION_STORE);
{noformat}
This call is repeated multiple times, this could be set somewhere else.
#2 In TestRMWebServicesCapacitySchedDynamicConfig:
{noformat}
validateSchedulerInfo(json, "weight", "root.default", "root.test1",
"root.test2");
{noformat}
"root.default", "root.test1" and "root.test2" are the same in all cases, you
might want to drop them
#3 In TestRMWebServicesCapacitySchedDynamicConfig
{noformat}
@Before
@Override
public void setUp() throws Exception {
super.setUp();
}
{noformat}
This method does nothing, can be removed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10512) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include mode of operation for CS
[ https://issues.apache.org/jira/browse/YARN-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17268454#comment-17268454 ] Szilard Nemeth commented on YARN-10512: --- Thanks a lot [~sunilg] for the commit and thanks [~pbacsko] for the review. Filed YARN-10580 as a follow-up. > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include > mode of operation for CS > -- > > Key: YARN-10512 > URL: https://issues.apache.org/jira/browse/YARN-10512 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Szilard Nemeth >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10512.001.patch, YARN-10512.002.patch, > YARN-10512.003.patch, YARN-10512.004.patch > > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > We would like to expose the mode of operation with the RM's /scheduler REST > endpoint. > The field name will be 'mode'. > All queue representations in the response will be uniformly hold any of the > mode values of: "percentage", "absolute", "weight". -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10580) Fix some issues in TestRMWebServicesCapacitySchedDynamicConfig
[
https://issues.apache.org/jira/browse/YARN-10580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szilard Nemeth updated YARN-10580:
--
Labels: newbie newbie++ (was: )
> Fix some issues in TestRMWebServicesCapacitySchedDynamicConfig
> --
>
> Key: YARN-10580
> URL: https://issues.apache.org/jira/browse/YARN-10580
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Szilard Nemeth
>Priority: Minor
> Labels: newbie, newbie++
>
> YARN-10512 introduced some changes that could be fixed, [~pbacsko]
> highlighted those issues with this comment.
> Pasting the contents of the comment as a reference
> #1 In TestRMWebServicesCapacitySchedDynamicConfig:
> {noformat}
> config.set(YarnConfiguration.SCHEDULER_CONFIGURATION_STORE_CLASS,
> YarnConfiguration.MEMORY_CONFIGURATION_STORE);
> {noformat}
> This call is repeated multiple times, this could be set somewhere else.
>
> #2 In TestRMWebServicesCapacitySchedDynamicConfig:
> {noformat}
> validateSchedulerInfo(json, "weight", "root.default", "root.test1",
> "root.test2");
> {noformat}
> "root.default", "root.test1" and "root.test2" are the same in all cases, you
> might want to drop them
>
> #3 In TestRMWebServicesCapacitySchedDynamicConfig
> {noformat}
> @Before
> @Override
> public void setUp() throws Exception {
> super.setUp();
> }
> {noformat}
> This method does nothing, can be removed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10581) CLONE - CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include weight values for queues
[ https://issues.apache.org/jira/browse/YARN-10581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-10581: -- Description: Under this umbrella (YARN-10496), weight-mode has been implemented for CS with YARN-10504. We would like to expose the type of the queue with the RM's /scheduler REST endpoint. was: Under this umbrella (YARN-10496), weight-mode has been implemented for CS with YARN-10504. We would like to expose the weight values for all queues with the RM's /scheduler REST endpoint. > CLONE - CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to > include weight values for queues > -- > > Key: YARN-10581 > URL: https://issues.apache.org/jira/browse/YARN-10581 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > We would like to expose the type of the queue with the RM's /scheduler REST > endpoint. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10581) CLONE - CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include weight values for queues
Szilard Nemeth created YARN-10581: - Summary: CLONE - CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include weight values for queues Key: YARN-10581 URL: https://issues.apache.org/jira/browse/YARN-10581 Project: Hadoop YARN Issue Type: Sub-task Reporter: Szilard Nemeth Assignee: Szilard Nemeth Under this umbrella (YARN-10496), weight-mode has been implemented for CS with YARN-10504. We would like to expose the weight values for all queues with the RM's /scheduler REST endpoint. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10581) CLONE - CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include weight values for queues
[ https://issues.apache.org/jira/browse/YARN-10581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-10581: -- Description: Under this umbrella (YARN-10496), weight-mode has been implemented for CS with YARN-10504. Auto-queue creation has been also imlemented with YARN-10506. Connected to this effort, we would like to expose the type of the queue with the RM's /scheduler REST endpoint. was: Under this umbrella (YARN-10496), weight-mode has been implemented for CS with YARN-10504. We would like to expose the type of the queue with the RM's /scheduler REST endpoint. > CLONE - CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to > include weight values for queues > -- > > Key: YARN-10581 > URL: https://issues.apache.org/jira/browse/YARN-10581 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > Auto-queue creation has been also imlemented with YARN-10506. > Connected to this effort, we would like to expose the type of the queue with > the RM's /scheduler REST endpoint. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10581) CLONE - CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include weight values for queues
[ https://issues.apache.org/jira/browse/YARN-10581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-10581: -- Description: Under this umbrella (YARN-10496), weight-mode has been implemented for CS with YARN-10504. Auto-queue creation has been also imlemented with YARN-10506. Connected to this effort, we would like to expose the type of the queue with the RM's /scheduler REST endpoint. The queue type should hold these values: * Auto-created parent queue: *autoCreatedParent* * Auto-created leaf queue: *autoCreatedLeaf* * Static parent: *staticParent* * Static leaf: *staticLeaf* was: Under this umbrella (YARN-10496), weight-mode has been implemented for CS with YARN-10504. Auto-queue creation has been also imlemented with YARN-10506. Connected to this effort, we would like to expose the type of the queue with the RM's /scheduler REST endpoint. > CLONE - CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to > include weight values for queues > -- > > Key: YARN-10581 > URL: https://issues.apache.org/jira/browse/YARN-10581 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > Auto-queue creation has been also imlemented with YARN-10506. > Connected to this effort, we would like to expose the type of the queue with > the RM's /scheduler REST endpoint. > The queue type should hold these values: > * Auto-created parent queue: *autoCreatedParent* > * Auto-created leaf queue: *autoCreatedLeaf* > * Static parent: *staticParent* > * Static leaf: *staticLeaf* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10581) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include queue creation type for queues
[ https://issues.apache.org/jira/browse/YARN-10581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-10581: -- Summary: CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include queue creation type for queues (was: CLONE - CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include weight values for queues) > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include > queue creation type for queues > > > Key: YARN-10581 > URL: https://issues.apache.org/jira/browse/YARN-10581 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > Auto-queue creation has been also imlemented with YARN-10506. > Connected to this effort, we would like to expose the type of the queue with > the RM's /scheduler REST endpoint. > The queue type should hold these values: > * Auto-created parent queue: *autoCreatedParent* > * Auto-created leaf queue: *autoCreatedLeaf* > * Static parent: *staticParent* > * Static leaf: *staticLeaf* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10531) Be able to disable user limit factor for CapacityScheduler Leaf Queue
[
https://issues.apache.org/jira/browse/YARN-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17268457#comment-17268457
]
zhuqi commented on YARN-10531:
--
cc [~wangda] [~sunilg]
If you any other advice for this?
Thanks.
> Be able to disable user limit factor for CapacityScheduler Leaf Queue
> -
>
> Key: YARN-10531
> URL: https://issues.apache.org/jira/browse/YARN-10531
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: zhuqi
>Priority: Major
> Attachments: YARN-10531.001.patch, YARN-10531.002.patch,
> YARN-10531.003.patch, YARN-10531.004.patch, YARN-10531.005.patch
>
>
> User limit factor is used to define max cap of how much resource can be
> consumed by single user.
> Under Auto Queue Creation context, it doesn't make much sense to set user
> limit factor, because initially every queue will set weight to 1.0, we want
> user can consume more resource if possible. It is hard to pre-determine how
> to set up user limit factor. So it makes more sense to add a new value (like
> -1) to indicate we will disable user limit factor
> Logic need to be changed is below:
> (Inside LeafQueue.java)
> {code}
> Resource maxUserLimit = Resources.none();
> if (schedulingMode == SchedulingMode.RESPECT_PARTITION_EXCLUSIVITY) {
> maxUserLimit = Resources.multiplyAndRoundDown(queueCapacity,
> getUserLimitFactor());
> } else if (schedulingMode == SchedulingMode.IGNORE_PARTITION_EXCLUSIVITY)
> {
> maxUserLimit = partitionResource;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10580) Fix some issues in TestRMWebServicesCapacitySchedDynamicConfig
[
https://issues.apache.org/jira/browse/YARN-10580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ankit Kumar reassigned YARN-10580:
--
Assignee: Ankit Kumar
> Fix some issues in TestRMWebServicesCapacitySchedDynamicConfig
> --
>
> Key: YARN-10580
> URL: https://issues.apache.org/jira/browse/YARN-10580
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Ankit Kumar
>Priority: Minor
> Labels: newbie, newbie++
>
> YARN-10512 introduced some changes that could be fixed, [~pbacsko]
> highlighted those issues with this comment.
> Pasting the contents of the comment as a reference
> #1 In TestRMWebServicesCapacitySchedDynamicConfig:
> {noformat}
> config.set(YarnConfiguration.SCHEDULER_CONFIGURATION_STORE_CLASS,
> YarnConfiguration.MEMORY_CONFIGURATION_STORE);
> {noformat}
> This call is repeated multiple times, this could be set somewhere else.
>
> #2 In TestRMWebServicesCapacitySchedDynamicConfig:
> {noformat}
> validateSchedulerInfo(json, "weight", "root.default", "root.test1",
> "root.test2");
> {noformat}
> "root.default", "root.test1" and "root.test2" are the same in all cases, you
> might want to drop them
>
> #3 In TestRMWebServicesCapacitySchedDynamicConfig
> {noformat}
> @Before
> @Override
> public void setUp() throws Exception {
> super.setUp();
> }
> {noformat}
> This method does nothing, can be removed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10352) Skip schedule on not heartbeated nodes in Multi Node Placement
[
https://issues.apache.org/jira/browse/YARN-10352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17268514#comment-17268514
]
Hadoop QA commented on YARN-10352:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
8s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
1s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 4
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
20s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 28m
26s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
47s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
58s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
41s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
12s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
23m 39s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
44s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
40s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m
6s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config;
considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
19s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
12s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
1s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m
1s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
27s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
27s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 47s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/520/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt{color}
| {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 4 new +
355 unchanged - 0 fixed = 359 total (was 355) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
21s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespa
[jira] [Created] (YARN-10582) Increase the test coverage for CSMappingPlacementRule
Gergely Pollak created YARN-10582: - Summary: Increase the test coverage for CSMappingPlacementRule Key: YARN-10582 URL: https://issues.apache.org/jira/browse/YARN-10582 Project: Hadoop YARN Issue Type: Sub-task Reporter: Gergely Pollak Assignee: Gergely Pollak We have somewhat limited coverage for cases introduced via YARN-10506, to make sure we won't have any regression we need to improve the coverage. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10583) Increase the test coverage for validation in MappingRuleValidationHelper
Gergely Pollak created YARN-10583: - Summary: Increase the test coverage for validation in MappingRuleValidationHelper Key: YARN-10583 URL: https://issues.apache.org/jira/browse/YARN-10583 Project: Hadoop YARN Issue Type: Sub-task Reporter: Gergely Pollak MappingRuleValidationHelper introduced new validation cases due to the changes in YARN-10506, we need to increase the coverage before we move the class into QueueManager -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10583) Increase the test coverage for validation in MappingRuleValidationHelper
[ https://issues.apache.org/jira/browse/YARN-10583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergely Pollak reassigned YARN-10583: - Assignee: Gergely Pollak > Increase the test coverage for validation in MappingRuleValidationHelper > > > Key: YARN-10583 > URL: https://issues.apache.org/jira/browse/YARN-10583 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > > MappingRuleValidationHelper introduced new validation cases due to the > changes in YARN-10506, we need to increase the coverage before we move the > class into QueueManager -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10584) Move the functionality of MappingRuleValidationHelper to CSQueueManager
Gergely Pollak created YARN-10584: - Summary: Move the functionality of MappingRuleValidationHelper to CSQueueManager Key: YARN-10584 URL: https://issues.apache.org/jira/browse/YARN-10584 Project: Hadoop YARN Issue Type: Sub-task Reporter: Gergely Pollak Assignee: Gergely Pollak The only reason we created the helper class was to make the features introduced in YARN-10506 accessible as soon as possible. Due to our current test helpers and frameworks, integrating these changes into the CapacitySchedulerQueueManager would have been more time consuming. However we need to remove this standalone class and merge the functionality into either the QueueManager or into CapacityScheduler. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10585) Create a class which can convert from legacy mapping rule format to the new JSON format
Gergely Pollak created YARN-10585: - Summary: Create a class which can convert from legacy mapping rule format to the new JSON format Key: YARN-10585 URL: https://issues.apache.org/jira/browse/YARN-10585 Project: Hadoop YARN Issue Type: Sub-task Reporter: Gergely Pollak Assignee: Gergely Pollak To make transition easier we need to create tooling to support the migration effort. The first step is to create a class which can migrate from legacy to the new JSON format. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10586) Create a command line tool for converting from legacy CS mapping rule format
Gergely Pollak created YARN-10586: - Summary: Create a command line tool for converting from legacy CS mapping rule format Key: YARN-10586 URL: https://issues.apache.org/jira/browse/YARN-10586 Project: Hadoop YARN Issue Type: Sub-task Reporter: Gergely Pollak Assignee: Gergely Pollak -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10565) Refactor CS queue initialization to simplify weight mode calculation
[
https://issues.apache.org/jira/browse/YARN-10565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benjamin Teke updated YARN-10565:
-
Description:
In YARN-10504 weight mode support was introduced to CS. This jira is a followup
to simplify and restructure the initialization, so that the weight
calculation/absolute/percentage mode is easier to understand and modify.
To be refactored:
* In ParentQueue.java#1099 the error message should be more specific, instead
of the {{LOG.error("Fatal issue found: e", e);}}
* AutoCreatedLeafQueue.clearConfigurableFields should clear NORMALIZED_WEIGHT
just to be on the safe side
*
was:In YARN-10504 weight mode support was introduced to CS. This jira is a
followup to simplify and restructure the initialization, so that the weight
calculation/absolute/percentage mode is easier to understand and modify.
> Refactor CS queue initialization to simplify weight mode calculation
>
>
> Key: YARN-10565
> URL: https://issues.apache.org/jira/browse/YARN-10565
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10565.001.patch, YARN-10565.002.patch
>
>
> In YARN-10504 weight mode support was introduced to CS. This jira is a
> followup to simplify and restructure the initialization, so that the weight
> calculation/absolute/percentage mode is easier to understand and modify.
> To be refactored:
> * In ParentQueue.java#1099 the error message should be more specific, instead
> of the {{LOG.error("Fatal issue found: e", e);}}
> * AutoCreatedLeafQueue.clearConfigurableFields should clear NORMALIZED_WEIGHT
> just to be on the safe side
> *
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10565) Refactor CS queue initialization to simplify weight mode calculation
[
https://issues.apache.org/jira/browse/YARN-10565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benjamin Teke updated YARN-10565:
-
Description:
In YARN-10504 weight mode support was introduced to CS. This jira is a followup
to simplify and restructure the initialization, so that the weight
calculation/absolute/percentage mode is easier to understand and modify.
To be refactored:
* In ParentQueue.java#1099 the error message should be more specific, instead
of the {{LOG.error("Fatal issue found: e", e);}}
* AutoCreatedLeafQueue.clearConfigurableFields should clear NORMALIZED_WEIGHT
just to be on the safe side
* Uncomment the commented assertions in
TestCapacitySchedulerAutoCreatedQueueBase.validateEffectiveMinResource
* Check whether the assertion modification in TestRMWebServices is absolutely
necessary or could be hiding a bug.
* Same for TestRMWebServicesForCSWithPartitions.java
was:
In YARN-10504 weight mode support was introduced to CS. This jira is a followup
to simplify and restructure the initialization, so that the weight
calculation/absolute/percentage mode is easier to understand and modify.
To be refactored:
* In ParentQueue.java#1099 the error message should be more specific, instead
of the {{LOG.error("Fatal issue found: e", e);}}
* AutoCreatedLeafQueue.clearConfigurableFields should clear NORMALIZED_WEIGHT
just to be on the safe side
*
> Refactor CS queue initialization to simplify weight mode calculation
>
>
> Key: YARN-10565
> URL: https://issues.apache.org/jira/browse/YARN-10565
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10565.001.patch, YARN-10565.002.patch
>
>
> In YARN-10504 weight mode support was introduced to CS. This jira is a
> followup to simplify and restructure the initialization, so that the weight
> calculation/absolute/percentage mode is easier to understand and modify.
> To be refactored:
> * In ParentQueue.java#1099 the error message should be more specific, instead
> of the {{LOG.error("Fatal issue found: e", e);}}
> * AutoCreatedLeafQueue.clearConfigurableFields should clear NORMALIZED_WEIGHT
> just to be on the safe side
> * Uncomment the commented assertions in
> TestCapacitySchedulerAutoCreatedQueueBase.validateEffectiveMinResource
> * Check whether the assertion modification in TestRMWebServices is absolutely
> necessary or could be hiding a bug.
> * Same for TestRMWebServicesForCSWithPartitions.java
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10352) Skip schedule on not heartbeated nodes in Multi Node Placement
[ https://issues.apache.org/jira/browse/YARN-10352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuqi updated YARN-10352: - Attachment: YARN-10352-010.patch > Skip schedule on not heartbeated nodes in Multi Node Placement > -- > > Key: YARN-10352 > URL: https://issues.apache.org/jira/browse/YARN-10352 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.3.0, 3.4.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Labels: capacityscheduler, multi-node-placement > Attachments: YARN-10352-001.patch, YARN-10352-002.patch, > YARN-10352-003.patch, YARN-10352-004.patch, YARN-10352-005.patch, > YARN-10352-006.patch, YARN-10352-007.patch, YARN-10352-008.patch, > YARN-10352-010.patch, YARN-10352.009.patch > > > When Node Recovery is Enabled, Stopping a NM won't unregister to RM. So RM > Active Nodes will be still having those stopped nodes until NM Liveliness > Monitor Expires after configured timeout > (yarn.nm.liveness-monitor.expiry-interval-ms = 10 mins). During this 10mins, > Multi Node Placement assigns the containers on those nodes. They need to > exclude the nodes which has not heartbeated for configured heartbeat interval > (yarn.resourcemanager.nodemanagers.heartbeat-interval-ms=1000ms) similar to > Asynchronous Capacity Scheduler Threads. > (CapacityScheduler#shouldSkipNodeSchedule) > *Repro:* > 1. Enable Multi Node Placement > (yarn.scheduler.capacity.multi-node-placement-enabled) + Node Recovery > Enabled (yarn.node.recovery.enabled) > 2. Have only one NM running say worker0 > 3. Stop worker0 and start any other NM say worker1 > 4. Submit a sleep job. The containers will timeout as assigned to stopped NM > worker0. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10579) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include weight values for queues
[ https://issues.apache.org/jira/browse/YARN-10579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-10579: -- Attachment: YARN-10579.001.patch > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include > weight values for queues > -- > > Key: YARN-10579 > URL: https://issues.apache.org/jira/browse/YARN-10579 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-10579.001.patch > > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > We would like to expose the weight values for all queues with the RM's > /scheduler REST endpoint. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10578) Fix Auto Queue Creation parent handling
[ https://issues.apache.org/jira/browse/YARN-10578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-10578: -- Fix Version/s: 3.4.0 > Fix Auto Queue Creation parent handling > --- > > Key: YARN-10578 > URL: https://issues.apache.org/jira/browse/YARN-10578 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Andras Gyori >Assignee: Andras Gyori >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10578.001.patch, YARN-10578.002.patch > > > YARN-10506 introduced the new auto queue creation logic, however a parent == > null check in CapacityScheduler#autoCreateLeafQueue will prevent a two levels > queue to be created. We need to revert it back to the normal logic, also, we > should wrap the auto queue handling with a lock. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10578) Fix Auto Queue Creation parent handling
[ https://issues.apache.org/jira/browse/YARN-10578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17268602#comment-17268602 ] Szilard Nemeth commented on YARN-10578: --- Hi [~gandras], Thanks for the patch. Latest patch LGTM, committed to trunk. Thanks [~wangda], [~zhuqi] and [~shuzirra] for the reviews. > Fix Auto Queue Creation parent handling > --- > > Key: YARN-10578 > URL: https://issues.apache.org/jira/browse/YARN-10578 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Andras Gyori >Assignee: Andras Gyori >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10578.001.patch, YARN-10578.002.patch > > > YARN-10506 introduced the new auto queue creation logic, however a parent == > null check in CapacityScheduler#autoCreateLeafQueue will prevent a two levels > queue to be created. We need to revert it back to the normal logic, also, we > should wrap the auto queue handling with a lock. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10579) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include weight values for queues
[ https://issues.apache.org/jira/browse/YARN-10579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17268638#comment-17268638 ] Andras Gyori commented on YARN-10579: - I have no comments regarding this, +1 non-binding. > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include > weight values for queues > -- > > Key: YARN-10579 > URL: https://issues.apache.org/jira/browse/YARN-10579 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-10579.001.patch > > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > We would like to expose the weight values for all queues with the RM's > /scheduler REST endpoint. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10587) Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode.
zhuqi created YARN-10587: Summary: Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode. Key: YARN-10587 URL: https://issues.apache.org/jira/browse/YARN-10587 Project: Hadoop YARN Issue Type: Sub-task Reporter: zhuqi Assignee: zhuqi -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10588) Percentage of queue and cluster is zero in WebUI
Bilwa S T created YARN-10588:
Summary: Percentage of queue and cluster is zero in WebUI
Key: YARN-10588
URL: https://issues.apache.org/jira/browse/YARN-10588
Project: Hadoop YARN
Issue Type: Bug
Reporter: Bilwa S T
Assignee: Bilwa S T
Steps to reproduce:
Configure below property in resource-types.xml
{code:java}
yarn.resource-types
yarn.io/gpu
{code}
Submit a job
In UI you can see % Of Queue and % Of Cluster is zero for the submitted
application
This is because in SchedulerApplicationAttempt has below check for calculating
queueUsagePerc and clusterUsagePerc
{code:java}
if (!calc.isInvalidDivisor(cluster)) {
float queueCapacityPerc = queue.getQueueInfo(false, false)
.getCapacity();
queueUsagePerc = calc.divide(cluster, usedResourceClone,
Resources.multiply(cluster, queueCapacityPerc)) * 100;
if (Float.isNaN(queueUsagePerc) || Float.isInfinite(queueUsagePerc)) {
queueUsagePerc = 0.0f;
}
clusterUsagePerc =
calc.divide(cluster, usedResourceClone, cluster) * 100;
}
{code}
calc.isInvalidDivisor(cluster) always returns true as gpu resource is 0
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10587) Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode.
[ https://issues.apache.org/jira/browse/YARN-10587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuqi updated YARN-10587: - Description: When introduced YARN-10504. The logic related to auto created leaf queue changed. The test in testAutoCreateLeafQueueCreation failed, we should fix the Error. > Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode. > - > > Key: YARN-10587 > URL: https://issues.apache.org/jira/browse/YARN-10587 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > > When introduced YARN-10504. > The logic related to auto created leaf queue changed. > The test in testAutoCreateLeafQueueCreation failed, we should fix the Error. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10579) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include weight values for queues
[ https://issues.apache.org/jira/browse/YARN-10579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17268665#comment-17268665 ] Peter Bacsko commented on YARN-10579: - +1 pending Jenkins. > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include > weight values for queues > -- > > Key: YARN-10579 > URL: https://issues.apache.org/jira/browse/YARN-10579 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-10579.001.patch > > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > We would like to expose the weight values for all queues with the RM's > /scheduler REST endpoint. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10352) Skip schedule on not heartbeated nodes in Multi Node Placement
[
https://issues.apache.org/jira/browse/YARN-10352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17268667#comment-17268667
]
Hadoop QA commented on YARN-10352:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
8s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
1s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 4
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
46s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
49s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
29s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
55s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
29s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
3s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
24m 45s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
44s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
42s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m
22s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
15s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/521/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt{color}
| {color:red} hadoop-yarn-api in trunk failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
39s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/521/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt{color}
| {color:red} hadoop-yarn-common in trunk failed. {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
35s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
5s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
5s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
48s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
48s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
35s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}
[jira] [Commented] (YARN-10587) Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode.
[
https://issues.apache.org/jira/browse/YARN-10587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17268670#comment-17268670
]
zhuqi commented on YARN-10587:
--
cc [~leftnoteasy] [~bteke]
When i deep into the testAutoCreateLeafQueueCreation failed about :
{code:java}
// TODO: Wangda: I think this test case is not correct, Sunil could help look
// into details.
{code}
I find the root reason is , when absolute mode enabled in auto created leaf
queue, we finally will update cap related for the UI in :
{code:java}
private void deriveCapacityFromAbsoluteConfigurations(String label,
Resource clusterResource, ResourceCalculator rc) {
/*
* In case when queues are configured with absolute resources, it is better
* to update capacity/max-capacity etc w.r.t absolute resource as well. In
* case of computation, these values wont be used any more. However for
* metrics and UI, its better these values are pre-computed here itself.
*/
// 1. Update capacity as a float based on parent's minResource
float f = rc.divide(clusterResource,
queueResourceQuotas.getEffectiveMinResource(label),
parent.getQueueResourceQuotas().getEffectiveMinResource(label));
queueCapacities.setCapacity(label, Float.isInfinite(f) ? 0 : f);
// 2. Update max-capacity as a float based on parent's maxResource
f = rc.divide(clusterResource,
queueResourceQuotas.getEffectiveMaxResource(label),
parent.getQueueResourceQuotas().getEffectiveMaxResource(label));
queueCapacities.setMaximumCapacity(label, Float.isInfinite(f) ? 0 : f);
// 3. Update absolute capacity as a float based on parent's minResource and
// cluster resource.
queueCapacities.setAbsoluteCapacity(label,
queueCapacities.getCapacity(label) * parent.getQueueCapacities()
.getAbsoluteCapacity(label));
// 4. Update absolute max-capacity as a float based on parent's maxResource
// and cluster resource.
queueCapacities.setAbsoluteMaximumCapacity(label,
queueCapacities.getMaximumCapacity(label) * parent.getQueueCapacities()
.getAbsoluteMaximumCapacity(label));
// Re-visit max applications for a queue based on absolute capacity if
// needed.
if (this instanceof LeafQueue) {
LeafQueue leafQueue = (LeafQueue) this;
CapacitySchedulerConfiguration conf = csContext.getConfiguration();
int maxApplications = conf.getMaximumApplicationsPerQueue(queuePath);
if (maxApplications < 0) {
int maxGlobalPerQueueApps = conf.getGlobalMaximumApplicationsPerQueue();
if (maxGlobalPerQueueApps > 0) {
maxApplications = (int) (maxGlobalPerQueueApps * queueCapacities
.getAbsoluteCapacity(label));
} else{
maxApplications =
(int) (conf.getMaximumSystemApplications() * queueCapacities
.getAbsoluteCapacity(label));
}
}
leafQueue.setMaxApplications(maxApplications);
int maxApplicationsPerUser = Math.min(maxApplications,
(int) (maxApplications * (leafQueue.getUsersManager().getUserLimit()
/ 100.0f) * leafQueue.getUsersManager().getUserLimitFactor()));
leafQueue.setMaxApplicationsPerUser(maxApplicationsPerUser);
LOG.info("LeafQueue:" + leafQueue.getQueuePath() + ", maxApplications="
+ maxApplications + ", maxApplicationsPerUser="
+ maxApplicationsPerUser + ", Abs Cap:" + queueCapacities
.getAbsoluteCapacity(label) + ", Cap: " + queueCapacities
.getCapacity(label) + ", MaxCap : " + queueCapacities
.getMaximumCapacity(label));
}
}
{code}
But the queueResourceQuotas.getEffectiveMinResource is not correct when we add
a absolute auto created leaf queue, just exceed the capacity:
{code:java}
@Override
public AutoCreatedLeafQueueConfig getInitialLeafQueueConfiguration(
AbstractAutoCreatedLeafQueue leafQueue)
throws SchedulerDynamicEditException {
...
float availableCapacity = managedParentQueue.getQueueCapacities().
getAbsoluteCapacity(nodeLabel) - parentQueueState.
getAbsoluteActivatedChildQueueCapacity(nodeLabel) + EPSILON;
if (availableCapacity >= leafQueueTemplateCapacities
.getAbsoluteCapacity(nodeLabel)) {
...
} else{
updateToZeroCapacity(capacities, nodeLabel, leafQueue);
}
...
}
{code}
In updateToZeroCapacity we should change to ,For absolute auto created leaf
queue:
{code:java}
private void updateToZeroCapacity(QueueCapacities capacities,
String nodeLabel, LeafQueue leafQueue) {
capacities.setCapacity(nodeLabel, 0.0f);
capacities.setMaximumCapacity(nodeLabel,
leafQueueTemplateCapacities.getMaximumCapacity(nodeLabel));
leafQueue.getQueueResourceQuotas().
setConfiguredMinResource(nodeLabel, Resource.newInstance(0,0));
}
{code}
Then in calculateEffectiveResourcesAndCapacity the ratio will be correct, when
the absolute auto created leaf queue will not add t
[jira] [Updated] (YARN-10587) Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode.
[ https://issues.apache.org/jira/browse/YARN-10587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuqi updated YARN-10587: - Attachment: YARN-10587.001.patch > Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode. > - > > Key: YARN-10587 > URL: https://issues.apache.org/jira/browse/YARN-10587 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > Attachments: YARN-10587.001.patch > > > When introduced YARN-10504. > The logic related to auto created leaf queue changed. > The test in testAutoCreateLeafQueueCreation failed, we should fix the Error. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10587) Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode.
[ https://issues.apache.org/jira/browse/YARN-10587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuqi updated YARN-10587: - Attachment: (was: YARN-10587.001.patch) > Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode. > - > > Key: YARN-10587 > URL: https://issues.apache.org/jira/browse/YARN-10587 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > > When introduced YARN-10504. > The logic related to auto created leaf queue changed. > The test in testAutoCreateLeafQueueCreation failed, we should fix the Error. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10587) Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode.
[ https://issues.apache.org/jira/browse/YARN-10587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuqi updated YARN-10587: - Attachment: (was: YARN-10587.001.patch) > Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode. > - > > Key: YARN-10587 > URL: https://issues.apache.org/jira/browse/YARN-10587 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > > When introduced YARN-10504. > The logic related to auto created leaf queue changed. > The test in testAutoCreateLeafQueueCreation failed, we should fix the Error. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10579) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include weight values for queues
[
https://issues.apache.org/jira/browse/YARN-10579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17268696#comment-17268696
]
Hadoop QA commented on YARN-10579:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
41s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
11s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 51s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
47s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 33s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/522/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
| {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 7 new + 54 unchanged - 0 fixed = 61 total (was 54) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/522/artifact/out/whitespace-eol.txt{color}
| {color:red} The patch has 4 line(s) that end in whitespace. Use git apply
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:gree
[jira] [Commented] (YARN-10579) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include weight values for queues
[ https://issues.apache.org/jira/browse/YARN-10579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17268712#comment-17268712 ] Sunil G commented on YARN-10579: [~snemeth] test failures are related. > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include > weight values for queues > -- > > Key: YARN-10579 > URL: https://issues.apache.org/jira/browse/YARN-10579 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-10579.001.patch > > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > We would like to expose the weight values for all queues with the RM's > /scheduler REST endpoint. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10587) Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode.
[ https://issues.apache.org/jira/browse/YARN-10587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17268743#comment-17268743 ] Wangda Tan commented on YARN-10587: --- Thanks [~zhuqi], I think the fix is correct, but I want [~sunilg] also take a look at the patch. > Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode. > - > > Key: YARN-10587 > URL: https://issues.apache.org/jira/browse/YARN-10587 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > Attachments: YARN-10587.001.patch > > > When introduced YARN-10504. > The logic related to auto created leaf queue changed. > The test in testAutoCreateLeafQueueCreation failed, we should fix the Error. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10531) Be able to disable user limit factor for CapacityScheduler Leaf Queue
[
https://issues.apache.org/jira/browse/YARN-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17268755#comment-17268755
]
Wangda Tan commented on YARN-10531:
---
Thanks [~zhuqi],
Two minor comments:
1) ParentQueue.java:
- We can remove:
FIXME: Ideally we should disable user limit factor, see YARN-10531
2) AbstractCSQueue:
Nit: Let's breakdown
{code}
1542 int maxApplicationsPerUser =
1543 leafQueue.getUsersManager().getUserLimitFactor() != -1
1544 ? Math.min(maxApplications,
1545 (int) (maxApplications
1546 * (leafQueue.getUsersManager().getUserLimit() /
100.0f)
1547 * leafQueue.getUsersManager().getUserLimitFactor()))
1548 : maxApplications;
{code}
Into multiple statements for better readability.
Thoughts? [~sunilg], [~shuzirra], [~snemeth], [~pbacsko]
> Be able to disable user limit factor for CapacityScheduler Leaf Queue
> -
>
> Key: YARN-10531
> URL: https://issues.apache.org/jira/browse/YARN-10531
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: zhuqi
>Priority: Major
> Attachments: YARN-10531.001.patch, YARN-10531.002.patch,
> YARN-10531.003.patch, YARN-10531.004.patch, YARN-10531.005.patch
>
>
> User limit factor is used to define max cap of how much resource can be
> consumed by single user.
> Under Auto Queue Creation context, it doesn't make much sense to set user
> limit factor, because initially every queue will set weight to 1.0, we want
> user can consume more resource if possible. It is hard to pre-determine how
> to set up user limit factor. So it makes more sense to add a new value (like
> -1) to indicate we will disable user limit factor
> Logic need to be changed is below:
> (Inside LeafQueue.java)
> {code}
> Resource maxUserLimit = Resources.none();
> if (schedulingMode == SchedulingMode.RESPECT_PARTITION_EXCLUSIVITY) {
> maxUserLimit = Resources.multiplyAndRoundDown(queueCapacity,
> getUserLimitFactor());
> } else if (schedulingMode == SchedulingMode.IGNORE_PARTITION_EXCLUSIVITY)
> {
> maxUserLimit = partitionResource;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10352) Skip schedule on not heartbeated nodes in Multi Node Placement
[ https://issues.apache.org/jira/browse/YARN-10352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17268757#comment-17268757 ] Wangda Tan commented on YARN-10352: --- cc: [~ztang] to do a review. > Skip schedule on not heartbeated nodes in Multi Node Placement > -- > > Key: YARN-10352 > URL: https://issues.apache.org/jira/browse/YARN-10352 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.3.0, 3.4.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Labels: capacityscheduler, multi-node-placement > Attachments: YARN-10352-001.patch, YARN-10352-002.patch, > YARN-10352-003.patch, YARN-10352-004.patch, YARN-10352-005.patch, > YARN-10352-006.patch, YARN-10352-007.patch, YARN-10352-008.patch, > YARN-10352-010.patch, YARN-10352.009.patch > > > When Node Recovery is Enabled, Stopping a NM won't unregister to RM. So RM > Active Nodes will be still having those stopped nodes until NM Liveliness > Monitor Expires after configured timeout > (yarn.nm.liveness-monitor.expiry-interval-ms = 10 mins). During this 10mins, > Multi Node Placement assigns the containers on those nodes. They need to > exclude the nodes which has not heartbeated for configured heartbeat interval > (yarn.resourcemanager.nodemanagers.heartbeat-interval-ms=1000ms) similar to > Asynchronous Capacity Scheduler Threads. > (CapacityScheduler#shouldSkipNodeSchedule) > *Repro:* > 1. Enable Multi Node Placement > (yarn.scheduler.capacity.multi-node-placement-enabled) + Node Recovery > Enabled (yarn.node.recovery.enabled) > 2. Have only one NM running say worker0 > 3. Stop worker0 and start any other NM say worker1 > 4. Submit a sleep job. The containers will timeout as assigned to stopped NM > worker0. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10587) Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode.
[
https://issues.apache.org/jira/browse/YARN-10587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17268821#comment-17268821
]
Hadoop QA commented on YARN-10587:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 30m
43s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
48s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 8s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
49s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 31s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/523/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
| {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 19 unchanged - 0 fixed = 20 total (was 19) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 35s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color
[jira] [Commented] (YARN-10579) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include weight values for queues
[ https://issues.apache.org/jira/browse/YARN-10579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17268849#comment-17268849 ] Szilard Nemeth commented on YARN-10579: --- [~wangda] Could you please help to review this? > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include > weight values for queues > -- > > Key: YARN-10579 > URL: https://issues.apache.org/jira/browse/YARN-10579 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-10579.001.patch, YARN-10579.002.patch > > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > We would like to expose the weight values for all queues with the RM's > /scheduler REST endpoint. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10579) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include weight values for queues
[ https://issues.apache.org/jira/browse/YARN-10579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-10579: -- Attachment: YARN-10579.002.patch > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include > weight values for queues > -- > > Key: YARN-10579 > URL: https://issues.apache.org/jira/browse/YARN-10579 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-10579.001.patch, YARN-10579.002.patch > > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > We would like to expose the weight values for all queues with the RM's > /scheduler REST endpoint. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10579) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include weight values for queues
[ https://issues.apache.org/jira/browse/YARN-10579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17268848#comment-17268848 ] Szilard Nemeth commented on YARN-10579: --- Thanks [~sunilg], Just uploading a patch that fixes those. Locally, all the failed tests were passed, except hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer, which is an unrelated flakey. > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include > weight values for queues > -- > > Key: YARN-10579 > URL: https://issues.apache.org/jira/browse/YARN-10579 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-10579.001.patch, YARN-10579.002.patch > > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > We would like to expose the weight values for all queues with the RM's > /scheduler REST endpoint. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10581) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include queue creation type for queues
[ https://issues.apache.org/jira/browse/YARN-10581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-10581: -- Attachment: YARN-10581.001.patch > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include > queue creation type for queues > > > Key: YARN-10581 > URL: https://issues.apache.org/jira/browse/YARN-10581 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-10581.001.patch > > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > Auto-queue creation has been also imlemented with YARN-10506. > Connected to this effort, we would like to expose the type of the queue with > the RM's /scheduler REST endpoint. > The queue type should hold these values: > * Auto-created parent queue: *autoCreatedParent* > * Auto-created leaf queue: *autoCreatedLeaf* > * Static parent: *staticParent* > * Static leaf: *staticLeaf* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10581) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include queue creation type for queues
[ https://issues.apache.org/jira/browse/YARN-10581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17268852#comment-17268852 ] Szilard Nemeth commented on YARN-10581: --- Attached first patch. This will only compile if YARN-10579 is merged in. > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include > queue creation type for queues > > > Key: YARN-10581 > URL: https://issues.apache.org/jira/browse/YARN-10581 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-10581.001.patch > > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > Auto-queue creation has been also imlemented with YARN-10506. > Connected to this effort, we would like to expose the type of the queue with > the RM's /scheduler REST endpoint. > The queue type should hold these values: > * Auto-created parent queue: *autoCreatedParent* > * Auto-created leaf queue: *autoCreatedLeaf* > * Static parent: *staticParent* > * Static leaf: *staticLeaf* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10579) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include weight values for queues
[ https://issues.apache.org/jira/browse/YARN-10579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17268857#comment-17268857 ] Wangda Tan commented on YARN-10579: --- Thanks [~snemeth], +1 to the latest patch. > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include > weight values for queues > -- > > Key: YARN-10579 > URL: https://issues.apache.org/jira/browse/YARN-10579 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-10579.001.patch, YARN-10579.002.patch > > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > We would like to expose the weight values for all queues with the RM's > /scheduler REST endpoint. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10579) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include weight values for queues
[
https://issues.apache.org/jira/browse/YARN-10579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17268913#comment-17268913
]
Hadoop QA commented on YARN-10579:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
12s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 3
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
3s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 34s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
50s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 32s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/524/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
| {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 7 new + 109 unchanged - 0 fixed = 116 total (was 109)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/524/artifact/out/whitespace-eol.txt{color}
| {color:red} The patch has 4 line(s) that end in whitespace. Use git apply
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:g
[jira] [Updated] (YARN-10587) Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode.
[ https://issues.apache.org/jira/browse/YARN-10587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuqi updated YARN-10587: - Attachment: YARN-10587.002.patch > Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode. > - > > Key: YARN-10587 > URL: https://issues.apache.org/jira/browse/YARN-10587 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > Attachments: YARN-10587.001.patch, YARN-10587.002.patch > > > When introduced YARN-10504. > The logic related to auto created leaf queue changed. > The test in testAutoCreateLeafQueueCreation failed, we should fix the Error. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10587) Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode.
[ https://issues.apache.org/jira/browse/YARN-10587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17268983#comment-17268983 ] zhuqi commented on YARN-10587: -- [~wangda] [~sunilg] Fix related TestAbsoluteResourceConfiguration in latest patch. > Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode. > - > > Key: YARN-10587 > URL: https://issues.apache.org/jira/browse/YARN-10587 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > Attachments: YARN-10587.001.patch, YARN-10587.002.patch > > > When introduced YARN-10504. > The logic related to auto created leaf queue changed. > The test in testAutoCreateLeafQueueCreation failed, we should fix the Error. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10531) Be able to disable user limit factor for CapacityScheduler Leaf Queue
[
https://issues.apache.org/jira/browse/YARN-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhuqi updated YARN-10531:
-
Attachment: YARN-10531.006.patch
> Be able to disable user limit factor for CapacityScheduler Leaf Queue
> -
>
> Key: YARN-10531
> URL: https://issues.apache.org/jira/browse/YARN-10531
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: zhuqi
>Priority: Major
> Attachments: YARN-10531.001.patch, YARN-10531.002.patch,
> YARN-10531.003.patch, YARN-10531.004.patch, YARN-10531.005.patch,
> YARN-10531.006.patch
>
>
> User limit factor is used to define max cap of how much resource can be
> consumed by single user.
> Under Auto Queue Creation context, it doesn't make much sense to set user
> limit factor, because initially every queue will set weight to 1.0, we want
> user can consume more resource if possible. It is hard to pre-determine how
> to set up user limit factor. So it makes more sense to add a new value (like
> -1) to indicate we will disable user limit factor
> Logic need to be changed is below:
> (Inside LeafQueue.java)
> {code}
> Resource maxUserLimit = Resources.none();
> if (schedulingMode == SchedulingMode.RESPECT_PARTITION_EXCLUSIVITY) {
> maxUserLimit = Resources.multiplyAndRoundDown(queueCapacity,
> getUserLimitFactor());
> } else if (schedulingMode == SchedulingMode.IGNORE_PARTITION_EXCLUSIVITY)
> {
> maxUserLimit = partitionResource;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10531) Be able to disable user limit factor for CapacityScheduler Leaf Queue
[
https://issues.apache.org/jira/browse/YARN-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhuqi updated YARN-10531:
-
Attachment: YARN-10531.007.patch
> Be able to disable user limit factor for CapacityScheduler Leaf Queue
> -
>
> Key: YARN-10531
> URL: https://issues.apache.org/jira/browse/YARN-10531
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: zhuqi
>Priority: Major
> Attachments: YARN-10531.001.patch, YARN-10531.002.patch,
> YARN-10531.003.patch, YARN-10531.004.patch, YARN-10531.005.patch,
> YARN-10531.006.patch, YARN-10531.007.patch
>
>
> User limit factor is used to define max cap of how much resource can be
> consumed by single user.
> Under Auto Queue Creation context, it doesn't make much sense to set user
> limit factor, because initially every queue will set weight to 1.0, we want
> user can consume more resource if possible. It is hard to pre-determine how
> to set up user limit factor. So it makes more sense to add a new value (like
> -1) to indicate we will disable user limit factor
> Logic need to be changed is below:
> (Inside LeafQueue.java)
> {code}
> Resource maxUserLimit = Resources.none();
> if (schedulingMode == SchedulingMode.RESPECT_PARTITION_EXCLUSIVITY) {
> maxUserLimit = Resources.multiplyAndRoundDown(queueCapacity,
> getUserLimitFactor());
> } else if (schedulingMode == SchedulingMode.IGNORE_PARTITION_EXCLUSIVITY)
> {
> maxUserLimit = partitionResource;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10531) Be able to disable user limit factor for CapacityScheduler Leaf Queue
[
https://issues.apache.org/jira/browse/YARN-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17269009#comment-17269009
]
zhuqi edited comment on YARN-10531 at 1/21/21, 3:29 AM:
Thanks [~wangda] for review.
I have fixed above, in latest patch.
was (Author: zhuqi):
[~wangda]
I have fixed above, in latest patch.
> Be able to disable user limit factor for CapacityScheduler Leaf Queue
> -
>
> Key: YARN-10531
> URL: https://issues.apache.org/jira/browse/YARN-10531
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: zhuqi
>Priority: Major
> Attachments: YARN-10531.001.patch, YARN-10531.002.patch,
> YARN-10531.003.patch, YARN-10531.004.patch, YARN-10531.005.patch,
> YARN-10531.006.patch, YARN-10531.007.patch
>
>
> User limit factor is used to define max cap of how much resource can be
> consumed by single user.
> Under Auto Queue Creation context, it doesn't make much sense to set user
> limit factor, because initially every queue will set weight to 1.0, we want
> user can consume more resource if possible. It is hard to pre-determine how
> to set up user limit factor. So it makes more sense to add a new value (like
> -1) to indicate we will disable user limit factor
> Logic need to be changed is below:
> (Inside LeafQueue.java)
> {code}
> Resource maxUserLimit = Resources.none();
> if (schedulingMode == SchedulingMode.RESPECT_PARTITION_EXCLUSIVITY) {
> maxUserLimit = Resources.multiplyAndRoundDown(queueCapacity,
> getUserLimitFactor());
> } else if (schedulingMode == SchedulingMode.IGNORE_PARTITION_EXCLUSIVITY)
> {
> maxUserLimit = partitionResource;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10531) Be able to disable user limit factor for CapacityScheduler Leaf Queue
[
https://issues.apache.org/jira/browse/YARN-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17269009#comment-17269009
]
zhuqi commented on YARN-10531:
--
[~wangda]
I have fixed above, in latest patch.
> Be able to disable user limit factor for CapacityScheduler Leaf Queue
> -
>
> Key: YARN-10531
> URL: https://issues.apache.org/jira/browse/YARN-10531
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: zhuqi
>Priority: Major
> Attachments: YARN-10531.001.patch, YARN-10531.002.patch,
> YARN-10531.003.patch, YARN-10531.004.patch, YARN-10531.005.patch,
> YARN-10531.006.patch, YARN-10531.007.patch
>
>
> User limit factor is used to define max cap of how much resource can be
> consumed by single user.
> Under Auto Queue Creation context, it doesn't make much sense to set user
> limit factor, because initially every queue will set weight to 1.0, we want
> user can consume more resource if possible. It is hard to pre-determine how
> to set up user limit factor. So it makes more sense to add a new value (like
> -1) to indicate we will disable user limit factor
> Logic need to be changed is below:
> (Inside LeafQueue.java)
> {code}
> Resource maxUserLimit = Resources.none();
> if (schedulingMode == SchedulingMode.RESPECT_PARTITION_EXCLUSIVITY) {
> maxUserLimit = Resources.multiplyAndRoundDown(queueCapacity,
> getUserLimitFactor());
> } else if (schedulingMode == SchedulingMode.IGNORE_PARTITION_EXCLUSIVITY)
> {
> maxUserLimit = partitionResource;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10559) Fair sharing intra-queue preemption support in Capacity Scheduler
[ https://issues.apache.org/jira/browse/YARN-10559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] VADAGA ANANYO RAO updated YARN-10559: - Attachment: YARN-10559.0008.patch > Fair sharing intra-queue preemption support in Capacity Scheduler > - > > Key: YARN-10559 > URL: https://issues.apache.org/jira/browse/YARN-10559 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 3.1.4 >Reporter: VADAGA ANANYO RAO >Assignee: VADAGA ANANYO RAO >Priority: Major > Attachments: FairOP_preemption-design_doc_v1.pdf, > FairOP_preemption-design_doc_v2.pdf, YARN-10559.0001.patch, > YARN-10559.0002.patch, YARN-10559.0003.patch, YARN-10559.0004.patch, > YARN-10559.0005.patch, YARN-10559.0006.patch, YARN-10559.0007.patch, > YARN-10559.0008.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > Usecase: > Due to the way Capacity Scheduler preemption works, If a single user submits > a large application to a queue (using 100% of resources), that job will not > be preempted by future applications from the same user within the same queue. > This implies that the later applications will be forced to wait for > completion of the long running application. This prevents multiple long > running, large, applications from running concurrently. > Support fair sharing among apps while preempting applications from same queue. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10587) Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode.
[
https://issues.apache.org/jira/browse/YARN-10587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17269036#comment-17269036
]
Hadoop QA commented on YARN-10587:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
20s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 2
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
47s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 21s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
48s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
48s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 22s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green}{color} | {c
[jira] [Commented] (YARN-10559) Fair sharing intra-queue preemption support in Capacity Scheduler
[
https://issues.apache.org/jira/browse/YARN-10559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17269050#comment-17269050
]
Hadoop QA commented on YARN-10559:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 44m
5s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 1s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 4
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 1m
53s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/529/artifact/out/branch-mvninstall-root.txt{color}
| {color:red} root in trunk failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
23s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/529/artifact/out/branch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt{color}
| {color:red} hadoop-yarn-server-resourcemanager in trunk failed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
25s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/529/artifact/out/branch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt{color}
| {color:red} hadoop-yarn-server-resourcemanager in trunk failed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 21s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/529/artifact/out/buildtool-branch-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
| {color:orange} The patch fails to run checkstyle in
hadoop-yarn-server-resourcemanager {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
23s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/529/artifact/out/branch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
| {color:red} hadoop-yarn-server-resourcemanager in trunk failed. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
1m 9s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
22s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/529/artifact/out/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt{color}
| {color:red} hadoop-yarn-server-resourcemanager in trunk failed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
23s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/529/artifact/out/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt{color}
| {color:red} hadoop-yarn-server-resourcemanager in trunk failed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01. {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m
18s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
23s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/529/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
| {color:red} hadoop-yarn-server-resourcemanager in trunk failed. {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
24s{color} |
{color:red}https://ci-hadoop.apache.or
[jira] [Commented] (YARN-10531) Be able to disable user limit factor for CapacityScheduler Leaf Queue
[
https://issues.apache.org/jira/browse/YARN-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17269052#comment-17269052
]
Hadoop QA commented on YARN-10531:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
40s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
1s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 2
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
58s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 54s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
47s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 59s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green}{color} | {col
[jira] [Commented] (YARN-10531) Be able to disable user limit factor for CapacityScheduler Leaf Queue
[
https://issues.apache.org/jira/browse/YARN-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17269053#comment-17269053
]
Hadoop QA commented on YARN-10531:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
30s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
1s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 2
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
12s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 41s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
55s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
52s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 7s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green}{color} | {c
[jira] [Commented] (YARN-10519) Refactor QueueMetricsForCustomResources class to move to yarn-common package
[ https://issues.apache.org/jira/browse/YARN-10519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17269067#comment-17269067 ] Bibin Chundatt commented on YARN-10519: --- Testcase failures are unrelated to patch attached and passing locally. Thank you [~minni31] for contribution .. Committed to trunk > Refactor QueueMetricsForCustomResources class to move to yarn-common package > > > Key: YARN-10519 > URL: https://issues.apache.org/jira/browse/YARN-10519 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Minni Mittal >Assignee: Minni Mittal >Priority: Major > Attachments: YARN-10519.v1.patch, YARN-10519.v2.patch, > YARN-10519.v3.patch, YARN-10519.v4.patch, YARN-10519.v5.patch, > YARN-10519.v6.patch, YARN-10519.v7.patch > > > Refactor the code for QueueMetricsForCustomResources to move the base classes > to yarn-common package. This helps in reusing the class in adding custom > resource types at NM level also. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10585) Create a class which can convert from legacy mapping rule format to the new JSON format
[
https://issues.apache.org/jira/browse/YARN-10585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17269089#comment-17269089
]
Hadoop QA commented on YARN-10585:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 26m
7s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
22s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 32s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
49s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
56s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 30s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/528/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
| {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 0s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
