[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN
[ https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947430#comment-16947430 ] Hadoop QA commented on YARN-9860: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 35s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 45s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 8s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 51s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 21s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 50s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 15m 49s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 55s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 26s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 15s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 16s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 7m 16s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 16s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 17s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 91 unchanged - 1 fixed = 91 total (was 92) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 0s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 17s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m 12s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 18m 55s{color} | {color:green} hadoop-yarn-services-core in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 23s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 40s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}119m 10s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=19.03.3 Server=19.03.3
[jira] [Created] (YARN-9880) In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is incorrect.
jenny created YARN-9880: --- Summary: In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is incorrect. Key: YARN-9880 URL: https://issues.apache.org/jira/browse/YARN-9880 Project: Hadoop YARN Issue Type: Bug Components: yarn-ui-v2 Affects Versions: 3.2.1, 3.1.1 Reporter: jenny -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9881) In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is incorrect.
jenny created YARN-9881: --- Summary: In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is incorrect. Key: YARN-9881 URL: https://issues.apache.org/jira/browse/YARN-9881 Project: Hadoop YARN Issue Type: Bug Components: yarn-ui-v2 Affects Versions: 3.2.1, 3.1.1 Reporter: jenny -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9699) Migration tool that help to generate CS config based on FS config (Phase 1)
[ https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-9699: - Attachment: YARN-9699-009.patch > Migration tool that help to generate CS config based on FS config (Phase 1) > --- > > Key: YARN-9699 > URL: https://issues.apache.org/jira/browse/YARN-9699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wanqiang Ji >Assignee: Peter Bacsko >Priority: Major > Attachments: FS_to_CS_migration_POC.patch, YARN-9699-003.patch, > YARN-9699-004.patch, YARN-9699-005.patch, YARN-9699-006.patch, > YARN-9699-007.patch, YARN-9699-008.patch, YARN-9699-009.patch, > YARN-9699.001.patch, YARN-9699.002.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9699) Migration tool that help to generate CS config based on FS config (Phase 1)
[ https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947449#comment-16947449 ] Szilard Nemeth commented on YARN-9699: -- Hi [~pbacsko]! As discussed, patch008 contained an error near console mode: I haven't printed into the output stream at all if console mode was on. Attached patch009 to address this issue. > Migration tool that help to generate CS config based on FS config (Phase 1) > --- > > Key: YARN-9699 > URL: https://issues.apache.org/jira/browse/YARN-9699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wanqiang Ji >Assignee: Peter Bacsko >Priority: Major > Attachments: FS_to_CS_migration_POC.patch, YARN-9699-003.patch, > YARN-9699-004.patch, YARN-9699-005.patch, YARN-9699-006.patch, > YARN-9699-007.patch, YARN-9699-008.patch, YARN-9699-009.patch, > YARN-9699.001.patch, YARN-9699.002.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-9881) In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is incorrect.
[ https://issues.apache.org/jira/browse/YARN-9881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prabhu Joseph resolved YARN-9881. - Resolution: Duplicate > In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is > incorrect. > - > > Key: YARN-9881 > URL: https://issues.apache.org/jira/browse/YARN-9881 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Affects Versions: 3.1.1, 3.2.1 >Reporter: jenny >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9881) In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is incorrect.
[ https://issues.apache.org/jira/browse/YARN-9881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947454#comment-16947454 ] Prabhu Joseph commented on YARN-9881: - [~cjn082030] This looks like a duplicate of YARN-9880. Will close this Jira as Duplicate. > In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is > incorrect. > - > > Key: YARN-9881 > URL: https://issues.apache.org/jira/browse/YARN-9881 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Affects Versions: 3.1.1, 3.2.1 >Reporter: jenny >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9699) Migration tool that help to generate CS config based on FS config (Phase 1)
[ https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947455#comment-16947455 ] Szilard Nemeth commented on YARN-9699: -- Also attached patch010 to address the remaining checkstyle / findbugs issues. We can ignore the "variable hides a field" kind of errors. > Migration tool that help to generate CS config based on FS config (Phase 1) > --- > > Key: YARN-9699 > URL: https://issues.apache.org/jira/browse/YARN-9699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wanqiang Ji >Assignee: Peter Bacsko >Priority: Major > Attachments: FS_to_CS_migration_POC.patch, YARN-9699-003.patch, > YARN-9699-004.patch, YARN-9699-005.patch, YARN-9699-006.patch, > YARN-9699-007.patch, YARN-9699-008.patch, YARN-9699-009.patch, > YARN-9699-010.patch, YARN-9699.001.patch, YARN-9699.002.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9699) Migration tool that help to generate CS config based on FS config (Phase 1)
[ https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-9699: - Attachment: YARN-9699-010.patch > Migration tool that help to generate CS config based on FS config (Phase 1) > --- > > Key: YARN-9699 > URL: https://issues.apache.org/jira/browse/YARN-9699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wanqiang Ji >Assignee: Peter Bacsko >Priority: Major > Attachments: FS_to_CS_migration_POC.patch, YARN-9699-003.patch, > YARN-9699-004.patch, YARN-9699-005.patch, YARN-9699-006.patch, > YARN-9699-007.patch, YARN-9699-008.patch, YARN-9699-009.patch, > YARN-9699-010.patch, YARN-9699.001.patch, YARN-9699.002.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9881) In YARN ui2 attempts tab, The running Application Attempt's Container's ElapsedTime is incorrect.
[ https://issues.apache.org/jira/browse/YARN-9881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jenny updated YARN-9881: Summary: In YARN ui2 attempts tab, The running Application Attempt's Container's ElapsedTime is incorrect. (was: In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is incorrect.) > In YARN ui2 attempts tab, The running Application Attempt's Container's > ElapsedTime is incorrect. > - > > Key: YARN-9881 > URL: https://issues.apache.org/jira/browse/YARN-9881 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Affects Versions: 3.1.1, 3.2.1 >Reporter: jenny >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9880) In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is incorrect.
[ https://issues.apache.org/jira/browse/YARN-9880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jenny updated YARN-9880: Description: In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is incorrect. When Browser and Yarn server are in different time zones, the Application Attempt's ElapsedTime is greater than the Application's ElapsedTime. > In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is > incorrect. > - > > Key: YARN-9880 > URL: https://issues.apache.org/jira/browse/YARN-9880 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Affects Versions: 3.1.1, 3.2.1 >Reporter: jenny >Priority: Major > > In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is > incorrect. > When Browser and Yarn server are in different time zones, the Application > Attempt's ElapsedTime is greater than the Application's ElapsedTime. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9880) In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is incorrect.
[ https://issues.apache.org/jira/browse/YARN-9880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947459#comment-16947459 ] Prabhu Joseph commented on YARN-9880: - [~cjn082030] Can you share how you are verifying Application Attempt's ElapsedTime is incorrect. Are you comparing with UI1. RM UI screenshots will help to under the issue better. > In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is > incorrect. > - > > Key: YARN-9880 > URL: https://issues.apache.org/jira/browse/YARN-9880 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Affects Versions: 3.1.1, 3.2.1 >Reporter: jenny >Priority: Major > > In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is > incorrect. > When Browser and Yarn server are in different time zones, the Application > Attempt's ElapsedTime is greater than the Application's ElapsedTime. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9699) Migration tool that help to generate CS config based on FS config (Phase 1)
[ https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947461#comment-16947461 ] Peter Bacsko commented on YARN-9699: Ok, here are my comments - I collected these when patch v2 was the latest but I believe most of them are still valid. #1 {noformat} 137 throw new IllegalArgumentException( 138 String.format("Missing %s parameter " + "(switch: %s|%s).", 139 cliOption.name, cliOption.shortSwitch, cliOption.longSwitch)); {noformat} {noformat} 158 if (isFile && file.isDirectory()) { 159 throw new IllegalArgumentException(String.format("Specified path %s is a directory but should be a file " + 160 "(As value of parameter %s)", filePath, cliOption.name)); 161 } else if (!isFile && !file.isDirectory()) { 162 throw new IllegalArgumentException(String.format("Specified path %s is not a directory " + 163 "(As value of parameter %s)", filePath, cliOption.name)); 164 } else if (!file.exists()) { 165 throw new IllegalArgumentException(String.format("Specified path %s does not exist " + 166 "(As value of parameter %s)", filePath, cliOption.name)); 167 } {noformat} These exceptions will propagate up to {{fsConfigConversionArgumentHandler.parseAndConvert(args);}} and errors will be printed with a stack trace. Wouldn't it be better to use a different exception and handle it without printing the stack trace? #2 {noformat} 81private enum OutputFile { 82 YARN_SITE_XML("yarn-site.xml"), 83 CAPACITY_SCHEDULER_XML("capacity-scheduler.xml"); 84 85 private final String name; 86 87 OutputFile(String name) { 88this.name = name; 89 } 90} {noformat} Do we need an enum for this purpose? #3 {noformat} if (maxAssign != -1) {noformat} "-1" is a magic number, replace it with {{FairSchedulerConfiguration.DEFAULT_MAX_ASSIGN}}. #4 {noformat} 435 if (queueMaxAmShare != 0.0f 436 && queueMaxAmShare != queueMaxAMshareDefault 437 && queueMaxAmShare != -1.0f) { 438 capacacitySchedulerConfig.set("yarn.scheduler.capacity." + queueName + ".maximum-am-resource-percent", 439 String.valueOf(queueMaxAmShare)); 440 } 441 442 if (queueMaxAmShare == -1.0f) { 443 capacacitySchedulerConfig.set("yarn.scheduler.capacity." + queueName + ".maximum-am-resource-percent", 444 "1.0"); 445 } {noformat} It's my part, but direct floating point comparison could be dangerous. Let's think this over. Basically we have to make sure that float converted from a string "-1.0" is the same as the {{-1.0}} literal in the code. #5 {noformat} 448 private void emitMaxRunningApps(String queueName, FSQueue queue) { 449 if (queue.getMaxRunningApps() != Integer.MAX_VALUE {noformat} {{Integer.MAX_VALUE}} is a magic number, extract it to a constant. #6 {{Resources.unbounded().compareTo(...)}} This comparison is repeated many times, should be extracted to a simple helper method. #7 {noformat} 681 } else if (children.size() == 1) { 682 // just a single element, probably this branch should 683 // not be reached 684 String queueName = children.get(0).getName(); 685 capacities.put(queueName, Capacity.newCapacity(HUNDRED)); 686 } {noformat} This branch is probably unnecessary. Handle {{children.size() == 0}} and {{children.size() == 1}} cases separately before walking through the children list. > Migration tool that help to generate CS config based on FS config (Phase 1) > --- > > Key: YARN-9699 > URL: https://issues.apache.org/jira/browse/YARN-9699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wanqiang Ji >Assignee: Peter Bacsko >Priority: Major > Attachments: FS_to_CS_migration_POC.patch, YARN-9699-003.patch, > YARN-9699-004.patch, YARN-9699-005.patch, YARN-9699-006.patch, > YARN-9699-007.patch, YARN-9699-008.patch, YARN-9699-009.patch, > YARN-9699-010.patch, YARN-9699.001.patch, YARN-9699.002.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9880) In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is incorrect.
[ https://issues.apache.org/jira/browse/YARN-9880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jenny updated YARN-9880: Attachment: image-2019-10-09-17-22-31-603.png > In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is > incorrect. > - > > Key: YARN-9880 > URL: https://issues.apache.org/jira/browse/YARN-9880 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Affects Versions: 3.1.1, 3.2.1 >Reporter: jenny >Priority: Major > Attachments: image-2019-10-09-17-22-31-603.png > > > In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is > incorrect. > When Browser and Yarn server are in different time zones, the Application > Attempt's ElapsedTime is greater than the Application's ElapsedTime. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9880) In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is incorrect.
[ https://issues.apache.org/jira/browse/YARN-9880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jenny updated YARN-9880: Attachment: image-2019-10-09-17-23-08-311.png > In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is > incorrect. > - > > Key: YARN-9880 > URL: https://issues.apache.org/jira/browse/YARN-9880 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Affects Versions: 3.1.1, 3.2.1 >Reporter: jenny >Priority: Major > Attachments: image-2019-10-09-17-22-31-603.png > > > In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is > incorrect. > When Browser and Yarn server are in different time zones, the Application > Attempt's ElapsedTime is greater than the Application's ElapsedTime. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9880) In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is incorrect.
[ https://issues.apache.org/jira/browse/YARN-9880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jenny updated YARN-9880: Attachment: (was: image-2019-10-09-17-23-08-311.png) > In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is > incorrect. > - > > Key: YARN-9880 > URL: https://issues.apache.org/jira/browse/YARN-9880 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Affects Versions: 3.1.1, 3.2.1 >Reporter: jenny >Priority: Major > Attachments: image-2019-10-09-17-22-31-603.png > > > In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is > incorrect. > When Browser and Yarn server are in different time zones, the Application > Attempt's ElapsedTime is greater than the Application's ElapsedTime. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9880) In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is incorrect.
[ https://issues.apache.org/jira/browse/YARN-9880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947476#comment-16947476 ] jenny commented on YARN-9880: - Hi [~Prabhu Joseph] , please see the screenshots . > In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is > incorrect. > - > > Key: YARN-9880 > URL: https://issues.apache.org/jira/browse/YARN-9880 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Affects Versions: 3.1.1, 3.2.1 >Reporter: jenny >Priority: Major > Attachments: image-2019-10-09-17-22-31-603.png > > > In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is > incorrect. > When Browser and Yarn server are in different time zones, the Application > Attempt's ElapsedTime is greater than the Application's ElapsedTime. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9880) In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is incorrect.
[ https://issues.apache.org/jira/browse/YARN-9880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947476#comment-16947476 ] jenny edited comment on YARN-9880 at 10/9/19 9:27 AM: -- Hi [~Prabhu Joseph] , please see the screenshots (UI2) . UI1 is right. was (Author: cjn082030): Hi [~Prabhu Joseph] , please see the screenshots . > In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is > incorrect. > - > > Key: YARN-9880 > URL: https://issues.apache.org/jira/browse/YARN-9880 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Affects Versions: 3.1.1, 3.2.1 >Reporter: jenny >Priority: Major > Attachments: image-2019-10-09-17-22-31-603.png > > > In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is > incorrect. > When Browser and Yarn server are in different time zones, the Application > Attempt's ElapsedTime is greater than the Application's ElapsedTime. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9256) Make ATSv2 compilation default with hbase.profile=2.0
[ https://issues.apache.org/jira/browse/YARN-9256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947487#comment-16947487 ] Prabhu Joseph commented on YARN-9256: - [~rohithsharma] I will work on this, will assign this Jira to me. > Make ATSv2 compilation default with hbase.profile=2.0 > - > > Key: YARN-9256 > URL: https://issues.apache.org/jira/browse/YARN-9256 > Project: Hadoop YARN > Issue Type: Task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Major > Attachments: YARN-9256.01.patch > > > By default Hadoop compiles with hbase.profile one which corresponds to > hbase.version=1.4 for ATSv2. Change compilation to hbase.profile=2.0 by > default in trunk. > This JIRA is to discuss for any concerns. > cc:/ [~vrushalic] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9256) Make ATSv2 compilation default with hbase.profile=2.0
[ https://issues.apache.org/jira/browse/YARN-9256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prabhu Joseph reassigned YARN-9256: --- Assignee: Prabhu Joseph (was: Rohith Sharma K S) > Make ATSv2 compilation default with hbase.profile=2.0 > - > > Key: YARN-9256 > URL: https://issues.apache.org/jira/browse/YARN-9256 > Project: Hadoop YARN > Issue Type: Task >Reporter: Rohith Sharma K S >Assignee: Prabhu Joseph >Priority: Major > Attachments: YARN-9256.01.patch > > > By default Hadoop compiles with hbase.profile one which corresponds to > hbase.version=1.4 for ATSv2. Change compilation to hbase.profile=2.0 by > default in trunk. > This JIRA is to discuss for any concerns. > cc:/ [~vrushalic] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Reopened] (YARN-9881) In YARN ui2 attempts tab, The running Application Attempt's Container's ElapsedTime is incorrect.
[ https://issues.apache.org/jira/browse/YARN-9881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prabhu Joseph reopened YARN-9881: - > In YARN ui2 attempts tab, The running Application Attempt's Container's > ElapsedTime is incorrect. > - > > Key: YARN-9881 > URL: https://issues.apache.org/jira/browse/YARN-9881 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Affects Versions: 3.1.1, 3.2.1 >Reporter: jenny >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (YARN-9881) In YARN ui2 attempts tab, The running Application Attempt's Container's ElapsedTime is incorrect.
[ https://issues.apache.org/jira/browse/YARN-9881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prabhu Joseph updated YARN-9881: Comment: was deleted (was: [~cjn082030] This looks like a duplicate of YARN-9880. Will close this Jira as Duplicate.) > In YARN ui2 attempts tab, The running Application Attempt's Container's > ElapsedTime is incorrect. > - > > Key: YARN-9881 > URL: https://issues.apache.org/jira/browse/YARN-9881 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Affects Versions: 3.1.1, 3.2.1 >Reporter: jenny >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9873) Mutation API Config Change updates Version Number
[ https://issues.apache.org/jira/browse/YARN-9873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947523#comment-16947523 ] Sunil G commented on YARN-9873: --- Thanks [~Prabhu Joseph] for addressing comments Pushing this in . +1 > Mutation API Config Change updates Version Number > -- > > Key: YARN-9873 > URL: https://issues.apache.org/jira/browse/YARN-9873 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Fix For: 3.3.0, 3.2.2 > > Attachments: YARN-9873-001.patch, YARN-9873-002.patch, > YARN-9873-003.patch, YARN-9873-004.patch > > > Version Number support for each Scheduler Config Change. This also helps to > know when the last change happened. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9873) Mutation API Config Change need to update Version Number
[ https://issues.apache.org/jira/browse/YARN-9873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-9873: -- Summary: Mutation API Config Change need to update Version Number (was: Mutation API Config Change updates Version Number ) > Mutation API Config Change need to update Version Number > - > > Key: YARN-9873 > URL: https://issues.apache.org/jira/browse/YARN-9873 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Fix For: 3.3.0, 3.2.2 > > Attachments: YARN-9873-001.patch, YARN-9873-002.patch, > YARN-9873-003.patch, YARN-9873-004.patch > > > Version Number support for each Scheduler Config Change. This also helps to > know when the last change happened. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9873) Mutation API Config Change need to update Version Number
[ https://issues.apache.org/jira/browse/YARN-9873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947524#comment-16947524 ] Sunil G commented on YARN-9873: --- Committed to trunk/branch-3.2 > Mutation API Config Change need to update Version Number > - > > Key: YARN-9873 > URL: https://issues.apache.org/jira/browse/YARN-9873 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Fix For: 3.3.0, 3.2.2 > > Attachments: YARN-9873-001.patch, YARN-9873-002.patch, > YARN-9873-003.patch, YARN-9873-004.patch > > > Version Number support for each Scheduler Config Change. This also helps to > know when the last change happened. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9873) Mutation API Config Change need to update Version Number
[ https://issues.apache.org/jira/browse/YARN-9873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947526#comment-16947526 ] Prabhu Joseph commented on YARN-9873: - Thanks [~sunilg]. > Mutation API Config Change need to update Version Number > - > > Key: YARN-9873 > URL: https://issues.apache.org/jira/browse/YARN-9873 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Fix For: 3.3.0, 3.2.2 > > Attachments: YARN-9873-001.patch, YARN-9873-002.patch, > YARN-9873-003.patch, YARN-9873-004.patch > > > Version Number support for each Scheduler Config Change. This also helps to > know when the last change happened. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9873) Mutation API Config Change need to update Version Number
[ https://issues.apache.org/jira/browse/YARN-9873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947536#comment-16947536 ] Hudson commented on YARN-9873: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17508 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17508/]) YARN-9873. Mutation API Config Change need to update Version Number. (sunilg: rev be901f49628a553cfe6a3942478cb528cce2b266) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/FSSchedulerConfigurationStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/MutableCSConfigurationProvider.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/MutableConfigurationProvider.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWSConsts.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ConfigVersionInfo.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesConfigurationMutation.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/InMemoryConfigurationStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/YarnConfigurationStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestFSSchedulerConfigurationStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/LeveldbConfigurationStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestZKConfigurationStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/ZKConfigurationStore.java > Mutation API Config Change need to update Version Number > - > > Key: YARN-9873 > URL: https://issues.apache.org/jira/browse/YARN-9873 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Fix For: 3.3.0, 3.2.2 > > Attachments: YARN-9873-001.patch, YARN-9873-002.patch, > YARN-9873-003.patch, YARN-9873-004.patch > > > Version Number support for each Scheduler Config Change. This also helps to > know when the last change happened. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9882) QueueMetrics not coming in Capacity Scheduler with Node Label Configuration
Gaurav Suman created YARN-9882: -- Summary: QueueMetrics not coming in Capacity Scheduler with Node Label Configuration Key: YARN-9882 URL: https://issues.apache.org/jira/browse/YARN-9882 Project: Hadoop YARN Issue Type: Bug Components: capacity scheduler, metrics, scheduler Reporter: Gaurav Suman I am having a capacity scheduler setup with two queues - "low-priority", "regular-priority". There are two node-labels "low" and "regular". low-priority queue has 100% access to "low" node-label and regular-priority queue has 100% access to "regular" node label. The yarn ui capacity scheduler configuration - [https://i.stack.imgur.com/gOARn.png] When i see the QueueMetrics emitted by queue "low-priority" and "regular-priority" in (http://rm-ip:port/jmx), then it shows correct values of availableMB and availableVCores, pendingMB=0 etc. but when I submit a job to any queue, there is no update in jmx metrics like pendingMB, pendingVcores, availableMB, availableVCores etc. only AppsRunning, ActiveApplications etc. are getting updated. The pendingMB, pendingVcores remains always 0 and there is no changes in availableMB, availableVcores, appsRunning and activeApplications shows correct value as 1. Not able to find why the metrics is not getting updated after job submission. The issue comes only when node-label is enabled. When node-label is disabled and only queue is used everything works fine. The capacity scheduler configuration(capacity-scheduler.xml): {code:java} yarn.scheduler.capacity.maximum-applications 5000 yarn.scheduler.capacity.maximum-am-resource-percent 0.2 yarn.scheduler.capacity.resource-calculator org.apache.hadoop.yarn.util.resource.DominantResourceCalculator The ResourceCalculator implementation to be used to compare Resources in the scheduler. The default i.e. DefaultResourceCalculator only uses Memory while DominantResourceCalculator uses dominant-resource to compare multi-dimensional resources such as Memory, CPU etc. yarn.scheduler.capacity.root.queues low-priority,regular-priority yarn.scheduler.capacity.root.capacity 100 yarn.scheduler.capacity.root.maximum-capacity 100 yarn.scheduler.capacity.root.accessible-node-labels * yarn.scheduler.capacity.root.accessible-node-labels.regular.capacity 100 yarn.scheduler.capacity.root.accessible-node-labels.regular.maximum-capacity 100 yarn.scheduler.capacity.root.accessible-node-labels.low.capacity 100 yarn.scheduler.capacity.root.accessible-node-labels.low.maximum-capacity 100 yarn.scheduler.capacity.root.default.state RUNNING The state of the default queue. State can be one of RUNNING or STOPPED. yarn.scheduler.capacity.root.default.acl_submit_applications * yarn.scheduler.capacity.root.default.acl_administer_queue * The ACL of who can administer jobs on the default queue. yarn.scheduler.capacity.node-locality-delay 40 yarn.scheduler.capacity.queue-mappings-override.enable false yarn.scheduler.capacity.root.low-priority.capacity 50 yarn.scheduler.capacity.root.low-priority.maximum-capacity 100 yarn.scheduler.capacity.root.low-priority.ordering-policy fair yarn.scheduler.capacity.root.low-priority.accessible-node-labels low yarn.scheduler.capacity.root.low-priority.default-node-label-expression low yarn.scheduler.capacity.root.low-priority.accessible-node-labels.low.capacity 100 yarn.scheduler.capacity.root.low-priority.accessible-node-labels.low.maximum-capacity 100 yarn.scheduler.capacity.root.low-priority.default.state RUNNING yarn.scheduler.capacity.root.regular-priority.capacity 50 yarn.scheduler.capacity.root.regular-priority.maximum-capacity 100 yarn.scheduler.capacity.root.regular-priority.ordering-policy fair yarn.scheduler.capacity.root.regular-priority.accessible-node-labels regular yarn.scheduler.capacity.root.regular-priority.default-node-label-expression regular yarn.scheduler.capacity.root.regular-priority.accessible-node-labels.regular.capacity 100 yarn.scheduler.capacity.root.regular-priority.accessible-node-labels.regular.maximum-capacity 100 yarn.scheduler.capacity.root.regular-priority.default.state RUNNING {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9693) When AMRMProxyService is enabled RMCommunicator will register with failure
[ https://issues.apache.org/jira/browse/YARN-9693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhoukang reassigned YARN-9693: -- Assignee: zhoukang > When AMRMProxyService is enabled RMCommunicator will register with failure > -- > > Key: YARN-9693 > URL: https://issues.apache.org/jira/browse/YARN-9693 > Project: Hadoop YARN > Issue Type: Improvement > Components: federation >Affects Versions: 3.1.2 >Reporter: zhoukang >Assignee: zhoukang >Priority: Major > > When we enable amrm proxy service, the RMCommunicator will register with > failure below: > {code:java} > 2019-07-23 17:12:44,794 INFO [TaskHeartbeatHandler PingChecker] > org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler > thread interrupted > 2019-07-23 17:12:44,794 ERROR [main] > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster > org.apache.hadoop.yarn.exceptions.YarnRuntimeException: > org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid > AMRMToken from appattempt_1563872237585_0001_02 > at > org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:186) > at > org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:123) > at > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:280) > at > org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.serviceStart(MRAppMaster.java:986) > at > org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) > at > org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1300) > at > org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(MRAppMaster.java:1768) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1716) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1764) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1698) > Caused by: org.apache.hadoop.security.token.SecretManager$InvalidToken: > Invalid AMRMToken from appattempt_1563872237585_0001_02 > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) > at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80) > at > org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) > at com.sun.proxy.$Proxy93.registerApplicationMaster(Unknown Source) > at > org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:170) > ... 14 more > Caused by: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): > Invalid AMRMToken from appattempt_1563872237585_0001_02 > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1541) > at org.apache.hadoo
[jira] [Comment Edited] (YARN-9693) When AMRMProxyService is enabled RMCommunicator will register with failure
[ https://issues.apache.org/jira/browse/YARN-9693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947566#comment-16947566 ] zhoukang edited comment on YARN-9693 at 10/9/19 11:10 AM: -- I will work on this.Patch on the way was (Author: cane): I will work on this.Path on the way > When AMRMProxyService is enabled RMCommunicator will register with failure > -- > > Key: YARN-9693 > URL: https://issues.apache.org/jira/browse/YARN-9693 > Project: Hadoop YARN > Issue Type: Improvement > Components: federation >Affects Versions: 3.1.2 >Reporter: zhoukang >Assignee: zhoukang >Priority: Major > > When we enable amrm proxy service, the RMCommunicator will register with > failure below: > {code:java} > 2019-07-23 17:12:44,794 INFO [TaskHeartbeatHandler PingChecker] > org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler > thread interrupted > 2019-07-23 17:12:44,794 ERROR [main] > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster > org.apache.hadoop.yarn.exceptions.YarnRuntimeException: > org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid > AMRMToken from appattempt_1563872237585_0001_02 > at > org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:186) > at > org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:123) > at > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:280) > at > org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.serviceStart(MRAppMaster.java:986) > at > org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) > at > org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1300) > at > org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(MRAppMaster.java:1768) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1716) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1764) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1698) > Caused by: org.apache.hadoop.security.token.SecretManager$InvalidToken: > Invalid AMRMToken from appattempt_1563872237585_0001_02 > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) > at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80) > at > org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) > at com.sun.proxy.$Proxy93.registerApplicationMaster(Unknown Source) > at > org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:170) > ... 14 more > Caused by: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$Inva
[jira] [Commented] (YARN-9693) When AMRMProxyService is enabled RMCommunicator will register with failure
[ https://issues.apache.org/jira/browse/YARN-9693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947566#comment-16947566 ] zhoukang commented on YARN-9693: I will work on this.Path on the way > When AMRMProxyService is enabled RMCommunicator will register with failure > -- > > Key: YARN-9693 > URL: https://issues.apache.org/jira/browse/YARN-9693 > Project: Hadoop YARN > Issue Type: Improvement > Components: federation >Affects Versions: 3.1.2 >Reporter: zhoukang >Assignee: zhoukang >Priority: Major > > When we enable amrm proxy service, the RMCommunicator will register with > failure below: > {code:java} > 2019-07-23 17:12:44,794 INFO [TaskHeartbeatHandler PingChecker] > org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler > thread interrupted > 2019-07-23 17:12:44,794 ERROR [main] > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster > org.apache.hadoop.yarn.exceptions.YarnRuntimeException: > org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid > AMRMToken from appattempt_1563872237585_0001_02 > at > org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:186) > at > org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:123) > at > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:280) > at > org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.serviceStart(MRAppMaster.java:986) > at > org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) > at > org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1300) > at > org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(MRAppMaster.java:1768) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1716) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1764) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1698) > Caused by: org.apache.hadoop.security.token.SecretManager$InvalidToken: > Invalid AMRMToken from appattempt_1563872237585_0001_02 > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) > at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80) > at > org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) > at com.sun.proxy.$Proxy93.registerApplicationMaster(Unknown Source) > at > org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:170) > ... 14 more > Caused by: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): > Invalid AMRMToken from appattempt_1563872237585_0001_02 > at org.apache.hadoop.ipc.Cli
[jira] [Commented] (YARN-9356) Add more tests to ratio method in TestResourceCalculator
[ https://issues.apache.org/jira/browse/YARN-9356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947571#comment-16947571 ] Szilard Nemeth commented on YARN-9356: -- Hi [~zsiegl]! Thanks for the patch! Commited to: trunk, branch-3.2, branch-3.1. Thanks [~adam.antal] for the review! > Add more tests to ratio method in TestResourceCalculator > - > > Key: YARN-9356 > URL: https://issues.apache.org/jira/browse/YARN-9356 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Zoltan Siegl >Priority: Major > Attachments: YARN-9356.001.patch, YARN-9356.001.patch > > > TestResourceCalculator has some edge-case testcases to verify how division by > zero is handled with ResourceCalculator. > We need other basic tests like we have for other ResourceCalculator methods. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9699) Migration tool that help to generate CS config based on FS config (Phase 1)
[ https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947577#comment-16947577 ] Hadoop QA commented on YARN-9699: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 41s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 16 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 32s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 56s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 21s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 19s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 15m 26s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 17s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 0s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 56s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 16s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 16s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 1m 18s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 55 new + 137 unchanged - 7 fixed = 192 total (was 144) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 15s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} xml {color} | {color:red} 0m 9s{color} | {color:red} The patch has 2 ill-formed XML file(s). {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 6s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 27s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 84m 41s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 22s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 39s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}159m 35s{color} | {color:black} {color} | \\ \\ ||
[jira] [Updated] (YARN-9128) Use SerializationUtils from apache commons to serialize / deserialize ResourceMappings
[ https://issues.apache.org/jira/browse/YARN-9128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-9128: - Fix Version/s: 3.3.0 > Use SerializationUtils from apache commons to serialize / deserialize > ResourceMappings > -- > > Key: YARN-9128 > URL: https://issues.apache.org/jira/browse/YARN-9128 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Zoltan Siegl >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9128.001.patch, YARN-9128.002.patch, > YARN-9128.003.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9128) Use SerializationUtils from apache commons to serialize / deserialize ResourceMappings
[ https://issues.apache.org/jira/browse/YARN-9128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947581#comment-16947581 ] Szilard Nemeth commented on YARN-9128: -- Hi [~zsiegl]! Thanks for this patch, pushed to trunk / branch-3.2 / branch-3.1! > Use SerializationUtils from apache commons to serialize / deserialize > ResourceMappings > -- > > Key: YARN-9128 > URL: https://issues.apache.org/jira/browse/YARN-9128 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Zoltan Siegl >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9128.001.patch, YARN-9128.002.patch, > YARN-9128.003.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9356) Add more tests to ratio method in TestResourceCalculator
[ https://issues.apache.org/jira/browse/YARN-9356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947582#comment-16947582 ] Hudson commented on YARN-9356: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17509 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17509/]) YARN-9356. Add more tests to ratio method in TestResourceCalculator. (snemeth: rev 35f093f5b3f310af49a8bb476e62173862f5af51) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/resource/TestResourceCalculator.java > Add more tests to ratio method in TestResourceCalculator > - > > Key: YARN-9356 > URL: https://issues.apache.org/jira/browse/YARN-9356 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Zoltan Siegl >Priority: Major > Fix For: 3.3.0, 3.2.2, 3.1.4 > > Attachments: YARN-9356.001.patch, YARN-9356.001.patch > > > TestResourceCalculator has some edge-case testcases to verify how division by > zero is handled with ResourceCalculator. > We need other basic tests like we have for other ResourceCalculator methods. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9541) TestCombinedSystemMetricsPublisher and TestSystemMetricsPublisherForV2 fail intermittent
[ https://issues.apache.org/jira/browse/YARN-9541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Siegl updated YARN-9541: --- Summary: TestCombinedSystemMetricsPublisher and TestSystemMetricsPublisherForV2 fail intermittent (was: TestCombinedSystemMetricsPublisher fails intermittent) > TestCombinedSystemMetricsPublisher and TestSystemMetricsPublisherForV2 fail > intermittent > > > Key: YARN-9541 > URL: https://issues.apache.org/jira/browse/YARN-9541 > Project: Hadoop YARN > Issue Type: Bug > Components: ATSv2 >Affects Versions: 3.2.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Minor > > org.apache.hadoop.yarn.server.resourcemanager.metrics.TestCombinedSystemMetricsPublisher.testTimelineServiceEventPublishingV1V2Enabled > {code} > Failing for the past 1 build (Since Failed#24071 ) > Took 0.19 sec. > Error Message > java.net.BindException: Problem binding to [0.0.0.0:10200] > java.net.BindException: Address already in use; For more details see: > http://wiki.apache.org/hadoop/BindException > Stacktrace > org.apache.hadoop.yarn.exceptions.YarnRuntimeException: > java.net.BindException: Problem binding to [0.0.0.0:10200] > java.net.BindException: Address already in use; For more details see: > http://wiki.apache.org/hadoop/BindException > at > org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.getServer(RpcServerFactoryPBImpl.java:139) > at > org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC.getServer(HadoopYarnProtoRPC.java:66) > at org.apache.hadoop.yarn.ipc.YarnRPC.getServer(YarnRPC.java:55) > at > org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryClientService.serviceStart(ApplicationHistoryClientService.java:94) > at > org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) > at > org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121) > at > org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceStart(ApplicationHistoryServer.java:120) > at > org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) > at > org.apache.hadoop.yarn.server.resourcemanager.metrics.TestCombinedSystemMetricsPublisher.testSetup(TestCombinedSystemMetricsPublisher.java:123) > at > org.apache.hadoop.yarn.server.resourcemanager.metrics.TestCombinedSystemMetricsPublisher.runTest(TestCombinedSystemMetricsPublisher.java:242) > at > org.apache.hadoop.yarn.server.resourcemanager.metrics.TestCombinedSystemMetricsPublisher.testTimelineServiceEventPublishingV1V2Enabled(TestCombinedSystemMetricsPublisher.java:252) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.net.BindException: Problem binding to [0.0.0.0:10200] > java.net.BindException: Address already in use; For more details see: > http://wiki.apache.org/hadoop/BindException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:833) > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:738) > at org.apache.hadoop.ipc.Server.bind(Server.java:599) > at org.apache.hadoop.ipc.Server$Listener.(Server.java:1121) > at org.apache.hadoop.ipc.Server.(Server.java:2976) > at org.apache.hadoop.ipc.RPC$Server.(RPC.java:1039) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Se
[jira] [Commented] (YARN-9541) TestCombinedSystemMetricsPublisher and TestSystemMetricsPublisherForV2 fail intermittent
[ https://issues.apache.org/jira/browse/YARN-9541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947583#comment-16947583 ] Zoltan Siegl commented on YARN-9541: TestCombinedSystemMetricsPublisher.testTimelineServiceEventPublishingV2Enabled TestCombinedSystemMetricsPublisher.testTimelineServiceEventPublishingV1V2Enabled TestSystemMetricsPublisherForV2.testPublishApplicationMetrics TestSystemMetricsPublisherForV2.testPublishAppAttemptMetrics TestSystemMetricsPublisherForV2.testPublishContainerMetrics These fail together consistently, and seemingly for the same route cause. > TestCombinedSystemMetricsPublisher and TestSystemMetricsPublisherForV2 fail > intermittent > > > Key: YARN-9541 > URL: https://issues.apache.org/jira/browse/YARN-9541 > Project: Hadoop YARN > Issue Type: Bug > Components: ATSv2 >Affects Versions: 3.2.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Minor > > org.apache.hadoop.yarn.server.resourcemanager.metrics.TestCombinedSystemMetricsPublisher.testTimelineServiceEventPublishingV1V2Enabled > {code} > Failing for the past 1 build (Since Failed#24071 ) > Took 0.19 sec. > Error Message > java.net.BindException: Problem binding to [0.0.0.0:10200] > java.net.BindException: Address already in use; For more details see: > http://wiki.apache.org/hadoop/BindException > Stacktrace > org.apache.hadoop.yarn.exceptions.YarnRuntimeException: > java.net.BindException: Problem binding to [0.0.0.0:10200] > java.net.BindException: Address already in use; For more details see: > http://wiki.apache.org/hadoop/BindException > at > org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.getServer(RpcServerFactoryPBImpl.java:139) > at > org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC.getServer(HadoopYarnProtoRPC.java:66) > at org.apache.hadoop.yarn.ipc.YarnRPC.getServer(YarnRPC.java:55) > at > org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryClientService.serviceStart(ApplicationHistoryClientService.java:94) > at > org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) > at > org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121) > at > org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceStart(ApplicationHistoryServer.java:120) > at > org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) > at > org.apache.hadoop.yarn.server.resourcemanager.metrics.TestCombinedSystemMetricsPublisher.testSetup(TestCombinedSystemMetricsPublisher.java:123) > at > org.apache.hadoop.yarn.server.resourcemanager.metrics.TestCombinedSystemMetricsPublisher.runTest(TestCombinedSystemMetricsPublisher.java:242) > at > org.apache.hadoop.yarn.server.resourcemanager.metrics.TestCombinedSystemMetricsPublisher.testTimelineServiceEventPublishingV1V2Enabled(TestCombinedSystemMetricsPublisher.java:252) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.net.BindException: Problem binding to [0.0.0.0:10200] > java.net.BindException: Address already in use; For more details see: > http://wiki.apache.org/hadoop/BindException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:833) > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:7
[jira] [Commented] (YARN-5106) Provide a builder interface for FairScheduler allocations for use in tests
[ https://issues.apache.org/jira/browse/YARN-5106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947590#comment-16947590 ] Szilard Nemeth commented on YARN-5106: -- Test failures are unrelated, see [~zsiegl]'s [comment|https://issues.apache.org/jira/browse/YARN-9541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947583#comment-16947583] > Provide a builder interface for FairScheduler allocations for use in tests > -- > > Key: YARN-5106 > URL: https://issues.apache.org/jira/browse/YARN-5106 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Zoltan Siegl >Priority: Major > Labels: newbie++ > Attachments: YARN-5106-branch-3.1.001.patch, > YARN-5106-branch-3.1.001.patch, YARN-5106-branch-3.1.001.patch, > YARN-5106-branch-3.1.002.patch, YARN-5106-branch-3.2.001.patch, > YARN-5106-branch-3.2.001.patch, YARN-5106-branch-3.2.002.patch, > YARN-5106.001.patch, YARN-5106.002.patch, YARN-5106.003.patch, > YARN-5106.004.patch, YARN-5106.005.patch, YARN-5106.006.patch, > YARN-5106.007.patch, YARN-5106.008.patch, YARN-5106.008.patch, > YARN-5106.008.patch, YARN-5106.009.patch, YARN-5106.010.patch, > YARN-5106.011.patch > > > Most, if not all, fair scheduler tests create an allocations XML file. Having > a helper class that potentially uses a builder would make the tests cleaner. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9699) Migration tool that help to generate CS config based on FS config (Phase 1)
[ https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947591#comment-16947591 ] Hadoop QA commented on YARN-9699: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 12m 30s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 16 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 58s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 35s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 25s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 23s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 34s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 56s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 21s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 16s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 55s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 39s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 1m 18s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 53 new + 136 unchanged - 7 fixed = 189 total (was 143) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 26s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} xml {color} | {color:red} 0m 10s{color} | {color:red} The patch has 2 ill-formed XML file(s). {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 11s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 29s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 81m 39s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 28s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 50s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}167m 18s{color} | {color:black} {color} | \\ \\ || Reason |
[jira] [Commented] (YARN-9128) Use SerializationUtils from apache commons to serialize / deserialize ResourceMappings
[ https://issues.apache.org/jira/browse/YARN-9128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947593#comment-16947593 ] Hudson commented on YARN-9128: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17510 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17510/]) YARN-9128. Use SerializationUtils from apache commons to serialize / (snemeth: rev 6f1ab9516869dbfdee034f4a17bccc379a39fc31) * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/TestResourceMappings.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ResourceMappings.java > Use SerializationUtils from apache commons to serialize / deserialize > ResourceMappings > -- > > Key: YARN-9128 > URL: https://issues.apache.org/jira/browse/YARN-9128 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Zoltan Siegl >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9128.001.patch, YARN-9128.002.patch, > YARN-9128.003.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5106) Provide a builder interface for FairScheduler allocations for use in tests
[ https://issues.apache.org/jira/browse/YARN-5106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947594#comment-16947594 ] Szilard Nemeth commented on YARN-5106: -- Hi [~zsiegl]! Just noticed that you have a lot of checkstyle failures for all patches. Could you please check these? [~pbacsko]: Do you have some time to review the patch? Thanks! > Provide a builder interface for FairScheduler allocations for use in tests > -- > > Key: YARN-5106 > URL: https://issues.apache.org/jira/browse/YARN-5106 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Zoltan Siegl >Priority: Major > Labels: newbie++ > Attachments: YARN-5106-branch-3.1.001.patch, > YARN-5106-branch-3.1.001.patch, YARN-5106-branch-3.1.001.patch, > YARN-5106-branch-3.1.002.patch, YARN-5106-branch-3.2.001.patch, > YARN-5106-branch-3.2.001.patch, YARN-5106-branch-3.2.002.patch, > YARN-5106.001.patch, YARN-5106.002.patch, YARN-5106.003.patch, > YARN-5106.004.patch, YARN-5106.005.patch, YARN-5106.006.patch, > YARN-5106.007.patch, YARN-5106.008.patch, YARN-5106.008.patch, > YARN-5106.008.patch, YARN-5106.009.patch, YARN-5106.010.patch, > YARN-5106.011.patch > > > Most, if not all, fair scheduler tests create an allocations XML file. Having > a helper class that potentially uses a builder would make the tests cleaner. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5106) Provide a builder interface for FairScheduler allocations for use in tests
[ https://issues.apache.org/jira/browse/YARN-5106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947596#comment-16947596 ] Zoltan Siegl commented on YARN-5106: Unit test failures are unrelated, they are constantly failing on branch-3.2 as descibed in https://issues.apache.org/jira/browse/YARN-9541 > Provide a builder interface for FairScheduler allocations for use in tests > -- > > Key: YARN-5106 > URL: https://issues.apache.org/jira/browse/YARN-5106 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Zoltan Siegl >Priority: Major > Labels: newbie++ > Attachments: YARN-5106-branch-3.1.001.patch, > YARN-5106-branch-3.1.001.patch, YARN-5106-branch-3.1.001.patch, > YARN-5106-branch-3.1.002.patch, YARN-5106-branch-3.2.001.patch, > YARN-5106-branch-3.2.001.patch, YARN-5106-branch-3.2.002.patch, > YARN-5106.001.patch, YARN-5106.002.patch, YARN-5106.003.patch, > YARN-5106.004.patch, YARN-5106.005.patch, YARN-5106.006.patch, > YARN-5106.007.patch, YARN-5106.008.patch, YARN-5106.008.patch, > YARN-5106.008.patch, YARN-5106.009.patch, YARN-5106.010.patch, > YARN-5106.011.patch > > > Most, if not all, fair scheduler tests create an allocations XML file. Having > a helper class that potentially uses a builder would make the tests cleaner. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9552) FairScheduler: NODE_UPDATE can cause NoSuchElementException
[ https://issues.apache.org/jira/browse/YARN-9552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947608#comment-16947608 ] Szilard Nemeth commented on YARN-9552: -- Hi [~pbacsko]! Committed your patches to branch-3.2 and branch-3.1 as well! Thanks! > FairScheduler: NODE_UPDATE can cause NoSuchElementException > --- > > Key: YARN-9552 > URL: https://issues.apache.org/jira/browse/YARN-9552 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9552-001.patch, YARN-9552-002.patch, > YARN-9552-003.patch, YARN-9552-004.patch, YARN-9552-branch-3.1.001.patch, > YARN-9552-branch-3.1.002.patch, YARN-9552-branch-3.2.001.patch, > YARN-9552-branch-3.2.002.patch, YARN-9552-branch-3.2.003.patch > > > We observed a race condition inside YARN with the following stack trace: > {noformat} > 18/11/07 06:45:09.559 SchedulerEventDispatcher:Event Processor ERROR > EventDispatcher: Error in handling event type NODE_UPDATE to the Event > Dispatcher > java.util.NoSuchElementException > at > java.util.concurrent.ConcurrentSkipListMap.firstKey(ConcurrentSkipListMap.java:2036) > at > java.util.concurrent.ConcurrentSkipListSet.first(ConcurrentSkipListSet.java:396) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.getNextPendingAsk(AppSchedulingInfo.java:373) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt.isOverAMShareLimit(FSAppAttempt.java:941) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt.assignContainer(FSAppAttempt.java:1373) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.assignContainer(FSLeafQueue.java:353) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.assignContainer(FSParentQueue.java:204) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.attemptScheduling(FairScheduler.java:1094) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.nodeUpdate(FairScheduler.java:961) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1183) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:132) > at > org.apache.hadoop.yarn.event.EventDispatcher$EventProcessor.run(EventDispatcher.java:66) > at java.lang.Thread.run(Thread.java:748) > {noformat} > This is basically the same as the one described in YARN-7382, but the root > cause is different. > When we create an application attempt, we create an {{FSAppAttempt}} object. > This contains an {{AppSchedulingInfo}} which contains a set of > {{SchedulerRequestKey}}. Initially, this set is empty and only initialized a > bit later on a separate thread during a state transition: > {noformat} > 2019-05-07 15:58:02,659 INFO [RM StateStore dispatcher] > recovery.RMStateStore (RMStateStore.java:transition(239)) - Storing info for > app: application_1557237478804_0001 > 2019-05-07 15:58:02,684 INFO [RM Event dispatcher] rmapp.RMAppImpl > (RMAppImpl.java:handle(903)) - application_1557237478804_0001 State change > from NEW_SAVING to SUBMITTED on event = APP_NEW_SAVED > 2019-05-07 15:58:02,690 INFO [SchedulerEventDispatcher:Event Processor] > fair.FairScheduler (FairScheduler.java:addApplication(490)) - Accepted > application application_1557237478804_0001 from user: bacskop, in queue: > root.bacskop, currently num of applications: 1 > 2019-05-07 15:58:02,698 INFO [RM Event dispatcher] rmapp.RMAppImpl > (RMAppImpl.java:handle(903)) - application_1557237478804_0001 State change > from SUBMITTED to ACCEPTED on event = APP_ACCEPTED > 2019-05-07 15:58:02,731 INFO [RM Event dispatcher] > resourcemanager.ApplicationMasterService > (ApplicationMasterService.java:registerAppAttempt(434)) - Registering app > attempt : appattempt_1557237478804_0001_01 > 2019-05-07 15:58:02,732 INFO [RM Event dispatcher] attempt.RMAppAttemptImpl > (RMAppAttemptImpl.java:handle(920)) - appattempt_1557237478804_0001_01 > State change from NEW to SUBMITTED on event = START > 2019-05-07 15:58:02,746 INFO [SchedulerEventDispatcher:Event Processor] > scheduler.SchedulerApplicationAttempt > (SchedulerApplicationAttempt.java:(207)) - *** In the constructor of > SchedulerApplicationAttempt > 2019-05-07 15:58:02,747 INFO [SchedulerEventDispatcher:Event Processor] > scheduler.SchedulerApplicationAttempt > (SchedulerApplicationAttempt.java:(230)) - *** Contents of > appSchedulin
[jira] [Created] (YARN-9883) Reshape SchedulerHealth class
Adam Antal created YARN-9883: Summary: Reshape SchedulerHealth class Key: YARN-9883 URL: https://issues.apache.org/jira/browse/YARN-9883 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager, yarn Affects Versions: 3.3.0 Reporter: Adam Antal The {{SchedulerHealth}} class has some flaws, for example: - It has no javadoc at all - All its objects are package-private: they should be private - The internal maps should be (Concurrent) EnumMaps instead of HashMaps: they are more efficient in storing Enums - schedulerHealthDetails only stores the last operation, its name should reflect that (just like lastSchedulerRunDetails) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9883) Reshape SchedulerHealth class
[ https://issues.apache.org/jira/browse/YARN-9883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton reassigned YARN-9883: -- Assignee: Kinga Marton > Reshape SchedulerHealth class > - > > Key: YARN-9883 > URL: https://issues.apache.org/jira/browse/YARN-9883 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, yarn >Affects Versions: 3.3.0 >Reporter: Adam Antal >Assignee: Kinga Marton >Priority: Minor > > The {{SchedulerHealth}} class has some flaws, for example: > - It has no javadoc at all > - All its objects are package-private: they should be private > - The internal maps should be (Concurrent) EnumMaps instead of HashMaps: they > are more efficient in storing Enums > - schedulerHealthDetails only stores the last operation, its name should > reflect that (just like lastSchedulerRunDetails) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9699) Migration tool that help to generate CS config based on FS config (Phase 1)
[ https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-9699: --- Attachment: YARN-9699-011.patch > Migration tool that help to generate CS config based on FS config (Phase 1) > --- > > Key: YARN-9699 > URL: https://issues.apache.org/jira/browse/YARN-9699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wanqiang Ji >Assignee: Peter Bacsko >Priority: Major > Attachments: FS_to_CS_migration_POC.patch, YARN-9699-003.patch, > YARN-9699-004.patch, YARN-9699-005.patch, YARN-9699-006.patch, > YARN-9699-007.patch, YARN-9699-008.patch, YARN-9699-009.patch, > YARN-9699-010.patch, YARN-9699-011.patch, YARN-9699.001.patch, > YARN-9699.002.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3890) FairScheduler should show the scheduler health metrics similar to ones added in CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-3890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947624#comment-16947624 ] Adam Antal commented on YARN-3890: -- Overall looks very nice! Thanks for working on this. Some comments: - I see that {{SchedulerHealth$updateSchedulerReleaseDetails()}} is not called by the FS after your patch. I think reporting those has also major importance. Would you wire that as well to FS? - I am not entirely convinced that testing is sufficient, so I suggest that TestSchedulerHealth class should be parameterized to use FairScheduler as well (along with CS) to ensure full integration. I'm also ok if you file a follow-up jira and resolve it in that. Question: Did you run it on a cluster, and verified that the HTML is rendered properly? (One should do that before the merge of this patch.) nits: - Remove TODO from {{FairScheduler.java}}. Was there anything you were supposed to put there? - In {{TestFSAppAttempt.java}} remove wildcard import - Decapitalize comments in {{TestFSAppAttempt$testHealthMetric}} - Checkstyle I also saw that {{SchedulerHealth}} class is in a really bad shape, I filed YARN-9883 for the cleanup. > FairScheduler should show the scheduler health metrics similar to ones added > in CapacityScheduler > - > > Key: YARN-3890 > URL: https://issues.apache.org/jira/browse/YARN-3890 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Reporter: Anubhav Dhoot >Assignee: Zoltan Siegl >Priority: Major > Attachments: YARN-3890.001.patch, YARN-3890.002.patch, > YARN-3890.003.patch > > > We should add information displayed in YARN-3293 in FairScheduler as well > possibly sharing the implementation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8453) Additional Unit tests to verify queue limit and max-limit with multiple resource types
[ https://issues.apache.org/jira/browse/YARN-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947628#comment-16947628 ] Szilard Nemeth commented on YARN-8453: -- Hi [~sunilg]! Couple of comments: # Could you rename static field A to QUEUE_A? Similarly with B -> QUEUE_B as well? # A_CAPACITY and B_CAPACITY could be private. # There are some calls to TestUtils.createResource: These can be fit into the previous line. # // Register NM1 with 10GB memory, 4 CPU and 0 GPU --> This comment should be about NM2. # nm2 variable is not used, please invoke registerNode, but don't assign it to a variable. # // Allocate one more containers which doesnt need GPU --> containers should be singular # There are 2 occasions where you use Arrays.asList with just one element: You could use Collections.singletonList() instead. # Can you extract some common code with testMaximumAllocationRefreshWithMultipleResourceTypes and your new test method? Thanks! > Additional Unit tests to verify queue limit and max-limit with multiple > resource types > --- > > Key: YARN-8453 > URL: https://issues.apache.org/jira/browse/YARN-8453 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Affects Versions: 3.0.2 >Reporter: Sunil G >Assignee: Sunil G >Priority: Major > Attachments: YARN-8453.001.patch > > > Post support of additional resource types other then CPU and Memory, it could > be possible that one such new resource is exhausted its quota on a given > queue. But other resources such as Memory / CPU is still there beyond its > guaranteed limit (under max-limit). Adding more units test to ensure we are > not starving such allocation requests -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8453) Additional Unit tests to verify queue limit and max-limit with multiple resource types
[ https://issues.apache.org/jira/browse/YARN-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947630#comment-16947630 ] Adam Antal commented on YARN-8453: -- If you don't have much time, I'd like to take this over [~sunilg]. > Additional Unit tests to verify queue limit and max-limit with multiple > resource types > --- > > Key: YARN-8453 > URL: https://issues.apache.org/jira/browse/YARN-8453 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Affects Versions: 3.0.2 >Reporter: Sunil G >Assignee: Sunil G >Priority: Major > Attachments: YARN-8453.001.patch > > > Post support of additional resource types other then CPU and Memory, it could > be possible that one such new resource is exhausted its quota on a given > queue. But other resources such as Memory / CPU is still there beyond its > guaranteed limit (under max-limit). Adding more units test to ensure we are > not starving such allocation requests -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9840) Capacity scheduler: add support for Secondary Group rule mapping
[ https://issues.apache.org/jira/browse/YARN-9840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947634#comment-16947634 ] Hadoop QA commented on YARN-9840: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 39s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 4s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 32s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 51s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 40s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 14s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 41s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 17s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 85m 38s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 28s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}143m 6s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=19.03.3 Server=19.03.3 Image:yetus/hadoop:1dde3efb91e | | JIRA Issue | YARN-9840 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12982508/YARN-9840.003.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 60c6e5cdbd74 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / be901f4 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_222 | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/24943/testReport/ | | Max. process+thread count | 833 (vs. ulimit of 5500) | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/24943/console | | Powered by | Apache Yetus 0.8.0 http://yetus.apache.org | This message was automatically generated. > Capacity scheduler: add support for S
[jira] [Commented] (YARN-9840) Capacity scheduler: add support for Secondary Group rule mapping
[ https://issues.apache.org/jira/browse/YARN-9840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947650#comment-16947650 ] Peter Bacsko commented on YARN-9840: +1 (non-binding). [~snemeth] please review & commit patch v3 if you think it's OK. > Capacity scheduler: add support for Secondary Group rule mapping > > > Key: YARN-9840 > URL: https://issues.apache.org/jira/browse/YARN-9840 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler >Reporter: Peter Bacsko >Assignee: Manikandan R >Priority: Major > Attachments: YARN-9840.001.patch, YARN-9840.002.patch, > YARN-9840.003.patch > > > Currently, Capacity Scheduler only supports primary group rule mapping like > this: > {{u:%user:%primary_group}} > Fair scheduler already supports secondary group placement rule. Let's add > this to CS to reduce the feature gap. > Class of interest: > https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9841) Capacity scheduler: add support for combined %user + %primary_group mapping
[ https://issues.apache.org/jira/browse/YARN-9841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947653#comment-16947653 ] Hadoop QA commented on YARN-9841: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m 2s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 3 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 19s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 35s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 46s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 38s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 14s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 29s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 9s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 16s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 84m 29s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 29s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}155m 0s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=19.03.3 Server=19.03.3 Image:yetus/hadoop:1dde3efb91e | | JIRA Issue | YARN-9841 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12982509/YARN-9841.003.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 3b2d970edcea 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / be901f4 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_222 | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/24944/testReport/ | | Max. process+thread count | 814 (vs. ulimit of 5500) | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/24944/console | | Powered by | Apache Yetus 0.8.0 http://yetus.apache.org | This message was automatically generated. > Capacity scheduler: add support for co
[jira] [Commented] (YARN-9841) Capacity scheduler: add support for combined %user + %primary_group mapping
[ https://issues.apache.org/jira/browse/YARN-9841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947654#comment-16947654 ] Peter Bacsko commented on YARN-9841: Thanks for the patch [~maniraj...@gmail.com]. I'm not sure whether this changeset conflicts with YARN-9840 so let's wait until that one gets committed. Just a minor thing related to tests: {noformat} Assert.assertEquals("Expected Queue is ", expectedQueue, ctx != null ? ctx.getQueue() : inputQueue); {noformat} You don't have to write "Expected Queue is ", because JUnit4 itself prints a similar message if there's an assertion failure. It's enough to write {{Assert.assertEquals("Queue", ...)}}. Could you pls modify this? Otherwise it looks good. > Capacity scheduler: add support for combined %user + %primary_group mapping > --- > > Key: YARN-9841 > URL: https://issues.apache.org/jira/browse/YARN-9841 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler >Reporter: Peter Bacsko >Assignee: Manikandan R >Priority: Major > Attachments: YARN-9841.001.patch, YARN-9841.001.patch, > YARN-9841.002.patch, YARN-9841.003.patch, YARN-9841.junit.patch > > > Right now in CS, using {{%primary_group}} with a parent queue is only > possible this way: > {{u:%user:parentqueue.%primary_group}} > Looking at > https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java, > we cannot do something like: > {{u:%user:%primary_group.%user}} > Fair Scheduler supports a nested rule where such a placement/mapping rule is > possible. This improvement would reduce this feature gap. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9841) Capacity scheduler: add support for combined %user + %primary_group mapping
[ https://issues.apache.org/jira/browse/YARN-9841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947702#comment-16947702 ] Manikandan R commented on YARN-9841: Thanks [~pbacsko]. Attached .004.patch. > Capacity scheduler: add support for combined %user + %primary_group mapping > --- > > Key: YARN-9841 > URL: https://issues.apache.org/jira/browse/YARN-9841 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler >Reporter: Peter Bacsko >Assignee: Manikandan R >Priority: Major > Attachments: YARN-9841.001.patch, YARN-9841.001.patch, > YARN-9841.002.patch, YARN-9841.003.patch, YARN-9841.004.patch, > YARN-9841.junit.patch > > > Right now in CS, using {{%primary_group}} with a parent queue is only > possible this way: > {{u:%user:parentqueue.%primary_group}} > Looking at > https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java, > we cannot do something like: > {{u:%user:%primary_group.%user}} > Fair Scheduler supports a nested rule where such a placement/mapping rule is > possible. This improvement would reduce this feature gap. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9841) Capacity scheduler: add support for combined %user + %primary_group mapping
[ https://issues.apache.org/jira/browse/YARN-9841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikandan R updated YARN-9841: --- Attachment: YARN-9841.004.patch > Capacity scheduler: add support for combined %user + %primary_group mapping > --- > > Key: YARN-9841 > URL: https://issues.apache.org/jira/browse/YARN-9841 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler >Reporter: Peter Bacsko >Assignee: Manikandan R >Priority: Major > Attachments: YARN-9841.001.patch, YARN-9841.001.patch, > YARN-9841.002.patch, YARN-9841.003.patch, YARN-9841.004.patch, > YARN-9841.junit.patch > > > Right now in CS, using {{%primary_group}} with a parent queue is only > possible this way: > {{u:%user:parentqueue.%primary_group}} > Looking at > https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java, > we cannot do something like: > {{u:%user:%primary_group.%user}} > Fair Scheduler supports a nested rule where such a placement/mapping rule is > possible. This improvement would reduce this feature gap. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9699) Migration tool that help to generate CS config based on FS config (Phase 1)
[ https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-9699: - Attachment: YARN-9699-012.patch > Migration tool that help to generate CS config based on FS config (Phase 1) > --- > > Key: YARN-9699 > URL: https://issues.apache.org/jira/browse/YARN-9699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wanqiang Ji >Assignee: Peter Bacsko >Priority: Major > Attachments: FS_to_CS_migration_POC.patch, YARN-9699-003.patch, > YARN-9699-004.patch, YARN-9699-005.patch, YARN-9699-006.patch, > YARN-9699-007.patch, YARN-9699-008.patch, YARN-9699-009.patch, > YARN-9699-010.patch, YARN-9699-011.patch, YARN-9699-012.patch, > YARN-9699.001.patch, YARN-9699.002.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9699) Migration tool that help to generate CS config based on FS config (Phase 1)
[ https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-9699: - Attachment: YARN-9699-013.patch > Migration tool that help to generate CS config based on FS config (Phase 1) > --- > > Key: YARN-9699 > URL: https://issues.apache.org/jira/browse/YARN-9699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wanqiang Ji >Assignee: Peter Bacsko >Priority: Major > Attachments: FS_to_CS_migration_POC.patch, YARN-9699-003.patch, > YARN-9699-004.patch, YARN-9699-005.patch, YARN-9699-006.patch, > YARN-9699-007.patch, YARN-9699-008.patch, YARN-9699-009.patch, > YARN-9699-010.patch, YARN-9699-011.patch, YARN-9699-012.patch, > YARN-9699-013.patch, YARN-9699.001.patch, YARN-9699.002.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-8453) Additional Unit tests to verify queue limit and max-limit with multiple resource types
[ https://issues.apache.org/jira/browse/YARN-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth reassigned YARN-8453: Assignee: Szilard Nemeth (was: Sunil G) > Additional Unit tests to verify queue limit and max-limit with multiple > resource types > --- > > Key: YARN-8453 > URL: https://issues.apache.org/jira/browse/YARN-8453 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Affects Versions: 3.0.2 >Reporter: Sunil G >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-8453.001.patch > > > Post support of additional resource types other then CPU and Memory, it could > be possible that one such new resource is exhausted its quota on a given > queue. But other resources such as Memory / CPU is still there beyond its > guaranteed limit (under max-limit). Adding more units test to ensure we are > not starving such allocation requests -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-8453) Additional Unit tests to verify queue limit and max-limit with multiple resource types
[ https://issues.apache.org/jira/browse/YARN-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth reassigned YARN-8453: Assignee: Adam Antal (was: Szilard Nemeth) > Additional Unit tests to verify queue limit and max-limit with multiple > resource types > --- > > Key: YARN-8453 > URL: https://issues.apache.org/jira/browse/YARN-8453 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Affects Versions: 3.0.2 >Reporter: Sunil G >Assignee: Adam Antal >Priority: Major > Attachments: YARN-8453.001.patch > > > Post support of additional resource types other then CPU and Memory, it could > be possible that one such new resource is exhausted its quota on a given > queue. But other resources such as Memory / CPU is still there beyond its > guaranteed limit (under max-limit). Adding more units test to ensure we are > not starving such allocation requests -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9699) Migration tool that help to generate CS config based on FS config (Phase 1)
[ https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947793#comment-16947793 ] Hadoop QA commented on YARN-9699: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 54s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 16 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 47s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 17s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 25s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 18s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 15m 21s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 15s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 1s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 55s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 15s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 15s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 1m 17s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 18 new + 137 unchanged - 7 fixed = 155 total (was 144) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 16s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} xml {color} | {color:red} 0m 10s{color} | {color:red} The patch has 2 ill-formed XML file(s). {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 7s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 25s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 87m 25s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 22s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} | | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 42s{color} | {color:red} The patch generated 1 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}167m 23s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | XML | Parsing Error(s): | | | hadoop-yarn-project/hadoop-yarn/hadoop-yar
[jira] [Commented] (YARN-9841) Capacity scheduler: add support for combined %user + %primary_group mapping
[ https://issues.apache.org/jira/browse/YARN-9841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947829#comment-16947829 ] Hadoop QA commented on YARN-9841: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 31s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 3 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 57s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 36s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 51s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 34s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 17s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 26s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 20s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 81m 31s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 32s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}133m 58s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption | \\ \\ || Subsystem || Report/Notes || | Docker | Client=19.03.3 Server=19.03.3 Image:yetus/hadoop:1dde3efb91e | | JIRA Issue | YARN-9841 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12982580/YARN-9841.004.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 68a0d5463fac 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 4b0a5bc | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_222 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/PreCommit-YARN-Build/24947/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/24947/testReport/ | | Max. process+thread count | 811 (vs. ulimit of 5500) | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/
[jira] [Commented] (YARN-9699) Migration tool that help to generate CS config based on FS config (Phase 1)
[ https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947843#comment-16947843 ] Hadoop QA commented on YARN-9699: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 46s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 16 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 52s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 24s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 18s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 24s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 20s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 15m 16s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 23s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 37s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 37s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 1m 22s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 18 new + 137 unchanged - 7 fixed = 155 total (was 144) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 14s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} xml {color} | {color:red} 0m 9s{color} | {color:red} The patch has 2 ill-formed XML file(s). {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 9s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 29s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 86m 25s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 22s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} | | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 40s{color} | {color:red} The patch generated 1 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}163m 58s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | XML | Parsing Error(s): | | | hadoop-yarn-project/hadoop-yarn/hadoop-yar
[jira] [Commented] (YARN-9699) Migration tool that help to generate CS config based on FS config (Phase 1)
[ https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947844#comment-16947844 ] Hadoop QA commented on YARN-9699: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 33s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 16 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 59s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 14s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 41s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 22s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 31s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 52s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 21s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 14s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 10s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 1m 19s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 18 new + 136 unchanged - 7 fixed = 154 total (was 143) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} xml {color} | {color:red} 0m 10s{color} | {color:red} The patch has 2 ill-formed XML file(s). {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 18s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 28s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 82m 4s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 27s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} | | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 50s{color} | {color:red} The patch generated 1 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}156m 57s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | XML | Parsing Error(s): | | | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/
[jira] [Commented] (YARN-9699) Migration tool that help to generate CS config based on FS config (Phase 1)
[ https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947855#comment-16947855 ] Hadoop QA commented on YARN-9699: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 26s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 16 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 44s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 51s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 20s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 24s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 33s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 59s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 20s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 16s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 5s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 18s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 18s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 1m 24s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 18 new + 136 unchanged - 7 fixed = 154 total (was 143) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} xml {color} | {color:red} 0m 10s{color} | {color:red} The patch has 2 ill-formed XML file(s). {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 18s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 28s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 12s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 81m 47s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 30s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} | | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 47s{color} | {color:red} The patch generated 1 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}156m 10s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | XML | Parsing Error(s): | | | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/
[jira] [Commented] (YARN-9552) FairScheduler: NODE_UPDATE can cause NoSuchElementException
[ https://issues.apache.org/jira/browse/YARN-9552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947857#comment-16947857 ] Steve Loughran commented on YARN-9552: -- trunk and branch-3.2 doesn't build; this looks suspiciously like the patch at fault. Reopening -can someone look @ this > FairScheduler: NODE_UPDATE can cause NoSuchElementException > --- > > Key: YARN-9552 > URL: https://issues.apache.org/jira/browse/YARN-9552 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Fix For: 3.3.0, 3.2.2, 3.1.4 > > Attachments: YARN-9552-001.patch, YARN-9552-002.patch, > YARN-9552-003.patch, YARN-9552-004.patch, YARN-9552-branch-3.1.001.patch, > YARN-9552-branch-3.1.002.patch, YARN-9552-branch-3.2.001.patch, > YARN-9552-branch-3.2.002.patch, YARN-9552-branch-3.2.003.patch > > > We observed a race condition inside YARN with the following stack trace: > {noformat} > 18/11/07 06:45:09.559 SchedulerEventDispatcher:Event Processor ERROR > EventDispatcher: Error in handling event type NODE_UPDATE to the Event > Dispatcher > java.util.NoSuchElementException > at > java.util.concurrent.ConcurrentSkipListMap.firstKey(ConcurrentSkipListMap.java:2036) > at > java.util.concurrent.ConcurrentSkipListSet.first(ConcurrentSkipListSet.java:396) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.getNextPendingAsk(AppSchedulingInfo.java:373) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt.isOverAMShareLimit(FSAppAttempt.java:941) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt.assignContainer(FSAppAttempt.java:1373) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.assignContainer(FSLeafQueue.java:353) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.assignContainer(FSParentQueue.java:204) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.attemptScheduling(FairScheduler.java:1094) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.nodeUpdate(FairScheduler.java:961) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1183) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:132) > at > org.apache.hadoop.yarn.event.EventDispatcher$EventProcessor.run(EventDispatcher.java:66) > at java.lang.Thread.run(Thread.java:748) > {noformat} > This is basically the same as the one described in YARN-7382, but the root > cause is different. > When we create an application attempt, we create an {{FSAppAttempt}} object. > This contains an {{AppSchedulingInfo}} which contains a set of > {{SchedulerRequestKey}}. Initially, this set is empty and only initialized a > bit later on a separate thread during a state transition: > {noformat} > 2019-05-07 15:58:02,659 INFO [RM StateStore dispatcher] > recovery.RMStateStore (RMStateStore.java:transition(239)) - Storing info for > app: application_1557237478804_0001 > 2019-05-07 15:58:02,684 INFO [RM Event dispatcher] rmapp.RMAppImpl > (RMAppImpl.java:handle(903)) - application_1557237478804_0001 State change > from NEW_SAVING to SUBMITTED on event = APP_NEW_SAVED > 2019-05-07 15:58:02,690 INFO [SchedulerEventDispatcher:Event Processor] > fair.FairScheduler (FairScheduler.java:addApplication(490)) - Accepted > application application_1557237478804_0001 from user: bacskop, in queue: > root.bacskop, currently num of applications: 1 > 2019-05-07 15:58:02,698 INFO [RM Event dispatcher] rmapp.RMAppImpl > (RMAppImpl.java:handle(903)) - application_1557237478804_0001 State change > from SUBMITTED to ACCEPTED on event = APP_ACCEPTED > 2019-05-07 15:58:02,731 INFO [RM Event dispatcher] > resourcemanager.ApplicationMasterService > (ApplicationMasterService.java:registerAppAttempt(434)) - Registering app > attempt : appattempt_1557237478804_0001_01 > 2019-05-07 15:58:02,732 INFO [RM Event dispatcher] attempt.RMAppAttemptImpl > (RMAppAttemptImpl.java:handle(920)) - appattempt_1557237478804_0001_01 > State change from NEW to SUBMITTED on event = START > 2019-05-07 15:58:02,746 INFO [SchedulerEventDispatcher:Event Processor] > scheduler.SchedulerApplicationAttempt > (SchedulerApplicationAttempt.java:(207)) - *** In the constructor of > SchedulerApplicationAttempt > 2019-05-07 15:58:02,747 INFO [SchedulerEventDispatcher:Event Processor] > scheduler.SchedulerApplicationAttempt > (SchedulerApplicationAtte
[jira] [Commented] (YARN-9552) FairScheduler: NODE_UPDATE can cause NoSuchElementException
[ https://issues.apache.org/jira/browse/YARN-9552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947862#comment-16947862 ] Peter Bacsko commented on YARN-9552: It's been on trunk since May, so it should be okay there. > FairScheduler: NODE_UPDATE can cause NoSuchElementException > --- > > Key: YARN-9552 > URL: https://issues.apache.org/jira/browse/YARN-9552 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Fix For: 3.3.0, 3.2.2, 3.1.4 > > Attachments: YARN-9552-001.patch, YARN-9552-002.patch, > YARN-9552-003.patch, YARN-9552-004.patch, YARN-9552-branch-3.1.001.patch, > YARN-9552-branch-3.1.002.patch, YARN-9552-branch-3.2.001.patch, > YARN-9552-branch-3.2.002.patch, YARN-9552-branch-3.2.003.patch > > > We observed a race condition inside YARN with the following stack trace: > {noformat} > 18/11/07 06:45:09.559 SchedulerEventDispatcher:Event Processor ERROR > EventDispatcher: Error in handling event type NODE_UPDATE to the Event > Dispatcher > java.util.NoSuchElementException > at > java.util.concurrent.ConcurrentSkipListMap.firstKey(ConcurrentSkipListMap.java:2036) > at > java.util.concurrent.ConcurrentSkipListSet.first(ConcurrentSkipListSet.java:396) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.getNextPendingAsk(AppSchedulingInfo.java:373) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt.isOverAMShareLimit(FSAppAttempt.java:941) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt.assignContainer(FSAppAttempt.java:1373) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.assignContainer(FSLeafQueue.java:353) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.assignContainer(FSParentQueue.java:204) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.attemptScheduling(FairScheduler.java:1094) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.nodeUpdate(FairScheduler.java:961) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1183) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:132) > at > org.apache.hadoop.yarn.event.EventDispatcher$EventProcessor.run(EventDispatcher.java:66) > at java.lang.Thread.run(Thread.java:748) > {noformat} > This is basically the same as the one described in YARN-7382, but the root > cause is different. > When we create an application attempt, we create an {{FSAppAttempt}} object. > This contains an {{AppSchedulingInfo}} which contains a set of > {{SchedulerRequestKey}}. Initially, this set is empty and only initialized a > bit later on a separate thread during a state transition: > {noformat} > 2019-05-07 15:58:02,659 INFO [RM StateStore dispatcher] > recovery.RMStateStore (RMStateStore.java:transition(239)) - Storing info for > app: application_1557237478804_0001 > 2019-05-07 15:58:02,684 INFO [RM Event dispatcher] rmapp.RMAppImpl > (RMAppImpl.java:handle(903)) - application_1557237478804_0001 State change > from NEW_SAVING to SUBMITTED on event = APP_NEW_SAVED > 2019-05-07 15:58:02,690 INFO [SchedulerEventDispatcher:Event Processor] > fair.FairScheduler (FairScheduler.java:addApplication(490)) - Accepted > application application_1557237478804_0001 from user: bacskop, in queue: > root.bacskop, currently num of applications: 1 > 2019-05-07 15:58:02,698 INFO [RM Event dispatcher] rmapp.RMAppImpl > (RMAppImpl.java:handle(903)) - application_1557237478804_0001 State change > from SUBMITTED to ACCEPTED on event = APP_ACCEPTED > 2019-05-07 15:58:02,731 INFO [RM Event dispatcher] > resourcemanager.ApplicationMasterService > (ApplicationMasterService.java:registerAppAttempt(434)) - Registering app > attempt : appattempt_1557237478804_0001_01 > 2019-05-07 15:58:02,732 INFO [RM Event dispatcher] attempt.RMAppAttemptImpl > (RMAppAttemptImpl.java:handle(920)) - appattempt_1557237478804_0001_01 > State change from NEW to SUBMITTED on event = START > 2019-05-07 15:58:02,746 INFO [SchedulerEventDispatcher:Event Processor] > scheduler.SchedulerApplicationAttempt > (SchedulerApplicationAttempt.java:(207)) - *** In the constructor of > SchedulerApplicationAttempt > 2019-05-07 15:58:02,747 INFO [SchedulerEventDispatcher:Event Processor] > scheduler.SchedulerApplicationAttempt > (SchedulerApplicationAttempt.java:(230)) - *** Contents of > appSchedulingInfo: [] > 2019-0
[jira] [Commented] (YARN-9128) Use SerializationUtils from apache commons to serialize / deserialize ResourceMappings
[ https://issues.apache.org/jira/browse/YARN-9128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947907#comment-16947907 ] Szilard Nemeth commented on YARN-9128: -- Hi [~zsiegl] ! Reverted YARN-9128 commits on branch-3.2 / branch-3.1 as it broke the build on these branches! The compilation failure was caused by that 3.2 / 3.1 don't have this class: org.apache.hadoop.yarn.server.nodemanager.api.deviceplugin.Device, which is being used in TestResourceMappings. [~zsiegl]: Please try to come up some other way to test AssignedResources#fromBytes / toBytes: Either with GpuDevice or FpgaDevice or any other artifical device object for tests. Please also assess how important to backport the change to branch-3.2 / branch-3.1. If you decice to backport it, please post the patches so we have green Jenkins results for both. Thanks! > Use SerializationUtils from apache commons to serialize / deserialize > ResourceMappings > -- > > Key: YARN-9128 > URL: https://issues.apache.org/jira/browse/YARN-9128 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Zoltan Siegl >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9128.001.patch, YARN-9128.002.patch, > YARN-9128.003.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9873) Mutation API Config Change need to update Version Number
[ https://issues.apache.org/jira/browse/YARN-9873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948129#comment-16948129 ] Akira Ajisaka commented on YARN-9873: - This broke branch-3.2 build. YARN-7243 changed the logging API from commons-logging to slf4j in trunk, but not in branch-3.2. I'll try to cherry-pick YARN-7243 to branch-3.2. > Mutation API Config Change need to update Version Number > - > > Key: YARN-9873 > URL: https://issues.apache.org/jira/browse/YARN-9873 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Fix For: 3.3.0, 3.2.2 > > Attachments: YARN-9873-001.patch, YARN-9873-002.patch, > YARN-9873-003.patch, YARN-9873-004.patch > > > Version Number support for each Scheduler Config Change. This also helps to > know when the last change happened. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7243) Moving logging APIs over to slf4j in hadoop-yarn-server-resourcemanager
[ https://issues.apache.org/jira/browse/YARN-7243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948144#comment-16948144 ] Akira Ajisaka commented on YARN-7243: - Created https://github.com/apache/hadoop/pull/1634 to backport this to branch-3.2. > Moving logging APIs over to slf4j in hadoop-yarn-server-resourcemanager > --- > > Key: YARN-7243 > URL: https://issues.apache.org/jira/browse/YARN-7243 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Yeliang Cang >Assignee: Prabhu Joseph >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-7243-007.patch, YARN-7243-008.patch, > YARN-7243-009.patch, YARN-7243.001.patch, YARN-7243.002.patch, > YARN-7243.003.patch, YARN-7243.004.patch, YARN-7243.005.patch, > YARN-7243.006.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9880) In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is incorrect.
[ https://issues.apache.org/jira/browse/YARN-9880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jenny updated YARN-9880: Description: In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is incorrect. When Browser and Yarn server are in different time zones, the Application Attempt's ElapsedTime is greater than the Application's ElapsedTime. I found this problem where hadoop3.1.1 was used, want to upgrade to the latest version of hadoop, but found the running Application Attempt's ElapsedTime is also incorrect. In UI1, get ElapsedTime from yarn server. In UI2, get currentTime from brower. Therefore, when the brower and the yarn server are in different time zones, the ElapsedTime is incorrect. While the Application‘s ElapsedTime is 29 Secs, the Application Attempt's ElapsedTime is 10 Hrs : 49 Mins : 55 Secs. was: In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is incorrect. When Browser and Yarn server are in different time zones, the Application Attempt's ElapsedTime is greater than the Application's ElapsedTime. > In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is > incorrect. > - > > Key: YARN-9880 > URL: https://issues.apache.org/jira/browse/YARN-9880 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Affects Versions: 3.1.1, 3.2.1 >Reporter: jenny >Priority: Major > Attachments: image-2019-10-09-17-22-31-603.png > > > In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is > incorrect. > When Browser and Yarn server are in different time zones, the Application > Attempt's ElapsedTime is greater than the Application's ElapsedTime. > I found this problem where hadoop3.1.1 was used, want to upgrade to the > latest version of hadoop, but found the running Application Attempt's > ElapsedTime is also incorrect. > In UI1, get ElapsedTime from yarn server. In UI2, get currentTime from > brower. Therefore, when the brower and the yarn server are in different time > zones, the ElapsedTime is incorrect. While the Application‘s ElapsedTime is > 29 Secs, the Application Attempt's ElapsedTime is 10 Hrs : 49 Mins : 55 Secs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7243) Moving logging APIs over to slf4j in hadoop-yarn-server-resourcemanager
[ https://issues.apache.org/jira/browse/YARN-7243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated YARN-7243: Fix Version/s: 3.2.2 Backported to branch-3.2. > Moving logging APIs over to slf4j in hadoop-yarn-server-resourcemanager > --- > > Key: YARN-7243 > URL: https://issues.apache.org/jira/browse/YARN-7243 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Yeliang Cang >Assignee: Prabhu Joseph >Priority: Major > Fix For: 3.3.0, 3.2.2 > > Attachments: YARN-7243-007.patch, YARN-7243-008.patch, > YARN-7243-009.patch, YARN-7243.001.patch, YARN-7243.002.patch, > YARN-7243.003.patch, YARN-7243.004.patch, YARN-7243.005.patch, > YARN-7243.006.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9873) Mutation API Config Change need to update Version Number
[ https://issues.apache.org/jira/browse/YARN-9873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948249#comment-16948249 ] Akira Ajisaka commented on YARN-9873: - Cherry-picked. > Mutation API Config Change need to update Version Number > - > > Key: YARN-9873 > URL: https://issues.apache.org/jira/browse/YARN-9873 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Fix For: 3.3.0, 3.2.2 > > Attachments: YARN-9873-001.patch, YARN-9873-002.patch, > YARN-9873-003.patch, YARN-9873-004.patch > > > Version Number support for each Scheduler Config Change. This also helps to > know when the last change happened. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org