[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN
[
https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947430#comment-16947430
]
Hadoop QA commented on YARN-9860:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
35s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 49s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
26s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 7m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
17s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated
0 new + 91 unchanged - 1 fixed = 91 total (was 92) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 0s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m
12s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 18m
55s{color} | {color:green} hadoop-yarn-services-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
23s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}119m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.3 Server=19.03.3
[jira] [Created] (YARN-9880) In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is incorrect.
jenny created YARN-9880: --- Summary: In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is incorrect. Key: YARN-9880 URL: https://issues.apache.org/jira/browse/YARN-9880 Project: Hadoop YARN Issue Type: Bug Components: yarn-ui-v2 Affects Versions: 3.2.1, 3.1.1 Reporter: jenny -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9881) In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is incorrect.
jenny created YARN-9881: --- Summary: In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is incorrect. Key: YARN-9881 URL: https://issues.apache.org/jira/browse/YARN-9881 Project: Hadoop YARN Issue Type: Bug Components: yarn-ui-v2 Affects Versions: 3.2.1, 3.1.1 Reporter: jenny -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9699) Migration tool that help to generate CS config based on FS config (Phase 1)
[ https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-9699: - Attachment: YARN-9699-009.patch > Migration tool that help to generate CS config based on FS config (Phase 1) > --- > > Key: YARN-9699 > URL: https://issues.apache.org/jira/browse/YARN-9699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wanqiang Ji >Assignee: Peter Bacsko >Priority: Major > Attachments: FS_to_CS_migration_POC.patch, YARN-9699-003.patch, > YARN-9699-004.patch, YARN-9699-005.patch, YARN-9699-006.patch, > YARN-9699-007.patch, YARN-9699-008.patch, YARN-9699-009.patch, > YARN-9699.001.patch, YARN-9699.002.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9699) Migration tool that help to generate CS config based on FS config (Phase 1)
[ https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947449#comment-16947449 ] Szilard Nemeth commented on YARN-9699: -- Hi [~pbacsko]! As discussed, patch008 contained an error near console mode: I haven't printed into the output stream at all if console mode was on. Attached patch009 to address this issue. > Migration tool that help to generate CS config based on FS config (Phase 1) > --- > > Key: YARN-9699 > URL: https://issues.apache.org/jira/browse/YARN-9699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wanqiang Ji >Assignee: Peter Bacsko >Priority: Major > Attachments: FS_to_CS_migration_POC.patch, YARN-9699-003.patch, > YARN-9699-004.patch, YARN-9699-005.patch, YARN-9699-006.patch, > YARN-9699-007.patch, YARN-9699-008.patch, YARN-9699-009.patch, > YARN-9699.001.patch, YARN-9699.002.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-9881) In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is incorrect.
[ https://issues.apache.org/jira/browse/YARN-9881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prabhu Joseph resolved YARN-9881. - Resolution: Duplicate > In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is > incorrect. > - > > Key: YARN-9881 > URL: https://issues.apache.org/jira/browse/YARN-9881 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Affects Versions: 3.1.1, 3.2.1 >Reporter: jenny >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9881) In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is incorrect.
[ https://issues.apache.org/jira/browse/YARN-9881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947454#comment-16947454 ] Prabhu Joseph commented on YARN-9881: - [~cjn082030] This looks like a duplicate of YARN-9880. Will close this Jira as Duplicate. > In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is > incorrect. > - > > Key: YARN-9881 > URL: https://issues.apache.org/jira/browse/YARN-9881 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Affects Versions: 3.1.1, 3.2.1 >Reporter: jenny >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9699) Migration tool that help to generate CS config based on FS config (Phase 1)
[ https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947455#comment-16947455 ] Szilard Nemeth commented on YARN-9699: -- Also attached patch010 to address the remaining checkstyle / findbugs issues. We can ignore the "variable hides a field" kind of errors. > Migration tool that help to generate CS config based on FS config (Phase 1) > --- > > Key: YARN-9699 > URL: https://issues.apache.org/jira/browse/YARN-9699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wanqiang Ji >Assignee: Peter Bacsko >Priority: Major > Attachments: FS_to_CS_migration_POC.patch, YARN-9699-003.patch, > YARN-9699-004.patch, YARN-9699-005.patch, YARN-9699-006.patch, > YARN-9699-007.patch, YARN-9699-008.patch, YARN-9699-009.patch, > YARN-9699-010.patch, YARN-9699.001.patch, YARN-9699.002.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9699) Migration tool that help to generate CS config based on FS config (Phase 1)
[ https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-9699: - Attachment: YARN-9699-010.patch > Migration tool that help to generate CS config based on FS config (Phase 1) > --- > > Key: YARN-9699 > URL: https://issues.apache.org/jira/browse/YARN-9699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wanqiang Ji >Assignee: Peter Bacsko >Priority: Major > Attachments: FS_to_CS_migration_POC.patch, YARN-9699-003.patch, > YARN-9699-004.patch, YARN-9699-005.patch, YARN-9699-006.patch, > YARN-9699-007.patch, YARN-9699-008.patch, YARN-9699-009.patch, > YARN-9699-010.patch, YARN-9699.001.patch, YARN-9699.002.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9881) In YARN ui2 attempts tab, The running Application Attempt's Container's ElapsedTime is incorrect.
[ https://issues.apache.org/jira/browse/YARN-9881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jenny updated YARN-9881: Summary: In YARN ui2 attempts tab, The running Application Attempt's Container's ElapsedTime is incorrect. (was: In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is incorrect.) > In YARN ui2 attempts tab, The running Application Attempt's Container's > ElapsedTime is incorrect. > - > > Key: YARN-9881 > URL: https://issues.apache.org/jira/browse/YARN-9881 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Affects Versions: 3.1.1, 3.2.1 >Reporter: jenny >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9880) In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is incorrect.
[ https://issues.apache.org/jira/browse/YARN-9880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jenny updated YARN-9880: Description: In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is incorrect. When Browser and Yarn server are in different time zones, the Application Attempt's ElapsedTime is greater than the Application's ElapsedTime. > In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is > incorrect. > - > > Key: YARN-9880 > URL: https://issues.apache.org/jira/browse/YARN-9880 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Affects Versions: 3.1.1, 3.2.1 >Reporter: jenny >Priority: Major > > In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is > incorrect. > When Browser and Yarn server are in different time zones, the Application > Attempt's ElapsedTime is greater than the Application's ElapsedTime. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9880) In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is incorrect.
[ https://issues.apache.org/jira/browse/YARN-9880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947459#comment-16947459 ] Prabhu Joseph commented on YARN-9880: - [~cjn082030] Can you share how you are verifying Application Attempt's ElapsedTime is incorrect. Are you comparing with UI1. RM UI screenshots will help to under the issue better. > In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is > incorrect. > - > > Key: YARN-9880 > URL: https://issues.apache.org/jira/browse/YARN-9880 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Affects Versions: 3.1.1, 3.2.1 >Reporter: jenny >Priority: Major > > In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is > incorrect. > When Browser and Yarn server are in different time zones, the Application > Attempt's ElapsedTime is greater than the Application's ElapsedTime. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9699) Migration tool that help to generate CS config based on FS config (Phase 1)
[
https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947461#comment-16947461
]
Peter Bacsko commented on YARN-9699:
Ok, here are my comments - I collected these when patch v2 was the latest but I
believe most of them are still valid.
#1
{noformat}
137 throw new IllegalArgumentException(
138 String.format("Missing %s parameter " + "(switch: %s|%s).",
139 cliOption.name, cliOption.shortSwitch,
cliOption.longSwitch));
{noformat}
{noformat}
158 if (isFile && file.isDirectory()) {
159 throw new IllegalArgumentException(String.format("Specified path
%s is a directory but should be a file " +
160 "(As value of parameter %s)", filePath, cliOption.name));
161 } else if (!isFile && !file.isDirectory()) {
162 throw new IllegalArgumentException(String.format("Specified path
%s is not a directory " +
163 "(As value of parameter %s)", filePath, cliOption.name));
164 } else if (!file.exists()) {
165 throw new IllegalArgumentException(String.format("Specified path
%s does not exist " +
166 "(As value of parameter %s)", filePath, cliOption.name));
167 }
{noformat}
These exceptions will propagate up to
{{fsConfigConversionArgumentHandler.parseAndConvert(args);}} and errors will be
printed with a stack trace.
Wouldn't it be better to use a different exception and handle it without
printing the stack trace?
#2
{noformat}
81private enum OutputFile {
82 YARN_SITE_XML("yarn-site.xml"),
83 CAPACITY_SCHEDULER_XML("capacity-scheduler.xml");
84
85 private final String name;
86
87 OutputFile(String name) {
88this.name = name;
89 }
90}
{noformat}
Do we need an enum for this purpose?
#3
{noformat}
if (maxAssign != -1)
{noformat}
"-1" is a magic number, replace it with
{{FairSchedulerConfiguration.DEFAULT_MAX_ASSIGN}}.
#4
{noformat}
435 if (queueMaxAmShare != 0.0f
436 && queueMaxAmShare != queueMaxAMshareDefault
437 && queueMaxAmShare != -1.0f) {
438 capacacitySchedulerConfig.set("yarn.scheduler.capacity." +
queueName + ".maximum-am-resource-percent",
439 String.valueOf(queueMaxAmShare));
440 }
441
442 if (queueMaxAmShare == -1.0f) {
443 capacacitySchedulerConfig.set("yarn.scheduler.capacity." +
queueName + ".maximum-am-resource-percent",
444 "1.0");
445 }
{noformat}
It's my part, but direct floating point comparison could be dangerous. Let's
think this over. Basically we have to make sure that float converted from a
string "-1.0" is the same as the {{-1.0}} literal in the code.
#5
{noformat}
448 private void emitMaxRunningApps(String queueName, FSQueue queue) {
449 if (queue.getMaxRunningApps() != Integer.MAX_VALUE
{noformat}
{{Integer.MAX_VALUE}} is a magic number, extract it to a constant.
#6
{{Resources.unbounded().compareTo(...)}}
This comparison is repeated many times, should be extracted to a simple helper
method.
#7
{noformat}
681 } else if (children.size() == 1) {
682 // just a single element, probably this branch should
683 // not be reached
684 String queueName = children.get(0).getName();
685 capacities.put(queueName, Capacity.newCapacity(HUNDRED));
686 }
{noformat}
This branch is probably unnecessary. Handle {{children.size() == 0}} and
{{children.size() == 1}} cases separately before walking through the children
list.
> Migration tool that help to generate CS config based on FS config (Phase 1)
> ---
>
> Key: YARN-9699
> URL: https://issues.apache.org/jira/browse/YARN-9699
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wanqiang Ji
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: FS_to_CS_migration_POC.patch, YARN-9699-003.patch,
> YARN-9699-004.patch, YARN-9699-005.patch, YARN-9699-006.patch,
> YARN-9699-007.patch, YARN-9699-008.patch, YARN-9699-009.patch,
> YARN-9699-010.patch, YARN-9699.001.patch, YARN-9699.002.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9880) In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is incorrect.
[ https://issues.apache.org/jira/browse/YARN-9880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jenny updated YARN-9880: Attachment: image-2019-10-09-17-22-31-603.png > In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is > incorrect. > - > > Key: YARN-9880 > URL: https://issues.apache.org/jira/browse/YARN-9880 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Affects Versions: 3.1.1, 3.2.1 >Reporter: jenny >Priority: Major > Attachments: image-2019-10-09-17-22-31-603.png > > > In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is > incorrect. > When Browser and Yarn server are in different time zones, the Application > Attempt's ElapsedTime is greater than the Application's ElapsedTime. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9880) In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is incorrect.
[ https://issues.apache.org/jira/browse/YARN-9880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jenny updated YARN-9880: Attachment: image-2019-10-09-17-23-08-311.png > In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is > incorrect. > - > > Key: YARN-9880 > URL: https://issues.apache.org/jira/browse/YARN-9880 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Affects Versions: 3.1.1, 3.2.1 >Reporter: jenny >Priority: Major > Attachments: image-2019-10-09-17-22-31-603.png > > > In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is > incorrect. > When Browser and Yarn server are in different time zones, the Application > Attempt's ElapsedTime is greater than the Application's ElapsedTime. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9880) In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is incorrect.
[ https://issues.apache.org/jira/browse/YARN-9880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jenny updated YARN-9880: Attachment: (was: image-2019-10-09-17-23-08-311.png) > In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is > incorrect. > - > > Key: YARN-9880 > URL: https://issues.apache.org/jira/browse/YARN-9880 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Affects Versions: 3.1.1, 3.2.1 >Reporter: jenny >Priority: Major > Attachments: image-2019-10-09-17-22-31-603.png > > > In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is > incorrect. > When Browser and Yarn server are in different time zones, the Application > Attempt's ElapsedTime is greater than the Application's ElapsedTime. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9880) In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is incorrect.
[ https://issues.apache.org/jira/browse/YARN-9880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947476#comment-16947476 ] jenny commented on YARN-9880: - Hi [~Prabhu Joseph] , please see the screenshots . > In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is > incorrect. > - > > Key: YARN-9880 > URL: https://issues.apache.org/jira/browse/YARN-9880 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Affects Versions: 3.1.1, 3.2.1 >Reporter: jenny >Priority: Major > Attachments: image-2019-10-09-17-22-31-603.png > > > In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is > incorrect. > When Browser and Yarn server are in different time zones, the Application > Attempt's ElapsedTime is greater than the Application's ElapsedTime. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9880) In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is incorrect.
[ https://issues.apache.org/jira/browse/YARN-9880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947476#comment-16947476 ] jenny edited comment on YARN-9880 at 10/9/19 9:27 AM: -- Hi [~Prabhu Joseph] , please see the screenshots (UI2) . UI1 is right. was (Author: cjn082030): Hi [~Prabhu Joseph] , please see the screenshots . > In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is > incorrect. > - > > Key: YARN-9880 > URL: https://issues.apache.org/jira/browse/YARN-9880 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Affects Versions: 3.1.1, 3.2.1 >Reporter: jenny >Priority: Major > Attachments: image-2019-10-09-17-22-31-603.png > > > In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is > incorrect. > When Browser and Yarn server are in different time zones, the Application > Attempt's ElapsedTime is greater than the Application's ElapsedTime. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9256) Make ATSv2 compilation default with hbase.profile=2.0
[ https://issues.apache.org/jira/browse/YARN-9256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947487#comment-16947487 ] Prabhu Joseph commented on YARN-9256: - [~rohithsharma] I will work on this, will assign this Jira to me. > Make ATSv2 compilation default with hbase.profile=2.0 > - > > Key: YARN-9256 > URL: https://issues.apache.org/jira/browse/YARN-9256 > Project: Hadoop YARN > Issue Type: Task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Major > Attachments: YARN-9256.01.patch > > > By default Hadoop compiles with hbase.profile one which corresponds to > hbase.version=1.4 for ATSv2. Change compilation to hbase.profile=2.0 by > default in trunk. > This JIRA is to discuss for any concerns. > cc:/ [~vrushalic] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9256) Make ATSv2 compilation default with hbase.profile=2.0
[ https://issues.apache.org/jira/browse/YARN-9256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prabhu Joseph reassigned YARN-9256: --- Assignee: Prabhu Joseph (was: Rohith Sharma K S) > Make ATSv2 compilation default with hbase.profile=2.0 > - > > Key: YARN-9256 > URL: https://issues.apache.org/jira/browse/YARN-9256 > Project: Hadoop YARN > Issue Type: Task >Reporter: Rohith Sharma K S >Assignee: Prabhu Joseph >Priority: Major > Attachments: YARN-9256.01.patch > > > By default Hadoop compiles with hbase.profile one which corresponds to > hbase.version=1.4 for ATSv2. Change compilation to hbase.profile=2.0 by > default in trunk. > This JIRA is to discuss for any concerns. > cc:/ [~vrushalic] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Reopened] (YARN-9881) In YARN ui2 attempts tab, The running Application Attempt's Container's ElapsedTime is incorrect.
[ https://issues.apache.org/jira/browse/YARN-9881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prabhu Joseph reopened YARN-9881: - > In YARN ui2 attempts tab, The running Application Attempt's Container's > ElapsedTime is incorrect. > - > > Key: YARN-9881 > URL: https://issues.apache.org/jira/browse/YARN-9881 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Affects Versions: 3.1.1, 3.2.1 >Reporter: jenny >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (YARN-9881) In YARN ui2 attempts tab, The running Application Attempt's Container's ElapsedTime is incorrect.
[ https://issues.apache.org/jira/browse/YARN-9881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prabhu Joseph updated YARN-9881: Comment: was deleted (was: [~cjn082030] This looks like a duplicate of YARN-9880. Will close this Jira as Duplicate.) > In YARN ui2 attempts tab, The running Application Attempt's Container's > ElapsedTime is incorrect. > - > > Key: YARN-9881 > URL: https://issues.apache.org/jira/browse/YARN-9881 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Affects Versions: 3.1.1, 3.2.1 >Reporter: jenny >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9873) Mutation API Config Change updates Version Number
[ https://issues.apache.org/jira/browse/YARN-9873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947523#comment-16947523 ] Sunil G commented on YARN-9873: --- Thanks [~Prabhu Joseph] for addressing comments Pushing this in . +1 > Mutation API Config Change updates Version Number > -- > > Key: YARN-9873 > URL: https://issues.apache.org/jira/browse/YARN-9873 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Fix For: 3.3.0, 3.2.2 > > Attachments: YARN-9873-001.patch, YARN-9873-002.patch, > YARN-9873-003.patch, YARN-9873-004.patch > > > Version Number support for each Scheduler Config Change. This also helps to > know when the last change happened. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9873) Mutation API Config Change need to update Version Number
[ https://issues.apache.org/jira/browse/YARN-9873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-9873: -- Summary: Mutation API Config Change need to update Version Number (was: Mutation API Config Change updates Version Number ) > Mutation API Config Change need to update Version Number > - > > Key: YARN-9873 > URL: https://issues.apache.org/jira/browse/YARN-9873 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Fix For: 3.3.0, 3.2.2 > > Attachments: YARN-9873-001.patch, YARN-9873-002.patch, > YARN-9873-003.patch, YARN-9873-004.patch > > > Version Number support for each Scheduler Config Change. This also helps to > know when the last change happened. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9873) Mutation API Config Change need to update Version Number
[ https://issues.apache.org/jira/browse/YARN-9873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947524#comment-16947524 ] Sunil G commented on YARN-9873: --- Committed to trunk/branch-3.2 > Mutation API Config Change need to update Version Number > - > > Key: YARN-9873 > URL: https://issues.apache.org/jira/browse/YARN-9873 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Fix For: 3.3.0, 3.2.2 > > Attachments: YARN-9873-001.patch, YARN-9873-002.patch, > YARN-9873-003.patch, YARN-9873-004.patch > > > Version Number support for each Scheduler Config Change. This also helps to > know when the last change happened. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9873) Mutation API Config Change need to update Version Number
[ https://issues.apache.org/jira/browse/YARN-9873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947526#comment-16947526 ] Prabhu Joseph commented on YARN-9873: - Thanks [~sunilg]. > Mutation API Config Change need to update Version Number > - > > Key: YARN-9873 > URL: https://issues.apache.org/jira/browse/YARN-9873 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Fix For: 3.3.0, 3.2.2 > > Attachments: YARN-9873-001.patch, YARN-9873-002.patch, > YARN-9873-003.patch, YARN-9873-004.patch > > > Version Number support for each Scheduler Config Change. This also helps to > know when the last change happened. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9873) Mutation API Config Change need to update Version Number
[ https://issues.apache.org/jira/browse/YARN-9873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947536#comment-16947536 ] Hudson commented on YARN-9873: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17508 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17508/]) YARN-9873. Mutation API Config Change need to update Version Number. (sunilg: rev be901f49628a553cfe6a3942478cb528cce2b266) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/FSSchedulerConfigurationStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/MutableCSConfigurationProvider.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/MutableConfigurationProvider.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWSConsts.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ConfigVersionInfo.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesConfigurationMutation.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/InMemoryConfigurationStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/YarnConfigurationStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestFSSchedulerConfigurationStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/LeveldbConfigurationStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestZKConfigurationStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/ZKConfigurationStore.java > Mutation API Config Change need to update Version Number > - > > Key: YARN-9873 > URL: https://issues.apache.org/jira/browse/YARN-9873 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Fix For: 3.3.0, 3.2.2 > > Attachments: YARN-9873-001.patch, YARN-9873-002.patch, > YARN-9873-003.patch, YARN-9873-004.patch > > > Version Number support for each Scheduler Config Change. This also helps to > know when the last change happened. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9882) QueueMetrics not coming in Capacity Scheduler with Node Label Configuration
Gaurav Suman created YARN-9882:
--
Summary: QueueMetrics not coming in Capacity Scheduler with Node
Label Configuration
Key: YARN-9882
URL: https://issues.apache.org/jira/browse/YARN-9882
Project: Hadoop YARN
Issue Type: Bug
Components: capacity scheduler, metrics, scheduler
Reporter: Gaurav Suman
I am having a capacity scheduler setup with two queues - "low-priority",
"regular-priority". There are two node-labels "low" and "regular". low-priority
queue has 100% access to "low" node-label and regular-priority queue has 100%
access to "regular" node label.
The yarn ui capacity scheduler configuration -
[https://i.stack.imgur.com/gOARn.png]
When i see the QueueMetrics emitted by queue "low-priority" and
"regular-priority" in (http://rm-ip:port/jmx), then it shows correct values of
availableMB and availableVCores, pendingMB=0 etc. but when I submit a job to
any queue, there is no update in jmx metrics like pendingMB, pendingVcores,
availableMB, availableVCores etc. only AppsRunning, ActiveApplications etc. are
getting updated. The pendingMB, pendingVcores remains always 0 and there is no
changes in availableMB, availableVcores, appsRunning and activeApplications
shows correct value as 1. Not able to find why the metrics is not getting
updated after job submission.
The issue comes only when node-label is enabled. When node-label is disabled
and only queue is used everything works fine.
The capacity scheduler configuration(capacity-scheduler.xml):
{code:java}
yarn.scheduler.capacity.maximum-applications
5000
yarn.scheduler.capacity.maximum-am-resource-percent
0.2
yarn.scheduler.capacity.resource-calculator
org.apache.hadoop.yarn.util.resource.DominantResourceCalculator
The ResourceCalculator implementation to be used to compare
Resources in the scheduler.
The default i.e. DefaultResourceCalculator only uses Memory while
DominantResourceCalculator uses dominant-resource to compare
multi-dimensional resources such as Memory, CPU etc.
yarn.scheduler.capacity.root.queues
low-priority,regular-priority
yarn.scheduler.capacity.root.capacity
100
yarn.scheduler.capacity.root.maximum-capacity
100
yarn.scheduler.capacity.root.accessible-node-labels
*
yarn.scheduler.capacity.root.accessible-node-labels.regular.capacity
100
yarn.scheduler.capacity.root.accessible-node-labels.regular.maximum-capacity
100
yarn.scheduler.capacity.root.accessible-node-labels.low.capacity
100
yarn.scheduler.capacity.root.accessible-node-labels.low.maximum-capacity
100
yarn.scheduler.capacity.root.default.state
RUNNING
The state of the default queue. State can be one of RUNNING or STOPPED.
yarn.scheduler.capacity.root.default.acl_submit_applications
*
yarn.scheduler.capacity.root.default.acl_administer_queue
*
The ACL of who can administer jobs on the default queue.
yarn.scheduler.capacity.node-locality-delay
40
yarn.scheduler.capacity.queue-mappings-override.enable
false
yarn.scheduler.capacity.root.low-priority.capacity
50
yarn.scheduler.capacity.root.low-priority.maximum-capacity
100
yarn.scheduler.capacity.root.low-priority.ordering-policy
fair
yarn.scheduler.capacity.root.low-priority.accessible-node-labels
low
yarn.scheduler.capacity.root.low-priority.default-node-label-expression
low
yarn.scheduler.capacity.root.low-priority.accessible-node-labels.low.capacity
100
yarn.scheduler.capacity.root.low-priority.accessible-node-labels.low.maximum-capacity
100
yarn.scheduler.capacity.root.low-priority.default.state
RUNNING
yarn.scheduler.capacity.root.regular-priority.capacity
50
yarn.scheduler.capacity.root.regular-priority.maximum-capacity
100
yarn.scheduler.capacity.root.regular-priority.ordering-policy
fair
yarn.scheduler.capacity.root.regular-priority.accessible-node-labels
regular
yarn.scheduler.capacity.root.regular-priority.default-node-label-expression
regular
yarn.scheduler.capacity.root.regular-priority.accessible-node-labels.regular.capacity
100
yarn.scheduler.capacity.root.regular-priority.accessible-node-labels.regular.maximum-capacity
100
yarn.scheduler.capacity.root.regular-priority.default.state
RUNNING
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9693) When AMRMProxyService is enabled RMCommunicator will register with failure
[
https://issues.apache.org/jira/browse/YARN-9693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhoukang reassigned YARN-9693:
--
Assignee: zhoukang
> When AMRMProxyService is enabled RMCommunicator will register with failure
> --
>
> Key: YARN-9693
> URL: https://issues.apache.org/jira/browse/YARN-9693
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: federation
>Affects Versions: 3.1.2
>Reporter: zhoukang
>Assignee: zhoukang
>Priority: Major
>
> When we enable amrm proxy service, the RMCommunicator will register with
> failure below:
> {code:java}
> 2019-07-23 17:12:44,794 INFO [TaskHeartbeatHandler PingChecker]
> org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler
> thread interrupted
> 2019-07-23 17:12:44,794 ERROR [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid
> AMRMToken from appattempt_1563872237585_0001_02
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:186)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:123)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:280)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.serviceStart(MRAppMaster.java:986)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1300)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(MRAppMaster.java:1768)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1716)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1764)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1698)
> Caused by: org.apache.hadoop.security.token.SecretManager$InvalidToken:
> Invalid AMRMToken from appattempt_1563872237585_0001_02
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
> at com.sun.proxy.$Proxy93.registerApplicationMaster(Unknown Source)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:170)
> ... 14 more
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> Invalid AMRMToken from appattempt_1563872237585_0001_02
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1541)
> at org.apache.hadoo
[jira] [Comment Edited] (YARN-9693) When AMRMProxyService is enabled RMCommunicator will register with failure
[
https://issues.apache.org/jira/browse/YARN-9693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947566#comment-16947566
]
zhoukang edited comment on YARN-9693 at 10/9/19 11:10 AM:
--
I will work on this.Patch on the way
was (Author: cane):
I will work on this.Path on the way
> When AMRMProxyService is enabled RMCommunicator will register with failure
> --
>
> Key: YARN-9693
> URL: https://issues.apache.org/jira/browse/YARN-9693
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: federation
>Affects Versions: 3.1.2
>Reporter: zhoukang
>Assignee: zhoukang
>Priority: Major
>
> When we enable amrm proxy service, the RMCommunicator will register with
> failure below:
> {code:java}
> 2019-07-23 17:12:44,794 INFO [TaskHeartbeatHandler PingChecker]
> org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler
> thread interrupted
> 2019-07-23 17:12:44,794 ERROR [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid
> AMRMToken from appattempt_1563872237585_0001_02
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:186)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:123)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:280)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.serviceStart(MRAppMaster.java:986)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1300)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(MRAppMaster.java:1768)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1716)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1764)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1698)
> Caused by: org.apache.hadoop.security.token.SecretManager$InvalidToken:
> Invalid AMRMToken from appattempt_1563872237585_0001_02
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
> at com.sun.proxy.$Proxy93.registerApplicationMaster(Unknown Source)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:170)
> ... 14 more
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$Inva
[jira] [Commented] (YARN-9693) When AMRMProxyService is enabled RMCommunicator will register with failure
[
https://issues.apache.org/jira/browse/YARN-9693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947566#comment-16947566
]
zhoukang commented on YARN-9693:
I will work on this.Path on the way
> When AMRMProxyService is enabled RMCommunicator will register with failure
> --
>
> Key: YARN-9693
> URL: https://issues.apache.org/jira/browse/YARN-9693
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: federation
>Affects Versions: 3.1.2
>Reporter: zhoukang
>Assignee: zhoukang
>Priority: Major
>
> When we enable amrm proxy service, the RMCommunicator will register with
> failure below:
> {code:java}
> 2019-07-23 17:12:44,794 INFO [TaskHeartbeatHandler PingChecker]
> org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler
> thread interrupted
> 2019-07-23 17:12:44,794 ERROR [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid
> AMRMToken from appattempt_1563872237585_0001_02
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:186)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:123)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:280)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.serviceStart(MRAppMaster.java:986)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1300)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(MRAppMaster.java:1768)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1716)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1764)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1698)
> Caused by: org.apache.hadoop.security.token.SecretManager$InvalidToken:
> Invalid AMRMToken from appattempt_1563872237585_0001_02
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
> at com.sun.proxy.$Proxy93.registerApplicationMaster(Unknown Source)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:170)
> ... 14 more
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> Invalid AMRMToken from appattempt_1563872237585_0001_02
> at org.apache.hadoop.ipc.Cli
[jira] [Commented] (YARN-9356) Add more tests to ratio method in TestResourceCalculator
[ https://issues.apache.org/jira/browse/YARN-9356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947571#comment-16947571 ] Szilard Nemeth commented on YARN-9356: -- Hi [~zsiegl]! Thanks for the patch! Commited to: trunk, branch-3.2, branch-3.1. Thanks [~adam.antal] for the review! > Add more tests to ratio method in TestResourceCalculator > - > > Key: YARN-9356 > URL: https://issues.apache.org/jira/browse/YARN-9356 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Zoltan Siegl >Priority: Major > Attachments: YARN-9356.001.patch, YARN-9356.001.patch > > > TestResourceCalculator has some edge-case testcases to verify how division by > zero is handled with ResourceCalculator. > We need other basic tests like we have for other ResourceCalculator methods. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9699) Migration tool that help to generate CS config based on FS config (Phase 1)
[
https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947577#comment-16947577
]
Hadoop QA commented on YARN-9699:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
41s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 16 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 26s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
16s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 18s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 55 new + 137 unchanged - 7 fixed = 192 total (was 144) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} xml {color} | {color:red} 0m 9s{color} |
{color:red} The patch has 2 ill-formed XML file(s). {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 6s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
27s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 84m
41s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
22s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}159m 35s{color} |
{color:black} {color} |
\\
\\
||
[jira] [Updated] (YARN-9128) Use SerializationUtils from apache commons to serialize / deserialize ResourceMappings
[ https://issues.apache.org/jira/browse/YARN-9128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-9128: - Fix Version/s: 3.3.0 > Use SerializationUtils from apache commons to serialize / deserialize > ResourceMappings > -- > > Key: YARN-9128 > URL: https://issues.apache.org/jira/browse/YARN-9128 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Zoltan Siegl >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9128.001.patch, YARN-9128.002.patch, > YARN-9128.003.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9128) Use SerializationUtils from apache commons to serialize / deserialize ResourceMappings
[ https://issues.apache.org/jira/browse/YARN-9128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947581#comment-16947581 ] Szilard Nemeth commented on YARN-9128: -- Hi [~zsiegl]! Thanks for this patch, pushed to trunk / branch-3.2 / branch-3.1! > Use SerializationUtils from apache commons to serialize / deserialize > ResourceMappings > -- > > Key: YARN-9128 > URL: https://issues.apache.org/jira/browse/YARN-9128 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Zoltan Siegl >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9128.001.patch, YARN-9128.002.patch, > YARN-9128.003.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9356) Add more tests to ratio method in TestResourceCalculator
[ https://issues.apache.org/jira/browse/YARN-9356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947582#comment-16947582 ] Hudson commented on YARN-9356: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17509 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17509/]) YARN-9356. Add more tests to ratio method in TestResourceCalculator. (snemeth: rev 35f093f5b3f310af49a8bb476e62173862f5af51) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/resource/TestResourceCalculator.java > Add more tests to ratio method in TestResourceCalculator > - > > Key: YARN-9356 > URL: https://issues.apache.org/jira/browse/YARN-9356 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Zoltan Siegl >Priority: Major > Fix For: 3.3.0, 3.2.2, 3.1.4 > > Attachments: YARN-9356.001.patch, YARN-9356.001.patch > > > TestResourceCalculator has some edge-case testcases to verify how division by > zero is handled with ResourceCalculator. > We need other basic tests like we have for other ResourceCalculator methods. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9541) TestCombinedSystemMetricsPublisher and TestSystemMetricsPublisherForV2 fail intermittent
[
https://issues.apache.org/jira/browse/YARN-9541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Siegl updated YARN-9541:
---
Summary: TestCombinedSystemMetricsPublisher and
TestSystemMetricsPublisherForV2 fail intermittent (was:
TestCombinedSystemMetricsPublisher fails intermittent)
> TestCombinedSystemMetricsPublisher and TestSystemMetricsPublisherForV2 fail
> intermittent
>
>
> Key: YARN-9541
> URL: https://issues.apache.org/jira/browse/YARN-9541
> Project: Hadoop YARN
> Issue Type: Bug
> Components: ATSv2
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
>
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TestCombinedSystemMetricsPublisher.testTimelineServiceEventPublishingV1V2Enabled
> {code}
> Failing for the past 1 build (Since Failed#24071 )
> Took 0.19 sec.
> Error Message
> java.net.BindException: Problem binding to [0.0.0.0:10200]
> java.net.BindException: Address already in use; For more details see:
> http://wiki.apache.org/hadoop/BindException
> Stacktrace
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.net.BindException: Problem binding to [0.0.0.0:10200]
> java.net.BindException: Address already in use; For more details see:
> http://wiki.apache.org/hadoop/BindException
> at
> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.getServer(RpcServerFactoryPBImpl.java:139)
> at
> org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC.getServer(HadoopYarnProtoRPC.java:66)
> at org.apache.hadoop.yarn.ipc.YarnRPC.getServer(YarnRPC.java:55)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryClientService.serviceStart(ApplicationHistoryClientService.java:94)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceStart(ApplicationHistoryServer.java:120)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TestCombinedSystemMetricsPublisher.testSetup(TestCombinedSystemMetricsPublisher.java:123)
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TestCombinedSystemMetricsPublisher.runTest(TestCombinedSystemMetricsPublisher.java:242)
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TestCombinedSystemMetricsPublisher.testTimelineServiceEventPublishingV1V2Enabled(TestCombinedSystemMetricsPublisher.java:252)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.net.BindException: Problem binding to [0.0.0.0:10200]
> java.net.BindException: Address already in use; For more details see:
> http://wiki.apache.org/hadoop/BindException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:833)
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:738)
> at org.apache.hadoop.ipc.Server.bind(Server.java:599)
> at org.apache.hadoop.ipc.Server$Listener.(Server.java:1121)
> at org.apache.hadoop.ipc.Server.(Server.java:2976)
> at org.apache.hadoop.ipc.RPC$Server.(RPC.java:1039)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Se
[jira] [Commented] (YARN-9541) TestCombinedSystemMetricsPublisher and TestSystemMetricsPublisherForV2 fail intermittent
[
https://issues.apache.org/jira/browse/YARN-9541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947583#comment-16947583
]
Zoltan Siegl commented on YARN-9541:
TestCombinedSystemMetricsPublisher.testTimelineServiceEventPublishingV2Enabled
TestCombinedSystemMetricsPublisher.testTimelineServiceEventPublishingV1V2Enabled
TestSystemMetricsPublisherForV2.testPublishApplicationMetrics
TestSystemMetricsPublisherForV2.testPublishAppAttemptMetrics
TestSystemMetricsPublisherForV2.testPublishContainerMetrics
These fail together consistently, and seemingly for the same route cause.
> TestCombinedSystemMetricsPublisher and TestSystemMetricsPublisherForV2 fail
> intermittent
>
>
> Key: YARN-9541
> URL: https://issues.apache.org/jira/browse/YARN-9541
> Project: Hadoop YARN
> Issue Type: Bug
> Components: ATSv2
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
>
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TestCombinedSystemMetricsPublisher.testTimelineServiceEventPublishingV1V2Enabled
> {code}
> Failing for the past 1 build (Since Failed#24071 )
> Took 0.19 sec.
> Error Message
> java.net.BindException: Problem binding to [0.0.0.0:10200]
> java.net.BindException: Address already in use; For more details see:
> http://wiki.apache.org/hadoop/BindException
> Stacktrace
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.net.BindException: Problem binding to [0.0.0.0:10200]
> java.net.BindException: Address already in use; For more details see:
> http://wiki.apache.org/hadoop/BindException
> at
> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.getServer(RpcServerFactoryPBImpl.java:139)
> at
> org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC.getServer(HadoopYarnProtoRPC.java:66)
> at org.apache.hadoop.yarn.ipc.YarnRPC.getServer(YarnRPC.java:55)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryClientService.serviceStart(ApplicationHistoryClientService.java:94)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceStart(ApplicationHistoryServer.java:120)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TestCombinedSystemMetricsPublisher.testSetup(TestCombinedSystemMetricsPublisher.java:123)
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TestCombinedSystemMetricsPublisher.runTest(TestCombinedSystemMetricsPublisher.java:242)
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TestCombinedSystemMetricsPublisher.testTimelineServiceEventPublishingV1V2Enabled(TestCombinedSystemMetricsPublisher.java:252)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.net.BindException: Problem binding to [0.0.0.0:10200]
> java.net.BindException: Address already in use; For more details see:
> http://wiki.apache.org/hadoop/BindException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:833)
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:7
[jira] [Commented] (YARN-5106) Provide a builder interface for FairScheduler allocations for use in tests
[ https://issues.apache.org/jira/browse/YARN-5106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947590#comment-16947590 ] Szilard Nemeth commented on YARN-5106: -- Test failures are unrelated, see [~zsiegl]'s [comment|https://issues.apache.org/jira/browse/YARN-9541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947583#comment-16947583] > Provide a builder interface for FairScheduler allocations for use in tests > -- > > Key: YARN-5106 > URL: https://issues.apache.org/jira/browse/YARN-5106 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Zoltan Siegl >Priority: Major > Labels: newbie++ > Attachments: YARN-5106-branch-3.1.001.patch, > YARN-5106-branch-3.1.001.patch, YARN-5106-branch-3.1.001.patch, > YARN-5106-branch-3.1.002.patch, YARN-5106-branch-3.2.001.patch, > YARN-5106-branch-3.2.001.patch, YARN-5106-branch-3.2.002.patch, > YARN-5106.001.patch, YARN-5106.002.patch, YARN-5106.003.patch, > YARN-5106.004.patch, YARN-5106.005.patch, YARN-5106.006.patch, > YARN-5106.007.patch, YARN-5106.008.patch, YARN-5106.008.patch, > YARN-5106.008.patch, YARN-5106.009.patch, YARN-5106.010.patch, > YARN-5106.011.patch > > > Most, if not all, fair scheduler tests create an allocations XML file. Having > a helper class that potentially uses a builder would make the tests cleaner. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9699) Migration tool that help to generate CS config based on FS config (Phase 1)
[
https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947591#comment-16947591
]
Hadoop QA commented on YARN-9699:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 12m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 16 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
58s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 56s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
39s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 18s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 53 new + 136 unchanged - 7 fixed = 189 total (was 143) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} xml {color} | {color:red} 0m 10s{color} |
{color:red} The patch has 2 ill-formed XML file(s). {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 11s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
29s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 81m 39s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
28s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
50s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}167m 18s{color} |
{color:black} {color} |
\\
\\
|| Reason |
[jira] [Commented] (YARN-9128) Use SerializationUtils from apache commons to serialize / deserialize ResourceMappings
[ https://issues.apache.org/jira/browse/YARN-9128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947593#comment-16947593 ] Hudson commented on YARN-9128: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17510 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17510/]) YARN-9128. Use SerializationUtils from apache commons to serialize / (snemeth: rev 6f1ab9516869dbfdee034f4a17bccc379a39fc31) * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/TestResourceMappings.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ResourceMappings.java > Use SerializationUtils from apache commons to serialize / deserialize > ResourceMappings > -- > > Key: YARN-9128 > URL: https://issues.apache.org/jira/browse/YARN-9128 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Zoltan Siegl >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9128.001.patch, YARN-9128.002.patch, > YARN-9128.003.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5106) Provide a builder interface for FairScheduler allocations for use in tests
[ https://issues.apache.org/jira/browse/YARN-5106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947594#comment-16947594 ] Szilard Nemeth commented on YARN-5106: -- Hi [~zsiegl]! Just noticed that you have a lot of checkstyle failures for all patches. Could you please check these? [~pbacsko]: Do you have some time to review the patch? Thanks! > Provide a builder interface for FairScheduler allocations for use in tests > -- > > Key: YARN-5106 > URL: https://issues.apache.org/jira/browse/YARN-5106 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Zoltan Siegl >Priority: Major > Labels: newbie++ > Attachments: YARN-5106-branch-3.1.001.patch, > YARN-5106-branch-3.1.001.patch, YARN-5106-branch-3.1.001.patch, > YARN-5106-branch-3.1.002.patch, YARN-5106-branch-3.2.001.patch, > YARN-5106-branch-3.2.001.patch, YARN-5106-branch-3.2.002.patch, > YARN-5106.001.patch, YARN-5106.002.patch, YARN-5106.003.patch, > YARN-5106.004.patch, YARN-5106.005.patch, YARN-5106.006.patch, > YARN-5106.007.patch, YARN-5106.008.patch, YARN-5106.008.patch, > YARN-5106.008.patch, YARN-5106.009.patch, YARN-5106.010.patch, > YARN-5106.011.patch > > > Most, if not all, fair scheduler tests create an allocations XML file. Having > a helper class that potentially uses a builder would make the tests cleaner. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5106) Provide a builder interface for FairScheduler allocations for use in tests
[ https://issues.apache.org/jira/browse/YARN-5106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947596#comment-16947596 ] Zoltan Siegl commented on YARN-5106: Unit test failures are unrelated, they are constantly failing on branch-3.2 as descibed in https://issues.apache.org/jira/browse/YARN-9541 > Provide a builder interface for FairScheduler allocations for use in tests > -- > > Key: YARN-5106 > URL: https://issues.apache.org/jira/browse/YARN-5106 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Zoltan Siegl >Priority: Major > Labels: newbie++ > Attachments: YARN-5106-branch-3.1.001.patch, > YARN-5106-branch-3.1.001.patch, YARN-5106-branch-3.1.001.patch, > YARN-5106-branch-3.1.002.patch, YARN-5106-branch-3.2.001.patch, > YARN-5106-branch-3.2.001.patch, YARN-5106-branch-3.2.002.patch, > YARN-5106.001.patch, YARN-5106.002.patch, YARN-5106.003.patch, > YARN-5106.004.patch, YARN-5106.005.patch, YARN-5106.006.patch, > YARN-5106.007.patch, YARN-5106.008.patch, YARN-5106.008.patch, > YARN-5106.008.patch, YARN-5106.009.patch, YARN-5106.010.patch, > YARN-5106.011.patch > > > Most, if not all, fair scheduler tests create an allocations XML file. Having > a helper class that potentially uses a builder would make the tests cleaner. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9552) FairScheduler: NODE_UPDATE can cause NoSuchElementException
[
https://issues.apache.org/jira/browse/YARN-9552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947608#comment-16947608
]
Szilard Nemeth commented on YARN-9552:
--
Hi [~pbacsko]!
Committed your patches to branch-3.2 and branch-3.1 as well!
Thanks!
> FairScheduler: NODE_UPDATE can cause NoSuchElementException
> ---
>
> Key: YARN-9552
> URL: https://issues.apache.org/jira/browse/YARN-9552
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9552-001.patch, YARN-9552-002.patch,
> YARN-9552-003.patch, YARN-9552-004.patch, YARN-9552-branch-3.1.001.patch,
> YARN-9552-branch-3.1.002.patch, YARN-9552-branch-3.2.001.patch,
> YARN-9552-branch-3.2.002.patch, YARN-9552-branch-3.2.003.patch
>
>
> We observed a race condition inside YARN with the following stack trace:
> {noformat}
> 18/11/07 06:45:09.559 SchedulerEventDispatcher:Event Processor ERROR
> EventDispatcher: Error in handling event type NODE_UPDATE to the Event
> Dispatcher
> java.util.NoSuchElementException
> at
> java.util.concurrent.ConcurrentSkipListMap.firstKey(ConcurrentSkipListMap.java:2036)
> at
> java.util.concurrent.ConcurrentSkipListSet.first(ConcurrentSkipListSet.java:396)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.getNextPendingAsk(AppSchedulingInfo.java:373)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt.isOverAMShareLimit(FSAppAttempt.java:941)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt.assignContainer(FSAppAttempt.java:1373)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.assignContainer(FSLeafQueue.java:353)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.assignContainer(FSParentQueue.java:204)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.attemptScheduling(FairScheduler.java:1094)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.nodeUpdate(FairScheduler.java:961)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1183)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:132)
> at
> org.apache.hadoop.yarn.event.EventDispatcher$EventProcessor.run(EventDispatcher.java:66)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> This is basically the same as the one described in YARN-7382, but the root
> cause is different.
> When we create an application attempt, we create an {{FSAppAttempt}} object.
> This contains an {{AppSchedulingInfo}} which contains a set of
> {{SchedulerRequestKey}}. Initially, this set is empty and only initialized a
> bit later on a separate thread during a state transition:
> {noformat}
> 2019-05-07 15:58:02,659 INFO [RM StateStore dispatcher]
> recovery.RMStateStore (RMStateStore.java:transition(239)) - Storing info for
> app: application_1557237478804_0001
> 2019-05-07 15:58:02,684 INFO [RM Event dispatcher] rmapp.RMAppImpl
> (RMAppImpl.java:handle(903)) - application_1557237478804_0001 State change
> from NEW_SAVING to SUBMITTED on event = APP_NEW_SAVED
> 2019-05-07 15:58:02,690 INFO [SchedulerEventDispatcher:Event Processor]
> fair.FairScheduler (FairScheduler.java:addApplication(490)) - Accepted
> application application_1557237478804_0001 from user: bacskop, in queue:
> root.bacskop, currently num of applications: 1
> 2019-05-07 15:58:02,698 INFO [RM Event dispatcher] rmapp.RMAppImpl
> (RMAppImpl.java:handle(903)) - application_1557237478804_0001 State change
> from SUBMITTED to ACCEPTED on event = APP_ACCEPTED
> 2019-05-07 15:58:02,731 INFO [RM Event dispatcher]
> resourcemanager.ApplicationMasterService
> (ApplicationMasterService.java:registerAppAttempt(434)) - Registering app
> attempt : appattempt_1557237478804_0001_01
> 2019-05-07 15:58:02,732 INFO [RM Event dispatcher] attempt.RMAppAttemptImpl
> (RMAppAttemptImpl.java:handle(920)) - appattempt_1557237478804_0001_01
> State change from NEW to SUBMITTED on event = START
> 2019-05-07 15:58:02,746 INFO [SchedulerEventDispatcher:Event Processor]
> scheduler.SchedulerApplicationAttempt
> (SchedulerApplicationAttempt.java:(207)) - *** In the constructor of
> SchedulerApplicationAttempt
> 2019-05-07 15:58:02,747 INFO [SchedulerEventDispatcher:Event Processor]
> scheduler.SchedulerApplicationAttempt
> (SchedulerApplicationAttempt.java:(230)) - *** Contents of
> appSchedulin
[jira] [Created] (YARN-9883) Reshape SchedulerHealth class
Adam Antal created YARN-9883:
Summary: Reshape SchedulerHealth class
Key: YARN-9883
URL: https://issues.apache.org/jira/browse/YARN-9883
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager, yarn
Affects Versions: 3.3.0
Reporter: Adam Antal
The {{SchedulerHealth}} class has some flaws, for example:
- It has no javadoc at all
- All its objects are package-private: they should be private
- The internal maps should be (Concurrent) EnumMaps instead of HashMaps: they
are more efficient in storing Enums
- schedulerHealthDetails only stores the last operation, its name should
reflect that (just like lastSchedulerRunDetails)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9883) Reshape SchedulerHealth class
[
https://issues.apache.org/jira/browse/YARN-9883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kinga Marton reassigned YARN-9883:
--
Assignee: Kinga Marton
> Reshape SchedulerHealth class
> -
>
> Key: YARN-9883
> URL: https://issues.apache.org/jira/browse/YARN-9883
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager, yarn
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Assignee: Kinga Marton
>Priority: Minor
>
> The {{SchedulerHealth}} class has some flaws, for example:
> - It has no javadoc at all
> - All its objects are package-private: they should be private
> - The internal maps should be (Concurrent) EnumMaps instead of HashMaps: they
> are more efficient in storing Enums
> - schedulerHealthDetails only stores the last operation, its name should
> reflect that (just like lastSchedulerRunDetails)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9699) Migration tool that help to generate CS config based on FS config (Phase 1)
[ https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-9699: --- Attachment: YARN-9699-011.patch > Migration tool that help to generate CS config based on FS config (Phase 1) > --- > > Key: YARN-9699 > URL: https://issues.apache.org/jira/browse/YARN-9699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wanqiang Ji >Assignee: Peter Bacsko >Priority: Major > Attachments: FS_to_CS_migration_POC.patch, YARN-9699-003.patch, > YARN-9699-004.patch, YARN-9699-005.patch, YARN-9699-006.patch, > YARN-9699-007.patch, YARN-9699-008.patch, YARN-9699-009.patch, > YARN-9699-010.patch, YARN-9699-011.patch, YARN-9699.001.patch, > YARN-9699.002.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3890) FairScheduler should show the scheduler health metrics similar to ones added in CapacityScheduler
[
https://issues.apache.org/jira/browse/YARN-3890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947624#comment-16947624
]
Adam Antal commented on YARN-3890:
--
Overall looks very nice! Thanks for working on this.
Some comments:
- I see that {{SchedulerHealth$updateSchedulerReleaseDetails()}} is not called
by the FS after your patch. I think reporting those has also major importance.
Would you wire that as well to FS?
- I am not entirely convinced that testing is sufficient, so I suggest that
TestSchedulerHealth class should be parameterized to use FairScheduler as well
(along with CS) to ensure full integration. I'm also ok if you file a follow-up
jira and resolve it in that.
Question: Did you run it on a cluster, and verified that the HTML is rendered
properly? (One should do that before the merge of this patch.)
nits:
- Remove TODO from {{FairScheduler.java}}. Was there anything you were supposed
to put there?
- In {{TestFSAppAttempt.java}} remove wildcard import
- Decapitalize comments in {{TestFSAppAttempt$testHealthMetric}}
- Checkstyle
I also saw that {{SchedulerHealth}} class is in a really bad shape, I filed
YARN-9883 for the cleanup.
> FairScheduler should show the scheduler health metrics similar to ones added
> in CapacityScheduler
> -
>
> Key: YARN-3890
> URL: https://issues.apache.org/jira/browse/YARN-3890
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Reporter: Anubhav Dhoot
>Assignee: Zoltan Siegl
>Priority: Major
> Attachments: YARN-3890.001.patch, YARN-3890.002.patch,
> YARN-3890.003.patch
>
>
> We should add information displayed in YARN-3293 in FairScheduler as well
> possibly sharing the implementation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8453) Additional Unit tests to verify queue limit and max-limit with multiple resource types
[ https://issues.apache.org/jira/browse/YARN-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947628#comment-16947628 ] Szilard Nemeth commented on YARN-8453: -- Hi [~sunilg]! Couple of comments: # Could you rename static field A to QUEUE_A? Similarly with B -> QUEUE_B as well? # A_CAPACITY and B_CAPACITY could be private. # There are some calls to TestUtils.createResource: These can be fit into the previous line. # // Register NM1 with 10GB memory, 4 CPU and 0 GPU --> This comment should be about NM2. # nm2 variable is not used, please invoke registerNode, but don't assign it to a variable. # // Allocate one more containers which doesnt need GPU --> containers should be singular # There are 2 occasions where you use Arrays.asList with just one element: You could use Collections.singletonList() instead. # Can you extract some common code with testMaximumAllocationRefreshWithMultipleResourceTypes and your new test method? Thanks! > Additional Unit tests to verify queue limit and max-limit with multiple > resource types > --- > > Key: YARN-8453 > URL: https://issues.apache.org/jira/browse/YARN-8453 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Affects Versions: 3.0.2 >Reporter: Sunil G >Assignee: Sunil G >Priority: Major > Attachments: YARN-8453.001.patch > > > Post support of additional resource types other then CPU and Memory, it could > be possible that one such new resource is exhausted its quota on a given > queue. But other resources such as Memory / CPU is still there beyond its > guaranteed limit (under max-limit). Adding more units test to ensure we are > not starving such allocation requests -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8453) Additional Unit tests to verify queue limit and max-limit with multiple resource types
[ https://issues.apache.org/jira/browse/YARN-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947630#comment-16947630 ] Adam Antal commented on YARN-8453: -- If you don't have much time, I'd like to take this over [~sunilg]. > Additional Unit tests to verify queue limit and max-limit with multiple > resource types > --- > > Key: YARN-8453 > URL: https://issues.apache.org/jira/browse/YARN-8453 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Affects Versions: 3.0.2 >Reporter: Sunil G >Assignee: Sunil G >Priority: Major > Attachments: YARN-8453.001.patch > > > Post support of additional resource types other then CPU and Memory, it could > be possible that one such new resource is exhausted its quota on a given > queue. But other resources such as Memory / CPU is still there beyond its > guaranteed limit (under max-limit). Adding more units test to ensure we are > not starving such allocation requests -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9840) Capacity scheduler: add support for Secondary Group rule mapping
[
https://issues.apache.org/jira/browse/YARN-9840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947634#comment-16947634
]
Hadoop QA commented on YARN-9840:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
39s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 40s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 41s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 85m
38s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}143m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.3 Server=19.03.3 Image:yetus/hadoop:1dde3efb91e |
| JIRA Issue | YARN-9840 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12982508/YARN-9840.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 60c6e5cdbd74 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24
10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / be901f4 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24943/testReport/ |
| Max. process+thread count | 833 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24943/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Capacity scheduler: add support for S
[jira] [Commented] (YARN-9840) Capacity scheduler: add support for Secondary Group rule mapping
[
https://issues.apache.org/jira/browse/YARN-9840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947650#comment-16947650
]
Peter Bacsko commented on YARN-9840:
+1 (non-binding). [~snemeth] please review & commit patch v3 if you think it's
OK.
> Capacity scheduler: add support for Secondary Group rule mapping
>
>
> Key: YARN-9840
> URL: https://issues.apache.org/jira/browse/YARN-9840
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Manikandan R
>Priority: Major
> Attachments: YARN-9840.001.patch, YARN-9840.002.patch,
> YARN-9840.003.patch
>
>
> Currently, Capacity Scheduler only supports primary group rule mapping like
> this:
> {{u:%user:%primary_group}}
> Fair scheduler already supports secondary group placement rule. Let's add
> this to CS to reduce the feature gap.
> Class of interest:
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9841) Capacity scheduler: add support for combined %user + %primary_group mapping
[
https://issues.apache.org/jira/browse/YARN-9841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947653#comment-16947653
]
Hadoop QA commented on YARN-9841:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m
2s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 38s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 9s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 84m
29s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}155m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.3 Server=19.03.3 Image:yetus/hadoop:1dde3efb91e |
| JIRA Issue | YARN-9841 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12982509/YARN-9841.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 3b2d970edcea 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / be901f4 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24944/testReport/ |
| Max. process+thread count | 814 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24944/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Capacity scheduler: add support for co
[jira] [Commented] (YARN-9841) Capacity scheduler: add support for combined %user + %primary_group mapping
[
https://issues.apache.org/jira/browse/YARN-9841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947654#comment-16947654
]
Peter Bacsko commented on YARN-9841:
Thanks for the patch [~maniraj...@gmail.com]. I'm not sure whether this
changeset conflicts with YARN-9840 so let's wait until that one gets committed.
Just a minor thing related to tests:
{noformat}
Assert.assertEquals("Expected Queue is ", expectedQueue,
ctx != null ? ctx.getQueue() : inputQueue);
{noformat}
You don't have to write "Expected Queue is ", because JUnit4 itself prints a
similar message if there's an assertion failure. It's enough to write
{{Assert.assertEquals("Queue", ...)}}.
Could you pls modify this? Otherwise it looks good.
> Capacity scheduler: add support for combined %user + %primary_group mapping
> ---
>
> Key: YARN-9841
> URL: https://issues.apache.org/jira/browse/YARN-9841
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Manikandan R
>Priority: Major
> Attachments: YARN-9841.001.patch, YARN-9841.001.patch,
> YARN-9841.002.patch, YARN-9841.003.patch, YARN-9841.junit.patch
>
>
> Right now in CS, using {{%primary_group}} with a parent queue is only
> possible this way:
> {{u:%user:parentqueue.%primary_group}}
> Looking at
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java,
> we cannot do something like:
> {{u:%user:%primary_group.%user}}
> Fair Scheduler supports a nested rule where such a placement/mapping rule is
> possible. This improvement would reduce this feature gap.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9841) Capacity scheduler: add support for combined %user + %primary_group mapping
[
https://issues.apache.org/jira/browse/YARN-9841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947702#comment-16947702
]
Manikandan R commented on YARN-9841:
Thanks [~pbacsko]. Attached .004.patch.
> Capacity scheduler: add support for combined %user + %primary_group mapping
> ---
>
> Key: YARN-9841
> URL: https://issues.apache.org/jira/browse/YARN-9841
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Manikandan R
>Priority: Major
> Attachments: YARN-9841.001.patch, YARN-9841.001.patch,
> YARN-9841.002.patch, YARN-9841.003.patch, YARN-9841.004.patch,
> YARN-9841.junit.patch
>
>
> Right now in CS, using {{%primary_group}} with a parent queue is only
> possible this way:
> {{u:%user:parentqueue.%primary_group}}
> Looking at
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java,
> we cannot do something like:
> {{u:%user:%primary_group.%user}}
> Fair Scheduler supports a nested rule where such a placement/mapping rule is
> possible. This improvement would reduce this feature gap.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9841) Capacity scheduler: add support for combined %user + %primary_group mapping
[
https://issues.apache.org/jira/browse/YARN-9841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikandan R updated YARN-9841:
---
Attachment: YARN-9841.004.patch
> Capacity scheduler: add support for combined %user + %primary_group mapping
> ---
>
> Key: YARN-9841
> URL: https://issues.apache.org/jira/browse/YARN-9841
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Manikandan R
>Priority: Major
> Attachments: YARN-9841.001.patch, YARN-9841.001.patch,
> YARN-9841.002.patch, YARN-9841.003.patch, YARN-9841.004.patch,
> YARN-9841.junit.patch
>
>
> Right now in CS, using {{%primary_group}} with a parent queue is only
> possible this way:
> {{u:%user:parentqueue.%primary_group}}
> Looking at
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java,
> we cannot do something like:
> {{u:%user:%primary_group.%user}}
> Fair Scheduler supports a nested rule where such a placement/mapping rule is
> possible. This improvement would reduce this feature gap.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9699) Migration tool that help to generate CS config based on FS config (Phase 1)
[ https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-9699: - Attachment: YARN-9699-012.patch > Migration tool that help to generate CS config based on FS config (Phase 1) > --- > > Key: YARN-9699 > URL: https://issues.apache.org/jira/browse/YARN-9699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wanqiang Ji >Assignee: Peter Bacsko >Priority: Major > Attachments: FS_to_CS_migration_POC.patch, YARN-9699-003.patch, > YARN-9699-004.patch, YARN-9699-005.patch, YARN-9699-006.patch, > YARN-9699-007.patch, YARN-9699-008.patch, YARN-9699-009.patch, > YARN-9699-010.patch, YARN-9699-011.patch, YARN-9699-012.patch, > YARN-9699.001.patch, YARN-9699.002.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9699) Migration tool that help to generate CS config based on FS config (Phase 1)
[ https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-9699: - Attachment: YARN-9699-013.patch > Migration tool that help to generate CS config based on FS config (Phase 1) > --- > > Key: YARN-9699 > URL: https://issues.apache.org/jira/browse/YARN-9699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wanqiang Ji >Assignee: Peter Bacsko >Priority: Major > Attachments: FS_to_CS_migration_POC.patch, YARN-9699-003.patch, > YARN-9699-004.patch, YARN-9699-005.patch, YARN-9699-006.patch, > YARN-9699-007.patch, YARN-9699-008.patch, YARN-9699-009.patch, > YARN-9699-010.patch, YARN-9699-011.patch, YARN-9699-012.patch, > YARN-9699-013.patch, YARN-9699.001.patch, YARN-9699.002.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-8453) Additional Unit tests to verify queue limit and max-limit with multiple resource types
[ https://issues.apache.org/jira/browse/YARN-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth reassigned YARN-8453: Assignee: Szilard Nemeth (was: Sunil G) > Additional Unit tests to verify queue limit and max-limit with multiple > resource types > --- > > Key: YARN-8453 > URL: https://issues.apache.org/jira/browse/YARN-8453 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Affects Versions: 3.0.2 >Reporter: Sunil G >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-8453.001.patch > > > Post support of additional resource types other then CPU and Memory, it could > be possible that one such new resource is exhausted its quota on a given > queue. But other resources such as Memory / CPU is still there beyond its > guaranteed limit (under max-limit). Adding more units test to ensure we are > not starving such allocation requests -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-8453) Additional Unit tests to verify queue limit and max-limit with multiple resource types
[ https://issues.apache.org/jira/browse/YARN-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth reassigned YARN-8453: Assignee: Adam Antal (was: Szilard Nemeth) > Additional Unit tests to verify queue limit and max-limit with multiple > resource types > --- > > Key: YARN-8453 > URL: https://issues.apache.org/jira/browse/YARN-8453 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Affects Versions: 3.0.2 >Reporter: Sunil G >Assignee: Adam Antal >Priority: Major > Attachments: YARN-8453.001.patch > > > Post support of additional resource types other then CPU and Memory, it could > be possible that one such new resource is exhausted its quota on a given > queue. But other resources such as Memory / CPU is still there beyond its > guaranteed limit (under max-limit). Adding more units test to ensure we are > not starving such allocation requests -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9699) Migration tool that help to generate CS config based on FS config (Phase 1)
[
https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947793#comment-16947793
]
Hadoop QA commented on YARN-9699:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
54s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 16 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 21s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
15s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 17s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 18 new + 137 unchanged - 7 fixed = 155 total (was 144) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} xml {color} | {color:red} 0m 10s{color} |
{color:red} The patch has 2 ill-formed XML file(s). {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 7s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
25s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 87m
25s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
22s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
42s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}167m 23s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| XML | Parsing Error(s): |
| |
hadoop-yarn-project/hadoop-yarn/hadoop-yar
[jira] [Commented] (YARN-9841) Capacity scheduler: add support for combined %user + %primary_group mapping
[
https://issues.apache.org/jira/browse/YARN-9841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947829#comment-16947829
]
Hadoop QA commented on YARN-9841:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
31s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 34s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 26s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 81m 31s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}133m 58s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.3 Server=19.03.3 Image:yetus/hadoop:1dde3efb91e |
| JIRA Issue | YARN-9841 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12982580/YARN-9841.004.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 68a0d5463fac 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 4b0a5bc |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/24947/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24947/testReport/ |
| Max. process+thread count | 811 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/
[jira] [Commented] (YARN-9699) Migration tool that help to generate CS config based on FS config (Phase 1)
[
https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947843#comment-16947843
]
Hadoop QA commented on YARN-9699:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
46s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 16 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
52s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 16s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
37s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 22s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 18 new + 137 unchanged - 7 fixed = 155 total (was 144) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} xml {color} | {color:red} 0m 9s{color} |
{color:red} The patch has 2 ill-formed XML file(s). {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 9s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
29s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 86m
25s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
22s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
40s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}163m 58s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| XML | Parsing Error(s): |
| |
hadoop-yarn-project/hadoop-yarn/hadoop-yar
[jira] [Commented] (YARN-9699) Migration tool that help to generate CS config based on FS config (Phase 1)
[
https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947844#comment-16947844
]
Hadoop QA commented on YARN-9699:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 16 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
59s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 52s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
10s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 19s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 18 new + 136 unchanged - 7 fixed = 154 total (was 143) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} xml {color} | {color:red} 0m 10s{color} |
{color:red} The patch has 2 ill-formed XML file(s). {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 18s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
28s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 82m 4s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
27s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
50s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}156m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| XML | Parsing Error(s): |
| |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/
[jira] [Commented] (YARN-9699) Migration tool that help to generate CS config based on FS config (Phase 1)
[
https://issues.apache.org/jira/browse/YARN-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947855#comment-16947855
]
Hadoop QA commented on YARN-9699:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 16 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 59s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
18s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 24s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 18 new + 136 unchanged - 7 fixed = 154 total (was 143) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} xml {color} | {color:red} 0m 10s{color} |
{color:red} The patch has 2 ill-formed XML file(s). {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 18s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
28s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 81m 47s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
30s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
47s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}156m 10s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| XML | Parsing Error(s): |
| |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/
[jira] [Commented] (YARN-9552) FairScheduler: NODE_UPDATE can cause NoSuchElementException
[
https://issues.apache.org/jira/browse/YARN-9552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947857#comment-16947857
]
Steve Loughran commented on YARN-9552:
--
trunk and branch-3.2 doesn't build; this looks suspiciously like the patch at
fault. Reopening -can someone look @ this
> FairScheduler: NODE_UPDATE can cause NoSuchElementException
> ---
>
> Key: YARN-9552
> URL: https://issues.apache.org/jira/browse/YARN-9552
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Fix For: 3.3.0, 3.2.2, 3.1.4
>
> Attachments: YARN-9552-001.patch, YARN-9552-002.patch,
> YARN-9552-003.patch, YARN-9552-004.patch, YARN-9552-branch-3.1.001.patch,
> YARN-9552-branch-3.1.002.patch, YARN-9552-branch-3.2.001.patch,
> YARN-9552-branch-3.2.002.patch, YARN-9552-branch-3.2.003.patch
>
>
> We observed a race condition inside YARN with the following stack trace:
> {noformat}
> 18/11/07 06:45:09.559 SchedulerEventDispatcher:Event Processor ERROR
> EventDispatcher: Error in handling event type NODE_UPDATE to the Event
> Dispatcher
> java.util.NoSuchElementException
> at
> java.util.concurrent.ConcurrentSkipListMap.firstKey(ConcurrentSkipListMap.java:2036)
> at
> java.util.concurrent.ConcurrentSkipListSet.first(ConcurrentSkipListSet.java:396)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.getNextPendingAsk(AppSchedulingInfo.java:373)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt.isOverAMShareLimit(FSAppAttempt.java:941)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt.assignContainer(FSAppAttempt.java:1373)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.assignContainer(FSLeafQueue.java:353)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.assignContainer(FSParentQueue.java:204)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.attemptScheduling(FairScheduler.java:1094)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.nodeUpdate(FairScheduler.java:961)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1183)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:132)
> at
> org.apache.hadoop.yarn.event.EventDispatcher$EventProcessor.run(EventDispatcher.java:66)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> This is basically the same as the one described in YARN-7382, but the root
> cause is different.
> When we create an application attempt, we create an {{FSAppAttempt}} object.
> This contains an {{AppSchedulingInfo}} which contains a set of
> {{SchedulerRequestKey}}. Initially, this set is empty and only initialized a
> bit later on a separate thread during a state transition:
> {noformat}
> 2019-05-07 15:58:02,659 INFO [RM StateStore dispatcher]
> recovery.RMStateStore (RMStateStore.java:transition(239)) - Storing info for
> app: application_1557237478804_0001
> 2019-05-07 15:58:02,684 INFO [RM Event dispatcher] rmapp.RMAppImpl
> (RMAppImpl.java:handle(903)) - application_1557237478804_0001 State change
> from NEW_SAVING to SUBMITTED on event = APP_NEW_SAVED
> 2019-05-07 15:58:02,690 INFO [SchedulerEventDispatcher:Event Processor]
> fair.FairScheduler (FairScheduler.java:addApplication(490)) - Accepted
> application application_1557237478804_0001 from user: bacskop, in queue:
> root.bacskop, currently num of applications: 1
> 2019-05-07 15:58:02,698 INFO [RM Event dispatcher] rmapp.RMAppImpl
> (RMAppImpl.java:handle(903)) - application_1557237478804_0001 State change
> from SUBMITTED to ACCEPTED on event = APP_ACCEPTED
> 2019-05-07 15:58:02,731 INFO [RM Event dispatcher]
> resourcemanager.ApplicationMasterService
> (ApplicationMasterService.java:registerAppAttempt(434)) - Registering app
> attempt : appattempt_1557237478804_0001_01
> 2019-05-07 15:58:02,732 INFO [RM Event dispatcher] attempt.RMAppAttemptImpl
> (RMAppAttemptImpl.java:handle(920)) - appattempt_1557237478804_0001_01
> State change from NEW to SUBMITTED on event = START
> 2019-05-07 15:58:02,746 INFO [SchedulerEventDispatcher:Event Processor]
> scheduler.SchedulerApplicationAttempt
> (SchedulerApplicationAttempt.java:(207)) - *** In the constructor of
> SchedulerApplicationAttempt
> 2019-05-07 15:58:02,747 INFO [SchedulerEventDispatcher:Event Processor]
> scheduler.SchedulerApplicationAttempt
> (SchedulerApplicationAtte
[jira] [Commented] (YARN-9552) FairScheduler: NODE_UPDATE can cause NoSuchElementException
[
https://issues.apache.org/jira/browse/YARN-9552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947862#comment-16947862
]
Peter Bacsko commented on YARN-9552:
It's been on trunk since May, so it should be okay there.
> FairScheduler: NODE_UPDATE can cause NoSuchElementException
> ---
>
> Key: YARN-9552
> URL: https://issues.apache.org/jira/browse/YARN-9552
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Fix For: 3.3.0, 3.2.2, 3.1.4
>
> Attachments: YARN-9552-001.patch, YARN-9552-002.patch,
> YARN-9552-003.patch, YARN-9552-004.patch, YARN-9552-branch-3.1.001.patch,
> YARN-9552-branch-3.1.002.patch, YARN-9552-branch-3.2.001.patch,
> YARN-9552-branch-3.2.002.patch, YARN-9552-branch-3.2.003.patch
>
>
> We observed a race condition inside YARN with the following stack trace:
> {noformat}
> 18/11/07 06:45:09.559 SchedulerEventDispatcher:Event Processor ERROR
> EventDispatcher: Error in handling event type NODE_UPDATE to the Event
> Dispatcher
> java.util.NoSuchElementException
> at
> java.util.concurrent.ConcurrentSkipListMap.firstKey(ConcurrentSkipListMap.java:2036)
> at
> java.util.concurrent.ConcurrentSkipListSet.first(ConcurrentSkipListSet.java:396)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.getNextPendingAsk(AppSchedulingInfo.java:373)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt.isOverAMShareLimit(FSAppAttempt.java:941)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt.assignContainer(FSAppAttempt.java:1373)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.assignContainer(FSLeafQueue.java:353)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.assignContainer(FSParentQueue.java:204)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.attemptScheduling(FairScheduler.java:1094)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.nodeUpdate(FairScheduler.java:961)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1183)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:132)
> at
> org.apache.hadoop.yarn.event.EventDispatcher$EventProcessor.run(EventDispatcher.java:66)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> This is basically the same as the one described in YARN-7382, but the root
> cause is different.
> When we create an application attempt, we create an {{FSAppAttempt}} object.
> This contains an {{AppSchedulingInfo}} which contains a set of
> {{SchedulerRequestKey}}. Initially, this set is empty and only initialized a
> bit later on a separate thread during a state transition:
> {noformat}
> 2019-05-07 15:58:02,659 INFO [RM StateStore dispatcher]
> recovery.RMStateStore (RMStateStore.java:transition(239)) - Storing info for
> app: application_1557237478804_0001
> 2019-05-07 15:58:02,684 INFO [RM Event dispatcher] rmapp.RMAppImpl
> (RMAppImpl.java:handle(903)) - application_1557237478804_0001 State change
> from NEW_SAVING to SUBMITTED on event = APP_NEW_SAVED
> 2019-05-07 15:58:02,690 INFO [SchedulerEventDispatcher:Event Processor]
> fair.FairScheduler (FairScheduler.java:addApplication(490)) - Accepted
> application application_1557237478804_0001 from user: bacskop, in queue:
> root.bacskop, currently num of applications: 1
> 2019-05-07 15:58:02,698 INFO [RM Event dispatcher] rmapp.RMAppImpl
> (RMAppImpl.java:handle(903)) - application_1557237478804_0001 State change
> from SUBMITTED to ACCEPTED on event = APP_ACCEPTED
> 2019-05-07 15:58:02,731 INFO [RM Event dispatcher]
> resourcemanager.ApplicationMasterService
> (ApplicationMasterService.java:registerAppAttempt(434)) - Registering app
> attempt : appattempt_1557237478804_0001_01
> 2019-05-07 15:58:02,732 INFO [RM Event dispatcher] attempt.RMAppAttemptImpl
> (RMAppAttemptImpl.java:handle(920)) - appattempt_1557237478804_0001_01
> State change from NEW to SUBMITTED on event = START
> 2019-05-07 15:58:02,746 INFO [SchedulerEventDispatcher:Event Processor]
> scheduler.SchedulerApplicationAttempt
> (SchedulerApplicationAttempt.java:(207)) - *** In the constructor of
> SchedulerApplicationAttempt
> 2019-05-07 15:58:02,747 INFO [SchedulerEventDispatcher:Event Processor]
> scheduler.SchedulerApplicationAttempt
> (SchedulerApplicationAttempt.java:(230)) - *** Contents of
> appSchedulingInfo: []
> 2019-0
[jira] [Commented] (YARN-9128) Use SerializationUtils from apache commons to serialize / deserialize ResourceMappings
[ https://issues.apache.org/jira/browse/YARN-9128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947907#comment-16947907 ] Szilard Nemeth commented on YARN-9128: -- Hi [~zsiegl] ! Reverted YARN-9128 commits on branch-3.2 / branch-3.1 as it broke the build on these branches! The compilation failure was caused by that 3.2 / 3.1 don't have this class: org.apache.hadoop.yarn.server.nodemanager.api.deviceplugin.Device, which is being used in TestResourceMappings. [~zsiegl]: Please try to come up some other way to test AssignedResources#fromBytes / toBytes: Either with GpuDevice or FpgaDevice or any other artifical device object for tests. Please also assess how important to backport the change to branch-3.2 / branch-3.1. If you decice to backport it, please post the patches so we have green Jenkins results for both. Thanks! > Use SerializationUtils from apache commons to serialize / deserialize > ResourceMappings > -- > > Key: YARN-9128 > URL: https://issues.apache.org/jira/browse/YARN-9128 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Zoltan Siegl >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9128.001.patch, YARN-9128.002.patch, > YARN-9128.003.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9873) Mutation API Config Change need to update Version Number
[ https://issues.apache.org/jira/browse/YARN-9873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948129#comment-16948129 ] Akira Ajisaka commented on YARN-9873: - This broke branch-3.2 build. YARN-7243 changed the logging API from commons-logging to slf4j in trunk, but not in branch-3.2. I'll try to cherry-pick YARN-7243 to branch-3.2. > Mutation API Config Change need to update Version Number > - > > Key: YARN-9873 > URL: https://issues.apache.org/jira/browse/YARN-9873 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Fix For: 3.3.0, 3.2.2 > > Attachments: YARN-9873-001.patch, YARN-9873-002.patch, > YARN-9873-003.patch, YARN-9873-004.patch > > > Version Number support for each Scheduler Config Change. This also helps to > know when the last change happened. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7243) Moving logging APIs over to slf4j in hadoop-yarn-server-resourcemanager
[ https://issues.apache.org/jira/browse/YARN-7243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948144#comment-16948144 ] Akira Ajisaka commented on YARN-7243: - Created https://github.com/apache/hadoop/pull/1634 to backport this to branch-3.2. > Moving logging APIs over to slf4j in hadoop-yarn-server-resourcemanager > --- > > Key: YARN-7243 > URL: https://issues.apache.org/jira/browse/YARN-7243 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Yeliang Cang >Assignee: Prabhu Joseph >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-7243-007.patch, YARN-7243-008.patch, > YARN-7243-009.patch, YARN-7243.001.patch, YARN-7243.002.patch, > YARN-7243.003.patch, YARN-7243.004.patch, YARN-7243.005.patch, > YARN-7243.006.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9880) In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is incorrect.
[ https://issues.apache.org/jira/browse/YARN-9880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jenny updated YARN-9880: Description: In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is incorrect. When Browser and Yarn server are in different time zones, the Application Attempt's ElapsedTime is greater than the Application's ElapsedTime. I found this problem where hadoop3.1.1 was used, want to upgrade to the latest version of hadoop, but found the running Application Attempt's ElapsedTime is also incorrect. In UI1, get ElapsedTime from yarn server. In UI2, get currentTime from brower. Therefore, when the brower and the yarn server are in different time zones, the ElapsedTime is incorrect. While the Application‘s ElapsedTime is 29 Secs, the Application Attempt's ElapsedTime is 10 Hrs : 49 Mins : 55 Secs. was: In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is incorrect. When Browser and Yarn server are in different time zones, the Application Attempt's ElapsedTime is greater than the Application's ElapsedTime. > In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is > incorrect. > - > > Key: YARN-9880 > URL: https://issues.apache.org/jira/browse/YARN-9880 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Affects Versions: 3.1.1, 3.2.1 >Reporter: jenny >Priority: Major > Attachments: image-2019-10-09-17-22-31-603.png > > > In YARN ui2 attempts tab, The running Application Attempt's ElapsedTime is > incorrect. > When Browser and Yarn server are in different time zones, the Application > Attempt's ElapsedTime is greater than the Application's ElapsedTime. > I found this problem where hadoop3.1.1 was used, want to upgrade to the > latest version of hadoop, but found the running Application Attempt's > ElapsedTime is also incorrect. > In UI1, get ElapsedTime from yarn server. In UI2, get currentTime from > brower. Therefore, when the brower and the yarn server are in different time > zones, the ElapsedTime is incorrect. While the Application‘s ElapsedTime is > 29 Secs, the Application Attempt's ElapsedTime is 10 Hrs : 49 Mins : 55 Secs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7243) Moving logging APIs over to slf4j in hadoop-yarn-server-resourcemanager
[ https://issues.apache.org/jira/browse/YARN-7243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated YARN-7243: Fix Version/s: 3.2.2 Backported to branch-3.2. > Moving logging APIs over to slf4j in hadoop-yarn-server-resourcemanager > --- > > Key: YARN-7243 > URL: https://issues.apache.org/jira/browse/YARN-7243 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Yeliang Cang >Assignee: Prabhu Joseph >Priority: Major > Fix For: 3.3.0, 3.2.2 > > Attachments: YARN-7243-007.patch, YARN-7243-008.patch, > YARN-7243-009.patch, YARN-7243.001.patch, YARN-7243.002.patch, > YARN-7243.003.patch, YARN-7243.004.patch, YARN-7243.005.patch, > YARN-7243.006.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9873) Mutation API Config Change need to update Version Number
[ https://issues.apache.org/jira/browse/YARN-9873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948249#comment-16948249 ] Akira Ajisaka commented on YARN-9873: - Cherry-picked. > Mutation API Config Change need to update Version Number > - > > Key: YARN-9873 > URL: https://issues.apache.org/jira/browse/YARN-9873 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Fix For: 3.3.0, 3.2.2 > > Attachments: YARN-9873-001.patch, YARN-9873-002.patch, > YARN-9873-003.patch, YARN-9873-004.patch > > > Version Number support for each Scheduler Config Change. This also helps to > know when the last change happened. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
