[jira] [Commented] (YARN-8448) AM HTTPS Support
[
https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560586#comment-16560586
]
genericqa commented on YARN-8448:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 11 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 28m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
40s{color} | {color:red} server-scm in trunk failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
36s{color} | {color:red} ozone-manager in trunk failed. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
21m 53s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
30s{color} | {color:red} server-scm in trunk failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
29s{color} | {color:red} ozone-manager in trunk failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
17s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
16s{color} | {color:red} server-scm in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
15s{color} | {color:red} ozone-manager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
3s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 27m 3s{color} |
{color:red} root generated 1 new + 11 unchanged - 0 fixed = 12 total (was 11)
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 27m 3s{color}
| {color:red} root generated 11 new + 1458 unchanged - 10 fixed = 1469 total
(was 1468) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
31s{color} | {color:red} server-scm in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
29s{color} | {color:red} ozone-manager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 4 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
18s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 10m
57s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
15s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
28s{color} | {color:red}
[jira] [Commented] (YARN-8588) Logging improvements for better debuggability
[
https://issues.apache.org/jira/browse/YARN-8588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560548#comment-16560548
]
genericqa commented on YARN-8588:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 26s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
42s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch
failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 45s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
15s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 75m
3s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}130m 3s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs |
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| | Redundant nullcheck of StringBuilder.toString(), which is known to be
non-null in
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.QueueManagementChange.toString()
Redundant null check at QueueManagementChange.java:is known to be non-null in
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.QueueManagementChange.toString()
Redundant null check at QueueManagementChange.java:[line 127] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | YARN-8588 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12933447/YARN-8588.1.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 3ca09639dc09 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14
08:53:28 UTC 2018 x86_64 x86_64
[jira] [Commented] (YARN-8600) RegistryDNS hang when remote lookup does not reply
[
https://issues.apache.org/jira/browse/YARN-8600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560536#comment-16560536
]
genericqa commented on YARN-8600:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 34s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 12s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
35s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
38s{color} | {color:green} hadoop-yarn-registry in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 49m 11s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry |
| | org.apache.hadoop.registry.server.dns.LookupTask.getResults() may expose
internal representation by returning LookupTask.records At LookupTask.java:by
returning LookupTask.records At LookupTask.java:[line 40] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | YARN-8600 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12933453/YARN-8600.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 4fa52ee0cdd0 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31
19:38:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 59adeb8 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1
[jira] [Commented] (YARN-8579) New AM attempt could not retrieve previous attempt component data
[
https://issues.apache.org/jira/browse/YARN-8579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560530#comment-16560530
]
genericqa commented on YARN-8579:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 27m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 33s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 19s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 71m 24s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 12m
32s{color} | {color:green} hadoop-yarn-services-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}165m 37s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart |
| | hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | YARN-8579 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12933440/YARN-8579.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux f670b595b9ac 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 2cccf40 |
| maven | version: Apache Maven 3.3.9 |
|
[jira] [Assigned] (YARN-8600) RegistryDNS hang when remote lookup does not reply
[ https://issues.apache.org/jira/browse/YARN-8600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang reassigned YARN-8600: --- Assignee: Eric Yang > RegistryDNS hang when remote lookup does not reply > -- > > Key: YARN-8600 > URL: https://issues.apache.org/jira/browse/YARN-8600 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 3.1.0 >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: YARN-8600.001.patch > > > If lookup type mismatch with the record to query, remote DNS server might not > reply. For example looking up a CNAME record with a PTR address: > 1.76.27.172.in-addr.arpa. This can hang registryDNS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8600) RegistryDNS hang when remote lookup does not reply
[ https://issues.apache.org/jira/browse/YARN-8600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-8600: Attachment: YARN-8600.001.patch > RegistryDNS hang when remote lookup does not reply > -- > > Key: YARN-8600 > URL: https://issues.apache.org/jira/browse/YARN-8600 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 3.1.0 >Reporter: Eric Yang >Priority: Major > Attachments: YARN-8600.001.patch > > > If lookup type mismatch with the record to query, remote DNS server might not > reply. For example looking up a CNAME record with a PTR address: > 1.76.27.172.in-addr.arpa. This can hang registryDNS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8600) RegistryDNS hang when remote lookup does not reply
Eric Yang created YARN-8600: --- Summary: RegistryDNS hang when remote lookup does not reply Key: YARN-8600 URL: https://issues.apache.org/jira/browse/YARN-8600 Project: Hadoop YARN Issue Type: Bug Components: yarn Affects Versions: 3.1.0 Reporter: Eric Yang If lookup type mismatch with the record to query, remote DNS server might not reply. For example looking up a CNAME record with a PTR address: 1.76.27.172.in-addr.arpa. This can hang registryDNS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8508) GPU does not get released even though the container is killed
[
https://issues.apache.org/jira/browse/YARN-8508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560502#comment-16560502
]
Hudson commented on YARN-8508:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14659 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/14659/])
YARN-8508. Release GPU resource for killed container. (eyang: rev
ed9d60e888d0acfd748fda7f66249f5b79a3ed6d)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutor.java
> GPU does not get released even though the container is killed
> --
>
> Key: YARN-8508
> URL: https://issues.apache.org/jira/browse/YARN-8508
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Sumana Sathish
>Assignee: Chandni Singh
>Priority: Major
> Fix For: 3.2.0, 3.1.2
>
> Attachments: YARN-8505.001.patch, YARN-8505.002.patch
>
>
> GPU failed to release even though the container using it is being killed
> {Code}
> 2018-07-06 05:22:26,201 INFO container.ContainerImpl
> (ContainerImpl.java:handle(2093)) - Container
> container_e20_1530854311763_0006_01_01 transitioned from RUNNING to
> KILLING
> 2018-07-06 05:22:26,250 INFO container.ContainerImpl
> (ContainerImpl.java:handle(2093)) - Container
> container_e20_1530854311763_0006_01_02 transitioned from RUNNING to
> KILLING
> 2018-07-06 05:22:26,251 INFO application.ApplicationImpl
> (ApplicationImpl.java:handle(632)) - Application
> application_1530854311763_0006 transitioned from RUNNING to
> FINISHING_CONTAINERS_WAIT
> 2018-07-06 05:22:26,251 INFO launcher.ContainerLaunch
> (ContainerLaunch.java:cleanupContainer(734)) - Cleaning up container
> container_e20_1530854311763_0006_01_02
> 2018-07-06 05:22:31,358 INFO launcher.ContainerLaunch
> (ContainerLaunch.java:getContainerPid(1102)) - Could not get pid for
> container_e20_1530854311763_0006_01_02. Waited for 5000 ms.
> 2018-07-06 05:22:31,358 WARN launcher.ContainerLaunch
> (ContainerLaunch.java:cleanupContainer(784)) - Container clean up before pid
> file created container_e20_1530854311763_0006_01_02
> 2018-07-06 05:22:31,359 INFO launcher.ContainerLaunch
> (ContainerLaunch.java:reapDockerContainerNoPid(940)) - Unable to obtain pid,
> but docker container request detected. Attempting to reap container
> container_e20_1530854311763_0006_01_02
> 2018-07-06 05:22:31,494 INFO nodemanager.LinuxContainerExecutor
> (LinuxContainerExecutor.java:deleteAsUser(828)) - Deleting absolute path :
> /grid/0/hadoop/yarn/local/usercache/hrt_qa/appcache/application_1530854311763_0006/container_e20_1530854311763_0006_01_02/launch_container.sh
> 2018-07-06 05:22:31,500 INFO nodemanager.LinuxContainerExecutor
> (LinuxContainerExecutor.java:deleteAsUser(828)) - Deleting absolute path :
> /grid/0/hadoop/yarn/local/usercache/hrt_qa/appcache/application_1530854311763_0006/container_e20_1530854311763_0006_01_02/container_tokens
> 2018-07-06 05:22:31,510 INFO container.ContainerImpl
> (ContainerImpl.java:handle(2093)) - Container
> container_e20_1530854311763_0006_01_01 transitioned from KILLING to
> CONTAINER_CLEANEDUP_AFTER_KILL
> 2018-07-06 05:22:31,510 INFO container.ContainerImpl
> (ContainerImpl.java:handle(2093)) - Container
> container_e20_1530854311763_0006_01_02 transitioned from KILLING to
> CONTAINER_CLEANEDUP_AFTER_KILL
> 2018-07-06 05:22:31,512 INFO container.ContainerImpl
> (ContainerImpl.java:handle(2093)) - Container
> container_e20_1530854311763_0006_01_01 transitioned from
> CONTAINER_CLEANEDUP_AFTER_KILL to DONE
> 2018-07-06 05:22:31,513 INFO container.ContainerImpl
> (ContainerImpl.java:handle(2093)) - Container
> container_e20_1530854311763_0006_01_02 transitioned from
> CONTAINER_CLEANEDUP_AFTER_KILL to DONE
> 2018-07-06 05:22:38,955 INFO container.ContainerImpl
> (ContainerImpl.java:handle(2093)) - Container
> container_e20_1530854311763_0007_01_02 transitioned from NEW to SCHEDULED
> {Code}
> New container requesting for GPU fails to launch
> {code}
> 2018-07-06 05:22:39,048 ERROR nodemanager.LinuxContainerExecutor
> (LinuxContainerExecutor.java:handleLaunchForLaunchType(550)) -
> ResourceHandlerChain.preStart() failed!
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.ResourceHandlerException:
> Failed to find enough GPUs,
> requestor=container_e20_1530854311763_0007_01_02, #RequestedGPUs=2,
> #availableGpus=1
> at
>
[jira] [Commented] (YARN-6966) NodeManager metrics may return wrong negative values when NM restart
[
https://issues.apache.org/jira/browse/YARN-6966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560503#comment-16560503
]
Haibo Chen commented on YARN-6966:
--
The branch-2 patch does not compile for me either
> NodeManager metrics may return wrong negative values when NM restart
>
>
> Key: YARN-6966
> URL: https://issues.apache.org/jira/browse/YARN-6966
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Yang Wang
>Assignee: Szilard Nemeth
>Priority: Major
> Fix For: 3.2.0, 3.1.2
>
> Attachments: YARN-6966-branch-2.001.patch,
> YARN-6966-branch-3.0.0.001.patch, YARN-6966.001.patch, YARN-6966.002.patch,
> YARN-6966.003.patch, YARN-6966.004.patch, YARN-6966.005.patch,
> YARN-6966.005.patch, YARN-6966.006.patch
>
>
> Just as YARN-6212. However, I think it is not a duplicate of YARN-3933.
> The primary cause of negative values is that metrics do not recover properly
> when NM restart.
> AllocatedContainers,ContainersLaunched,AllocatedGB,AvailableGB,AllocatedVCores,AvailableVCores
> in metrics also need to recover when NM restart.
> This should be done in ContainerManagerImpl#recoverContainer.
> The scenario could be reproduction by the following steps:
> # Make sure
> YarnConfiguration.NM_RECOVERY_ENABLED=true,YarnConfiguration.NM_RECOVERY_SUPERVISED=true
> in NM
> # Submit an application and keep running
> # Restart NM
> # Stop the application
> # Now you get the negative values
> {code}
> /jmx?qry=Hadoop:service=NodeManager,name=NodeManagerMetrics
> {code}
> {code}
> {
> name: "Hadoop:service=NodeManager,name=NodeManagerMetrics",
> modelerType: "NodeManagerMetrics",
> tag.Context: "yarn",
> tag.Hostname: "hadoop.com",
> ContainersLaunched: 0,
> ContainersCompleted: 0,
> ContainersFailed: 2,
> ContainersKilled: 0,
> ContainersIniting: 0,
> ContainersRunning: 0,
> AllocatedGB: 0,
> AllocatedContainers: -2,
> AvailableGB: 160,
> AllocatedVCores: -11,
> AvailableVCores: 3611,
> ContainerLaunchDurationNumOps: 2,
> ContainerLaunchDurationAvgTime: 6,
> BadLocalDirs: 0,
> BadLogDirs: 0,
> GoodLocalDirsDiskUtilizationPerc: 2,
> GoodLogDirsDiskUtilizationPerc: 2
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6966) NodeManager metrics may return wrong negative values when NM restart
[
https://issues.apache.org/jira/browse/YARN-6966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560486#comment-16560486
]
Haibo Chen commented on YARN-6966:
--
[~snemeth] We should not be doing patch for branch-3.0.0, as it is already
released. Instead, we should create a patch for branch-3.0 which is the next
3.0.x maintenance release.
> NodeManager metrics may return wrong negative values when NM restart
>
>
> Key: YARN-6966
> URL: https://issues.apache.org/jira/browse/YARN-6966
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Yang Wang
>Assignee: Szilard Nemeth
>Priority: Major
> Fix For: 3.2.0, 3.1.2
>
> Attachments: YARN-6966-branch-2.001.patch,
> YARN-6966-branch-3.0.0.001.patch, YARN-6966.001.patch, YARN-6966.002.patch,
> YARN-6966.003.patch, YARN-6966.004.patch, YARN-6966.005.patch,
> YARN-6966.005.patch, YARN-6966.006.patch
>
>
> Just as YARN-6212. However, I think it is not a duplicate of YARN-3933.
> The primary cause of negative values is that metrics do not recover properly
> when NM restart.
> AllocatedContainers,ContainersLaunched,AllocatedGB,AvailableGB,AllocatedVCores,AvailableVCores
> in metrics also need to recover when NM restart.
> This should be done in ContainerManagerImpl#recoverContainer.
> The scenario could be reproduction by the following steps:
> # Make sure
> YarnConfiguration.NM_RECOVERY_ENABLED=true,YarnConfiguration.NM_RECOVERY_SUPERVISED=true
> in NM
> # Submit an application and keep running
> # Restart NM
> # Stop the application
> # Now you get the negative values
> {code}
> /jmx?qry=Hadoop:service=NodeManager,name=NodeManagerMetrics
> {code}
> {code}
> {
> name: "Hadoop:service=NodeManager,name=NodeManagerMetrics",
> modelerType: "NodeManagerMetrics",
> tag.Context: "yarn",
> tag.Hostname: "hadoop.com",
> ContainersLaunched: 0,
> ContainersCompleted: 0,
> ContainersFailed: 2,
> ContainersKilled: 0,
> ContainersIniting: 0,
> ContainersRunning: 0,
> AllocatedGB: 0,
> AllocatedContainers: -2,
> AvailableGB: 160,
> AllocatedVCores: -11,
> AvailableVCores: 3611,
> ContainerLaunchDurationNumOps: 2,
> ContainerLaunchDurationAvgTime: 6,
> BadLocalDirs: 0,
> BadLogDirs: 0,
> GoodLocalDirsDiskUtilizationPerc: 2,
> GoodLogDirsDiskUtilizationPerc: 2
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8508) GPU does not get released even though the container is killed
[
https://issues.apache.org/jira/browse/YARN-8508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560479#comment-16560479
]
Eric Yang commented on YARN-8508:
-
+1 looks good to me.
> GPU does not get released even though the container is killed
> --
>
> Key: YARN-8508
> URL: https://issues.apache.org/jira/browse/YARN-8508
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Sumana Sathish
>Assignee: Chandni Singh
>Priority: Major
> Attachments: YARN-8505.001.patch, YARN-8505.002.patch
>
>
> GPU failed to release even though the container using it is being killed
> {Code}
> 2018-07-06 05:22:26,201 INFO container.ContainerImpl
> (ContainerImpl.java:handle(2093)) - Container
> container_e20_1530854311763_0006_01_01 transitioned from RUNNING to
> KILLING
> 2018-07-06 05:22:26,250 INFO container.ContainerImpl
> (ContainerImpl.java:handle(2093)) - Container
> container_e20_1530854311763_0006_01_02 transitioned from RUNNING to
> KILLING
> 2018-07-06 05:22:26,251 INFO application.ApplicationImpl
> (ApplicationImpl.java:handle(632)) - Application
> application_1530854311763_0006 transitioned from RUNNING to
> FINISHING_CONTAINERS_WAIT
> 2018-07-06 05:22:26,251 INFO launcher.ContainerLaunch
> (ContainerLaunch.java:cleanupContainer(734)) - Cleaning up container
> container_e20_1530854311763_0006_01_02
> 2018-07-06 05:22:31,358 INFO launcher.ContainerLaunch
> (ContainerLaunch.java:getContainerPid(1102)) - Could not get pid for
> container_e20_1530854311763_0006_01_02. Waited for 5000 ms.
> 2018-07-06 05:22:31,358 WARN launcher.ContainerLaunch
> (ContainerLaunch.java:cleanupContainer(784)) - Container clean up before pid
> file created container_e20_1530854311763_0006_01_02
> 2018-07-06 05:22:31,359 INFO launcher.ContainerLaunch
> (ContainerLaunch.java:reapDockerContainerNoPid(940)) - Unable to obtain pid,
> but docker container request detected. Attempting to reap container
> container_e20_1530854311763_0006_01_02
> 2018-07-06 05:22:31,494 INFO nodemanager.LinuxContainerExecutor
> (LinuxContainerExecutor.java:deleteAsUser(828)) - Deleting absolute path :
> /grid/0/hadoop/yarn/local/usercache/hrt_qa/appcache/application_1530854311763_0006/container_e20_1530854311763_0006_01_02/launch_container.sh
> 2018-07-06 05:22:31,500 INFO nodemanager.LinuxContainerExecutor
> (LinuxContainerExecutor.java:deleteAsUser(828)) - Deleting absolute path :
> /grid/0/hadoop/yarn/local/usercache/hrt_qa/appcache/application_1530854311763_0006/container_e20_1530854311763_0006_01_02/container_tokens
> 2018-07-06 05:22:31,510 INFO container.ContainerImpl
> (ContainerImpl.java:handle(2093)) - Container
> container_e20_1530854311763_0006_01_01 transitioned from KILLING to
> CONTAINER_CLEANEDUP_AFTER_KILL
> 2018-07-06 05:22:31,510 INFO container.ContainerImpl
> (ContainerImpl.java:handle(2093)) - Container
> container_e20_1530854311763_0006_01_02 transitioned from KILLING to
> CONTAINER_CLEANEDUP_AFTER_KILL
> 2018-07-06 05:22:31,512 INFO container.ContainerImpl
> (ContainerImpl.java:handle(2093)) - Container
> container_e20_1530854311763_0006_01_01 transitioned from
> CONTAINER_CLEANEDUP_AFTER_KILL to DONE
> 2018-07-06 05:22:31,513 INFO container.ContainerImpl
> (ContainerImpl.java:handle(2093)) - Container
> container_e20_1530854311763_0006_01_02 transitioned from
> CONTAINER_CLEANEDUP_AFTER_KILL to DONE
> 2018-07-06 05:22:38,955 INFO container.ContainerImpl
> (ContainerImpl.java:handle(2093)) - Container
> container_e20_1530854311763_0007_01_02 transitioned from NEW to SCHEDULED
> {Code}
> New container requesting for GPU fails to launch
> {code}
> 2018-07-06 05:22:39,048 ERROR nodemanager.LinuxContainerExecutor
> (LinuxContainerExecutor.java:handleLaunchForLaunchType(550)) -
> ResourceHandlerChain.preStart() failed!
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.ResourceHandlerException:
> Failed to find enough GPUs,
> requestor=container_e20_1530854311763_0007_01_02, #RequestedGPUs=2,
> #availableGpus=1
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.gpu.GpuResourceAllocator.internalAssignGpus(GpuResourceAllocator.java:225)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.gpu.GpuResourceAllocator.assignGpus(GpuResourceAllocator.java:173)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.gpu.GpuResourceHandlerImpl.preStart(GpuResourceHandlerImpl.java:98)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.ResourceHandlerChain.preStart(ResourceHandlerChain.java:75)
> at
>
[jira] [Updated] (YARN-8508) GPU does not get released even though the container is killed
[
https://issues.apache.org/jira/browse/YARN-8508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated YARN-8508:
Target Version/s: 3.2.0, 3.1.2
> GPU does not get released even though the container is killed
> --
>
> Key: YARN-8508
> URL: https://issues.apache.org/jira/browse/YARN-8508
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Sumana Sathish
>Assignee: Chandni Singh
>Priority: Major
> Attachments: YARN-8505.001.patch, YARN-8505.002.patch
>
>
> GPU failed to release even though the container using it is being killed
> {Code}
> 2018-07-06 05:22:26,201 INFO container.ContainerImpl
> (ContainerImpl.java:handle(2093)) - Container
> container_e20_1530854311763_0006_01_01 transitioned from RUNNING to
> KILLING
> 2018-07-06 05:22:26,250 INFO container.ContainerImpl
> (ContainerImpl.java:handle(2093)) - Container
> container_e20_1530854311763_0006_01_02 transitioned from RUNNING to
> KILLING
> 2018-07-06 05:22:26,251 INFO application.ApplicationImpl
> (ApplicationImpl.java:handle(632)) - Application
> application_1530854311763_0006 transitioned from RUNNING to
> FINISHING_CONTAINERS_WAIT
> 2018-07-06 05:22:26,251 INFO launcher.ContainerLaunch
> (ContainerLaunch.java:cleanupContainer(734)) - Cleaning up container
> container_e20_1530854311763_0006_01_02
> 2018-07-06 05:22:31,358 INFO launcher.ContainerLaunch
> (ContainerLaunch.java:getContainerPid(1102)) - Could not get pid for
> container_e20_1530854311763_0006_01_02. Waited for 5000 ms.
> 2018-07-06 05:22:31,358 WARN launcher.ContainerLaunch
> (ContainerLaunch.java:cleanupContainer(784)) - Container clean up before pid
> file created container_e20_1530854311763_0006_01_02
> 2018-07-06 05:22:31,359 INFO launcher.ContainerLaunch
> (ContainerLaunch.java:reapDockerContainerNoPid(940)) - Unable to obtain pid,
> but docker container request detected. Attempting to reap container
> container_e20_1530854311763_0006_01_02
> 2018-07-06 05:22:31,494 INFO nodemanager.LinuxContainerExecutor
> (LinuxContainerExecutor.java:deleteAsUser(828)) - Deleting absolute path :
> /grid/0/hadoop/yarn/local/usercache/hrt_qa/appcache/application_1530854311763_0006/container_e20_1530854311763_0006_01_02/launch_container.sh
> 2018-07-06 05:22:31,500 INFO nodemanager.LinuxContainerExecutor
> (LinuxContainerExecutor.java:deleteAsUser(828)) - Deleting absolute path :
> /grid/0/hadoop/yarn/local/usercache/hrt_qa/appcache/application_1530854311763_0006/container_e20_1530854311763_0006_01_02/container_tokens
> 2018-07-06 05:22:31,510 INFO container.ContainerImpl
> (ContainerImpl.java:handle(2093)) - Container
> container_e20_1530854311763_0006_01_01 transitioned from KILLING to
> CONTAINER_CLEANEDUP_AFTER_KILL
> 2018-07-06 05:22:31,510 INFO container.ContainerImpl
> (ContainerImpl.java:handle(2093)) - Container
> container_e20_1530854311763_0006_01_02 transitioned from KILLING to
> CONTAINER_CLEANEDUP_AFTER_KILL
> 2018-07-06 05:22:31,512 INFO container.ContainerImpl
> (ContainerImpl.java:handle(2093)) - Container
> container_e20_1530854311763_0006_01_01 transitioned from
> CONTAINER_CLEANEDUP_AFTER_KILL to DONE
> 2018-07-06 05:22:31,513 INFO container.ContainerImpl
> (ContainerImpl.java:handle(2093)) - Container
> container_e20_1530854311763_0006_01_02 transitioned from
> CONTAINER_CLEANEDUP_AFTER_KILL to DONE
> 2018-07-06 05:22:38,955 INFO container.ContainerImpl
> (ContainerImpl.java:handle(2093)) - Container
> container_e20_1530854311763_0007_01_02 transitioned from NEW to SCHEDULED
> {Code}
> New container requesting for GPU fails to launch
> {code}
> 2018-07-06 05:22:39,048 ERROR nodemanager.LinuxContainerExecutor
> (LinuxContainerExecutor.java:handleLaunchForLaunchType(550)) -
> ResourceHandlerChain.preStart() failed!
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.ResourceHandlerException:
> Failed to find enough GPUs,
> requestor=container_e20_1530854311763_0007_01_02, #RequestedGPUs=2,
> #availableGpus=1
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.gpu.GpuResourceAllocator.internalAssignGpus(GpuResourceAllocator.java:225)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.gpu.GpuResourceAllocator.assignGpus(GpuResourceAllocator.java:173)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.gpu.GpuResourceHandlerImpl.preStart(GpuResourceHandlerImpl.java:98)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.ResourceHandlerChain.preStart(ResourceHandlerChain.java:75)
> at
>
[jira] [Commented] (YARN-7948) Enable refreshing maximum allocation for multiple resource types
[ https://issues.apache.org/jira/browse/YARN-7948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560478#comment-16560478 ] Haibo Chen commented on YARN-7948: -- Memory and CPU are always default resources, so I think it's necessary to cover that as well. I have one question for the patch. Why do we need to define CustomResourceTypesConfigurationProvider? I don't see it is called in the code path involved in this patch. Am I missing something? > Enable refreshing maximum allocation for multiple resource types > > > Key: YARN-7948 > URL: https://issues.apache.org/jira/browse/YARN-7948 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 3.0.0 >Reporter: Yufei Gu >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-7948.001.patch > > > YARN-7738 did the same thing for CS. We need a fix for FS. We could fix it by > moving the refresh code from class CS to class AbstractYARNScheduler. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8588) Logging improvements for better debuggability
[ https://issues.apache.org/jira/browse/YARN-8588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated YARN-8588: --- Attachment: (was: YARN-8518.1.patch) > Logging improvements for better debuggability > - > > Key: YARN-8588 > URL: https://issues.apache.org/jira/browse/YARN-8588 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad >Priority: Major > Attachments: YARN-8588.1.patch > > > Capacity allocations decided in GuaranteedCapacityOvertimePolicy are > available via AutoCreatedLeafQueueConfig. However this class lacks a toString > and some other DEBUG level logs are needed for better debuggability -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8588) Logging improvements for better debuggability
[ https://issues.apache.org/jira/browse/YARN-8588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated YARN-8588: --- Attachment: YARN-8588.1.patch > Logging improvements for better debuggability > - > > Key: YARN-8588 > URL: https://issues.apache.org/jira/browse/YARN-8588 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad >Priority: Major > Attachments: YARN-8588.1.patch > > > Capacity allocations decided in GuaranteedCapacityOvertimePolicy are > available via AutoCreatedLeafQueueConfig. However this class lacks a toString > and some other DEBUG level logs are needed for better debuggability -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8588) Logging improvements for better debuggability
[ https://issues.apache.org/jira/browse/YARN-8588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated YARN-8588: --- Attachment: YARN-8518.1.patch > Logging improvements for better debuggability > - > > Key: YARN-8588 > URL: https://issues.apache.org/jira/browse/YARN-8588 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad >Priority: Major > Attachments: YARN-8588.1.patch > > > Capacity allocations decided in GuaranteedCapacityOvertimePolicy are > available via AutoCreatedLeafQueueConfig. However this class lacks a toString > and some other DEBUG level logs are needed for better debuggability -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8571) Validate service principal format prior to launching yarn service
[
https://issues.apache.org/jira/browse/YARN-8571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560472#comment-16560472
]
Eric Yang commented on YARN-8571:
-
[~billie.rinaldi] Thank you for the commit.
> Validate service principal format prior to launching yarn service
> -
>
> Key: YARN-8571
> URL: https://issues.apache.org/jira/browse/YARN-8571
> Project: Hadoop YARN
> Issue Type: Bug
> Components: security, yarn
>Affects Versions: 3.1.0, 3.1.1
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Fix For: 3.2.0, 3.1.2
>
> Attachments: YARN-8571.001.patch, YARN-8571.002.patch
>
>
> Hadoop client and server interaction is designed to validate the service
> principal before RPC request is permitted. In YARN service, the same
> security model is enforced to prevent replay attack. However, end user
> might submit JSON that looks like this to YARN service REST API:
> {code}
> {
> "name": "sleeper-service",
> "version": "1.0.0",
> "components" :
> [
> {
> "name": "sleeper",
> "number_of_containers": 2,
> "launch_command": "sleep 90",
> "resource": {
> "cpus": 1,
> "memory": "256"
> }
> }
> ],
> "kerberos_principal" : {
> "principal_name" : "ambari...@example.com",
> "keytab" : "file:///etc/security/keytabs/smokeuser.headless.keytab"
> }
> }
> {code}
> The kerberos principal is end user kerberos principal instead of service
> principal. This does not work properly because YARN service application
> master requires to run with a service principal to communicate with YARN CLI
> client via Hadoop RPC. Without breaking Hadoop security design in this JIRA,
> it might be in our best interest to validate principal_name during
> submission, and report error message when someone tries to run YARN service
> with user principal.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8407) Container launch exception in AM log should be printed in ERROR level
[
https://issues.apache.org/jira/browse/YARN-8407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560456#comment-16560456
]
genericqa commented on YARN-8407:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 57s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 44s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 11m
37s{color} | {color:green} hadoop-yarn-services-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 68m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | YARN-8407 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12933438/YARN-8407.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 138eab0e8125 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 2cccf40 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/21413/testReport/ |
| Max. process+thread count | 741 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/21413/console |
| Powered by |
[jira] [Commented] (YARN-8596) Allow SQLFederationStateStore to submit the same app in the same subcluster
[ https://issues.apache.org/jira/browse/YARN-8596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560440#comment-16560440 ] Hudson commented on YARN-8596: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14658 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14658/]) YARN-8596. Allow SQLFederationStateStore to submit the same app in the (inigoiri: rev 79091cf76f6e966f64ac1d65e43e95782695e678) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/impl/SQLFederationStateStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/FederationStateStoreBaseTest.java > Allow SQLFederationStateStore to submit the same app in the same subcluster > --- > > Key: YARN-8596 > URL: https://issues.apache.org/jira/browse/YARN-8596 > Project: Hadoop YARN > Issue Type: Bug > Components: federation >Reporter: Íñigo Goiri >Assignee: Giovanni Matteo Fumarola >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-8596.v1.patch, YARN-8596.v2.patch > > > Wrong behavior in SQLFederationStateStore when we want to submit the same > application in the same subcluster -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8522) Application fails with InvalidResourceRequestException
[
https://issues.apache.org/jira/browse/YARN-8522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560421#comment-16560421
]
Gour Saha commented on YARN-8522:
-
[~Zian Chen] 002 patch looks ok. I don't have a good setup to test this. Were
you able to reproduce this issue in a cluster without your patch and then test
that your patch fixes it? Do you think we can write a test for it?
> Application fails with InvalidResourceRequestException
> --
>
> Key: YARN-8522
> URL: https://issues.apache.org/jira/browse/YARN-8522
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Yesha Vora
>Assignee: Zian Chen
>Priority: Major
> Attachments: YARN-8522.001.patch, YARN-8522.002.patch
>
>
> Launch multiple streaming app simultaneously. Here, sometimes one of the
> application fails with below stack trace.
> {code}
> 18/07/02 07:14:32 INFO retry.RetryInvocationHandler:
> java.net.ConnectException: Call From xx.xx.xx.xx/xx.xx.xx.xx to
> xx.xx.xx.xx:8032 failed on connection exception: java.net.ConnectException:
> Connection refused; For more details see:
> http://wiki.apache.org/hadoop/ConnectionRefused, while invoking
> ApplicationClientProtocolPBClientImpl.submitApplication over null. Retrying
> after sleeping for 3ms.
> 18/07/02 07:14:32 WARN client.RequestHedgingRMFailoverProxyProvider:
> Invocation returned exception:
> org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
> resource request, only one resource request with * is allowed
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:502)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:389)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:320)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:645)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:277)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:563)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678)
> on [rm2], so propagating back to caller.
> 18/07/02 07:14:32 INFO mapreduce.JobSubmitter: Cleaning up the staging area
> /user/hrt_qa/.staging/job_1530515284077_0007
> 18/07/02 07:14:32 ERROR streaming.StreamJob: Error Launching job :
> org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
> resource request, only one resource request with * is allowed
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:502)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:389)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:320)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:645)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:277)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:563)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678)
> Streaming
[jira] [Commented] (YARN-7708) [GPG] Load based policy generator
[
https://issues.apache.org/jira/browse/YARN-7708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560420#comment-16560420
]
genericqa commented on YARN-7708:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m
10s{color} | {color:red} Docker failed to build yetus/hadoop:abb62dd. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-7708 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12933411/YARN-7708-YARN-7402.04.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/21415/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> [GPG] Load based policy generator
> -
>
> Key: YARN-7708
> URL: https://issues.apache.org/jira/browse/YARN-7708
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Carlo Curino
>Assignee: Young Chen
>Priority: Major
> Attachments: YARN-7708-YARN-7402.01.cumulative.patch,
> YARN-7708-YARN-7402.01.patch, YARN-7708-YARN-7402.02.cumulative.patch,
> YARN-7708-YARN-7402.02.patch, YARN-7708-YARN-7402.03.cumulative.patch,
> YARN-7708-YARN-7402.03.patch, YARN-7708-YARN-7402.03.patch,
> YARN-7708-YARN-7402.04.cumulative.patch, YARN-7708-YARN-7402.04.patch,
> YARN-7708-YARN-7402.05.cumulative.patch,
> YARN-7708-YARN-7402.06.cumulative.patch,
> YARN-7708-YARN-7402.07.cumulative.patch
>
>
> This policy reads load from the "pendingQueueLength" metrics and provides
> scaling into a set of weights that influence the AMRMProxy and Router
> behaviors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7833) [PERF/TEST] Extend SLS to support simulation of a Federated Environment
[
https://issues.apache.org/jira/browse/YARN-7833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560418#comment-16560418
]
genericqa commented on YARN-7833:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 27m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 31m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 57s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
8s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 28m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 28m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
35s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 74 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 35s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
36s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
3s{color} | {color:red} hadoop-tools/hadoop-sls generated 1 new + 0 unchanged -
0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
54s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
22s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 69m 51s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
38s{color} | {color:green} hadoop-yarn-server-router in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 10m 34s{color}
|
[jira] [Commented] (YARN-8579) New AM attempt could not retrieve previous attempt component data
[
https://issues.apache.org/jira/browse/YARN-8579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560415#comment-16560415
]
Gour Saha commented on YARN-8579:
-
Thanks [~csingh] for the review. I uploaded 003 with your suggestion.
I do have one fundamental question though. I don't understand why for FAIR
scheduler the below assert fails (which means no NMTokens are sent over even
with this patch). The method where I made the code change is a common method
which is called by both Fair and Capacity Schedulers. Any idea? That's why I
had to enable this assert for CAPACITY scheduler only. I don't have a cluster
setup where I can test FairScheduler.
{code}
if (getSchedulerType().equals(SchedulerType.CAPACITY)) {
Assert.assertEquals(1, nmTokens.size());
// container 3 is running on node 2
Assert.assertEquals(nm2Address,
nmTokens.get(0).getNodeId().toString());
}
{code}
> New AM attempt could not retrieve previous attempt component data
> -
>
> Key: YARN-8579
> URL: https://issues.apache.org/jira/browse/YARN-8579
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.1.1
>Reporter: Yesha Vora
>Assignee: Gour Saha
>Priority: Critical
> Fix For: 3.2.0, 3.1.2
>
> Attachments: YARN-8579.001.patch, YARN-8579.002.patch,
> YARN-8579.003.patch
>
>
> Steps:
> 1) Launch httpd-docker
> 2) Wait for app to be in STABLE state
> 3) Run validation for app (It takes around 3 mins)
> 4) Stop all Zks
> 5) Wait 60 sec
> 6) Kill AM
> 7) wait for 30 sec
> 8) Start all ZKs
> 9) Wait for application to finish
> 10) Validate expected containers of the app
> Expected behavior:
> New attempt of AM should start and docker containers launched by 1st attempt
> should be recovered by new attempt.
> Actual behavior:
> New AM attempt starts. It can not recover 1st attempt docker containers. It
> can not read component details from ZK.
> Thus, it starts new attempt for all containers.
> {code}
> 2018-07-19 22:42:47,595 [main] INFO service.ServiceScheduler - Registering
> appattempt_1531977563978_0015_02, fault-test-zkrm-httpd-docker into
> registry
> 2018-07-19 22:42:47,611 [main] INFO service.ServiceScheduler - Received 1
> containers from previous attempt.
> 2018-07-19 22:42:47,642 [main] INFO service.ServiceScheduler - Could not
> read component paths:
> `/users/hrt-qa/services/yarn-service/fault-test-zkrm-httpd-docker/components':
> No such file or directory: KeeperErrorCode = NoNode for
> /registry/users/hrt-qa/services/yarn-service/fault-test-zkrm-httpd-docker/components
> 2018-07-19 22:42:47,643 [main] INFO service.ServiceScheduler - Handling
> container_e08_1531977563978_0015_01_03 from previous attempt
> 2018-07-19 22:42:47,643 [main] INFO service.ServiceScheduler - Record not
> found in registry for container container_e08_1531977563978_0015_01_03
> from previous attempt, releasing
> 2018-07-19 22:42:47,649 [AMRM Callback Handler Thread] INFO
> impl.TimelineV2ClientImpl - Updated timeline service address to xxx:33019
> 2018-07-19 22:42:47,651 [main] INFO service.ServiceScheduler - Triggering
> initial evaluation of component httpd
> 2018-07-19 22:42:47,652 [main] INFO component.Component - [INIT COMPONENT
> httpd]: 2 instances.
> 2018-07-19 22:42:47,652 [main] INFO component.Component - [COMPONENT httpd]
> Requesting for 2 container(s){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8579) New AM attempt could not retrieve previous attempt component data
[
https://issues.apache.org/jira/browse/YARN-8579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560414#comment-16560414
]
genericqa commented on YARN-8579:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 11s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 23s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 69m 17s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 12m 1s{color}
| {color:red} hadoop-yarn-services-core in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}153m 59s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.fair.policies.TestDominantResourceFairnessPolicy
|
| | hadoop.yarn.service.TestYarnNativeServices |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | YARN-8579 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12933419/YARN-8579.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 349032c56f6c 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14
08:53:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / b429f19 |
| maven | version: Apache Maven 3.3.9 |
| Default Java |
[jira] [Updated] (YARN-8596) Allow SQLFederationStateStore to submit the same app in the same subcluster
[ https://issues.apache.org/jira/browse/YARN-8596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated YARN-8596: -- Summary: Allow SQLFederationStateStore to submit the same app in the same subcluster (was: Wrong behavior in SQLFederationStateStore when submitting the same app in the same subcluster) > Allow SQLFederationStateStore to submit the same app in the same subcluster > --- > > Key: YARN-8596 > URL: https://issues.apache.org/jira/browse/YARN-8596 > Project: Hadoop YARN > Issue Type: Bug > Components: federation >Reporter: Íñigo Goiri >Assignee: Giovanni Matteo Fumarola >Priority: Major > Attachments: YARN-8596.v1.patch, YARN-8596.v2.patch > > > Wrong behavior in SQLFederationStateStore when we want to submit the same > application in the same subcluster -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8579) New AM attempt could not retrieve previous attempt component data
[
https://issues.apache.org/jira/browse/YARN-8579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gour Saha updated YARN-8579:
Attachment: YARN-8579.003.patch
> New AM attempt could not retrieve previous attempt component data
> -
>
> Key: YARN-8579
> URL: https://issues.apache.org/jira/browse/YARN-8579
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.1.1
>Reporter: Yesha Vora
>Assignee: Gour Saha
>Priority: Critical
> Fix For: 3.2.0, 3.1.2
>
> Attachments: YARN-8579.001.patch, YARN-8579.002.patch,
> YARN-8579.003.patch
>
>
> Steps:
> 1) Launch httpd-docker
> 2) Wait for app to be in STABLE state
> 3) Run validation for app (It takes around 3 mins)
> 4) Stop all Zks
> 5) Wait 60 sec
> 6) Kill AM
> 7) wait for 30 sec
> 8) Start all ZKs
> 9) Wait for application to finish
> 10) Validate expected containers of the app
> Expected behavior:
> New attempt of AM should start and docker containers launched by 1st attempt
> should be recovered by new attempt.
> Actual behavior:
> New AM attempt starts. It can not recover 1st attempt docker containers. It
> can not read component details from ZK.
> Thus, it starts new attempt for all containers.
> {code}
> 2018-07-19 22:42:47,595 [main] INFO service.ServiceScheduler - Registering
> appattempt_1531977563978_0015_02, fault-test-zkrm-httpd-docker into
> registry
> 2018-07-19 22:42:47,611 [main] INFO service.ServiceScheduler - Received 1
> containers from previous attempt.
> 2018-07-19 22:42:47,642 [main] INFO service.ServiceScheduler - Could not
> read component paths:
> `/users/hrt-qa/services/yarn-service/fault-test-zkrm-httpd-docker/components':
> No such file or directory: KeeperErrorCode = NoNode for
> /registry/users/hrt-qa/services/yarn-service/fault-test-zkrm-httpd-docker/components
> 2018-07-19 22:42:47,643 [main] INFO service.ServiceScheduler - Handling
> container_e08_1531977563978_0015_01_03 from previous attempt
> 2018-07-19 22:42:47,643 [main] INFO service.ServiceScheduler - Record not
> found in registry for container container_e08_1531977563978_0015_01_03
> from previous attempt, releasing
> 2018-07-19 22:42:47,649 [AMRM Callback Handler Thread] INFO
> impl.TimelineV2ClientImpl - Updated timeline service address to xxx:33019
> 2018-07-19 22:42:47,651 [main] INFO service.ServiceScheduler - Triggering
> initial evaluation of component httpd
> 2018-07-19 22:42:47,652 [main] INFO component.Component - [INIT COMPONENT
> httpd]: 2 instances.
> 2018-07-19 22:42:47,652 [main] INFO component.Component - [COMPONENT httpd]
> Requesting for 2 container(s){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7833) [PERF/TEST] Extend SLS to support simulation of a Federated Environment
[
https://issues.apache.org/jira/browse/YARN-7833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560387#comment-16560387
]
genericqa commented on YARN-7833:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 32m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 1s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
12s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 28m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 28m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
32s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 137 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 47s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
50s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
9s{color} | {color:red} hadoop-tools/hadoop-sls generated 1 new + 0 unchanged -
0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
55s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
24s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 85m 7s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
|
[jira] [Updated] (YARN-8407) Container launch exception in AM log should be printed in ERROR level
[
https://issues.apache.org/jira/browse/YARN-8407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yesha Vora updated YARN-8407:
-
Attachment: YARN-8407.003.patch
> Container launch exception in AM log should be printed in ERROR level
> -
>
> Key: YARN-8407
> URL: https://issues.apache.org/jira/browse/YARN-8407
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Yesha Vora
>Assignee: Yesha Vora
>Priority: Major
> Attachments: YARN-8407.001.patch, YARN-8407.002.patch,
> YARN-8407.003.patch
>
>
> when a container launch is failing due to docker image not available is
> logged as INFO level in AM log.
> Container launch failure should be logged as ERROR.
> Steps:
> launch httpd yarn-service application with invalid docker image
>
> {code:java}
> 2018-06-07 01:51:32,966 [Component dispatcher] INFO
> instance.ComponentInstance - [COMPINSTANCE httpd-0 :
> container_e05_1528335963594_0001_01_02]:
> container_e05_1528335963594_0001_01_02 completed. Reinsert back to
> pending list and requested a new container.
> exitStatus=-1, diagnostics=[2018-06-07 01:51:02.363]Exception from
> container-launch.
> Container id: container_e05_1528335963594_0001_01_02
> Exit code: 7
> Exception message: Launch container failed
> Shell error output: Unable to find image 'xxx/httpd:0.1' locally
> Trying to pull repository xxx/httpd ...
> /usr/bin/docker-current: Get https://xxx/v1/_ping: dial tcp: lookup xxx on
> yyy: no such host.
> See '/usr/bin/docker-current run --help'.
> Shell output: main : command provided 4
> main : run as user is hbase
> main : requested yarn user is hbase
> Creating script paths...
> Creating local dirs...
> Getting exit code file...
> Changing effective user to root...
> Wrote the exit code 7 to
> /grid/0/hadoop/yarn/local/nmPrivate/application_1528335963594_0001/container_e05_1528335963594_0001_01_02/container_e05_1528335963594_0001_01_02.pid.exitcode
> [2018-06-07 01:51:02.393]Diagnostic message from attempt :
> [2018-06-07 01:51:02.394]Container exited with a non-zero exit code 7. Last
> 4096 bytes of stderr.txt :
> [2018-06-07 01:51:32.428]Could not find
> nmPrivate/application_1528335963594_0001/container_e05_1528335963594_0001_01_02//container_e05_1528335963594_0001_01_02.pid
> in any of the directories
> 2018-06-07 01:51:32,966 [Component dispatcher] INFO
> instance.ComponentInstance - [COMPINSTANCE httpd-0 :
> container_e05_1528335963594_0001_01_02] Transitioned from STARTED to INIT
> on STOP event{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8596) Wrong behavior in SQLFederationStateStore when submitting the same app in the same subcluster
[
https://issues.apache.org/jira/browse/YARN-8596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560380#comment-16560380
]
genericqa commented on YARN-8596:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 49s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 3s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
12s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 57s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | YARN-8596 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12933427/YARN-8596.v2.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux ce95fad80619 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31
19:38:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / b429f19 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/21412/testReport/ |
| Max. process+thread count | 407 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/21412/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Wrong behavior in SQLFederationStateStore when submitting the same app in the
>
[jira] [Commented] (YARN-8517) getContainer and getContainers ResourceManager REST API methods are not documented
[
https://issues.apache.org/jira/browse/YARN-8517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560361#comment-16560361
]
Hudson commented on YARN-8517:
--
FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #14657 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/14657/])
YARN-8517. getContainer and getContainers ResourceManager REST API (rkanter:
rev 2cccf4061cc4021c48e29879700dbc94f832b7d1)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/DefaultAMSProcessor.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/exceptions/InvalidResourceRequestException.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java
> getContainer and getContainers ResourceManager REST API methods are not
> documented
> --
>
> Key: YARN-8517
> URL: https://issues.apache.org/jira/browse/YARN-8517
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Reporter: Szilard Nemeth
>Assignee: Antal Bálint Steinbach
>Priority: Major
> Labels: newbie, newbie++
> Fix For: 3.2.0
>
> Attachments: YARN-8517.001.patch, YARN-8517.002.patch,
> YARN-8517.003.patch, YARN-8517.004.patch, YARN-8517.005.patch
>
>
> Looking at the documentation here:
> https://hadoop.apache.org/docs/r3.1.0/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html
> I cannot find documentation for 2 RM REST endpoints:
> - /apps/\{appid\}/appattempts/\{appattemptid\}/containers
> - /apps/\{appid\}/appattempts/\{appattemptid\}/containers/\{containerid\}
> I suppose they are not intentionally undocumented.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8566) Add diagnostic message for unschedulable containers
[
https://issues.apache.org/jira/browse/YARN-8566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560359#comment-16560359
]
Hudson commented on YARN-8566:
--
FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #14656 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/14656/])
YARN-8566. Add diagnostic message for unschedulable containers (snemeth
(rkanter: rev fecbac499e2ae6b3334773a997d454a518f43e01)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerRest.md
> Add diagnostic message for unschedulable containers
> ---
>
> Key: YARN-8566
> URL: https://issues.apache.org/jira/browse/YARN-8566
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: YARN-8566.001.patch, YARN-8566.002.patch,
> YARN-8566.003.patch, YARN-8566.004.patch, YARN-8566.005.patch,
> YARN-8566.006.patch, YARN-8566.007.patch
>
>
> If a queue is configured with maxResources set to 0 for a resource, and an
> application is submitted to that queue that requests that resource, that
> application will remain pending until it is removed or moved to a different
> queue. This behavior can be realized without extended resources, but it’s
> unlikely a user will create a queue that allows 0 memory or CPU. As the
> number of resources in the system increases, this scenario will become more
> common, and it will become harder to recognize these cases. Therefore, the
> scheduler should indicate in the diagnostic string for an application if it
> was not scheduled because of a 0 maxResources setting.
> Example configuration (fair-scheduler.xml) :
> {code:java}
>
> 10
>
> 1 mb,2vcores
> 9 mb,4vcores, 0gpu
> 50
> -1.0f
> 2.0
> fair
>
>
> {code}
> Command:
> {code:java}
> yarn jar
> "./share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.0-SNAPSHOT.jar" pi
> -Dmapreduce.job.queuename=sample_queue -Dmapreduce.map.resource.gpu=1 1 1000;
> {code}
> The job hangs and the application diagnostic info is empty.
> Given that an exception is thrown before any mapper/reducer container is
> created, the diagnostic message of the AM should be updated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8566) Add diagnostic message for unschedulable containers
[
https://issues.apache.org/jira/browse/YARN-8566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560354#comment-16560354
]
Robert Kanter commented on YARN-8566:
-
+1 LGTM
> Add diagnostic message for unschedulable containers
> ---
>
> Key: YARN-8566
> URL: https://issues.apache.org/jira/browse/YARN-8566
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: YARN-8566.001.patch, YARN-8566.002.patch,
> YARN-8566.003.patch, YARN-8566.004.patch, YARN-8566.005.patch,
> YARN-8566.006.patch, YARN-8566.007.patch
>
>
> If a queue is configured with maxResources set to 0 for a resource, and an
> application is submitted to that queue that requests that resource, that
> application will remain pending until it is removed or moved to a different
> queue. This behavior can be realized without extended resources, but it’s
> unlikely a user will create a queue that allows 0 memory or CPU. As the
> number of resources in the system increases, this scenario will become more
> common, and it will become harder to recognize these cases. Therefore, the
> scheduler should indicate in the diagnostic string for an application if it
> was not scheduled because of a 0 maxResources setting.
> Example configuration (fair-scheduler.xml) :
> {code:java}
>
> 10
>
> 1 mb,2vcores
> 9 mb,4vcores, 0gpu
> 50
> -1.0f
> 2.0
> fair
>
>
> {code}
> Command:
> {code:java}
> yarn jar
> "./share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.0-SNAPSHOT.jar" pi
> -Dmapreduce.job.queuename=sample_queue -Dmapreduce.map.resource.gpu=1 1 1000;
> {code}
> The job hangs and the application diagnostic info is empty.
> Given that an exception is thrown before any mapper/reducer container is
> created, the diagnostic message of the AM should be updated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8517) getContainer and getContainers ResourceManager REST API methods are not documented
[
https://issues.apache.org/jira/browse/YARN-8517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560339#comment-16560339
]
Robert Kanter commented on YARN-8517:
-
+1 LGTM
> getContainer and getContainers ResourceManager REST API methods are not
> documented
> --
>
> Key: YARN-8517
> URL: https://issues.apache.org/jira/browse/YARN-8517
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Reporter: Szilard Nemeth
>Assignee: Antal Bálint Steinbach
>Priority: Major
> Labels: newbie, newbie++
> Attachments: YARN-8517.001.patch, YARN-8517.002.patch,
> YARN-8517.003.patch, YARN-8517.004.patch, YARN-8517.005.patch
>
>
> Looking at the documentation here:
> https://hadoop.apache.org/docs/r3.1.0/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html
> I cannot find documentation for 2 RM REST endpoints:
> - /apps/\{appid\}/appattempts/\{appattemptid\}/containers
> - /apps/\{appid\}/appattempts/\{appattemptid\}/containers/\{containerid\}
> I suppose they are not intentionally undocumented.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8596) Wrong behavior in SQLFederationStateStore when submitting the same app in the same subcluster
[ https://issues.apache.org/jira/browse/YARN-8596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560320#comment-16560320 ] Íñigo Goiri commented on YARN-8596: --- [^YARN-8596.v2.patch] LGTM. +1 Let's see what Yetus say. > Wrong behavior in SQLFederationStateStore when submitting the same app in the > same subcluster > - > > Key: YARN-8596 > URL: https://issues.apache.org/jira/browse/YARN-8596 > Project: Hadoop YARN > Issue Type: Bug > Components: federation >Reporter: Íñigo Goiri >Assignee: Giovanni Matteo Fumarola >Priority: Major > Attachments: YARN-8596.v1.patch, YARN-8596.v2.patch > > > Wrong behavior in SQLFederationStateStore when we want to submit the same > application in the same subcluster -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8596) Wrong behavior in SQLFederationStateStore when submitting the same app in the same subcluster
[ https://issues.apache.org/jira/browse/YARN-8596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560164#comment-16560164 ] Giovanni Matteo Fumarola edited comment on YARN-8596 at 7/27/18 8:53 PM: - Thanks [~elgoiri] for the feedback. Attached v2. was (Author: giovanni.fumarola): Thanks [~elgoiri] for the feedback. 1) The application was not inserted into the statestore because was already present. The message should be fine like that. 2) I can move the else if next to the } but I will have to move the 2 comments within the if block. That will reduce the readability of the code. > Wrong behavior in SQLFederationStateStore when submitting the same app in the > same subcluster > - > > Key: YARN-8596 > URL: https://issues.apache.org/jira/browse/YARN-8596 > Project: Hadoop YARN > Issue Type: Bug > Components: federation >Reporter: Íñigo Goiri >Assignee: Giovanni Matteo Fumarola >Priority: Major > Attachments: YARN-8596.v1.patch, YARN-8596.v2.patch > > > Wrong behavior in SQLFederationStateStore when we want to submit the same > application in the same subcluster -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8596) Wrong behavior in SQLFederationStateStore when submitting the same app in the same subcluster
[ https://issues.apache.org/jira/browse/YARN-8596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated YARN-8596: --- Attachment: YARN-8596.v2.patch > Wrong behavior in SQLFederationStateStore when submitting the same app in the > same subcluster > - > > Key: YARN-8596 > URL: https://issues.apache.org/jira/browse/YARN-8596 > Project: Hadoop YARN > Issue Type: Bug > Components: federation >Reporter: Íñigo Goiri >Assignee: Giovanni Matteo Fumarola >Priority: Major > Attachments: YARN-8596.v1.patch, YARN-8596.v2.patch > > > Wrong behavior in SQLFederationStateStore when we want to submit the same > application in the same subcluster -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8448) AM HTTPS Support
[
https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560254#comment-16560254
]
Robert Kanter commented on YARN-8448:
-
The 002 patch is mostly just some refactoring to split {{ProxyCAManager}} into
{{ProxyCA}} and {{ProxyCAManager}}. This was necessary as I started looking at
YARN-8449 (RM HA support) because {{ProxyCA}} needs to be in the
{{hadoop-yarn-server-web-proxy}} module (to work with the proxy code) while
{{ProxyCAManager}} needs to be in the {{hadoop-yarn-server-resourcemanager}}
module (to work with the {{RMStateStore}}. Most of the code is the same, but
some things got moved around and reorganized. I also beefed up some tests in
{{TestProxyCA}} (which used to be named {{TestProxyCAManager}}.
> AM HTTPS Support
>
>
> Key: YARN-8448
> URL: https://issues.apache.org/jira/browse/YARN-8448
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Major
> Attachments: YARN-8448.001.patch, YARN-8448.002.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8448) AM HTTPS Support
[ https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated YARN-8448: Attachment: YARN-8448.002.patch > AM HTTPS Support > > > Key: YARN-8448 > URL: https://issues.apache.org/jira/browse/YARN-8448 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Major > Attachments: YARN-8448.001.patch, YARN-8448.002.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8579) New AM attempt could not retrieve previous attempt component data
[
https://issues.apache.org/jira/browse/YARN-8579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560218#comment-16560218
]
Chandni Singh edited comment on YARN-8579 at 7/27/18 8:01 PM:
--
[~gsaha] Thanks for debugging the issue. patch 2 looks good to me.
Just a nitpick. Since we use slf4j, we can use it instead of string
concatenation in the log stmt
{code:java}
LOG.info("Containers recovered after AM registered: " + containers);
{code}
to
{code:java}
LOG.info("Containers recovered after AM registered: {} ", containers);
{code}
was (Author: csingh):
[~gsaha] Thanks for debugging the issue. patch 2 looks good to me.
Just a nitpick. Since we use slf4j, we can use it instead of string
concatenation in the log stmt
{code:java}
LOG.info("Containers recovered after AM registered: ", containers);
{code}
to
{code:java}
LOG.info("Containers recovered after AM registered: {} ", containers);
{code}
> New AM attempt could not retrieve previous attempt component data
> -
>
> Key: YARN-8579
> URL: https://issues.apache.org/jira/browse/YARN-8579
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.1.1
>Reporter: Yesha Vora
>Assignee: Gour Saha
>Priority: Critical
> Fix For: 3.2.0, 3.1.2
>
> Attachments: YARN-8579.001.patch, YARN-8579.002.patch
>
>
> Steps:
> 1) Launch httpd-docker
> 2) Wait for app to be in STABLE state
> 3) Run validation for app (It takes around 3 mins)
> 4) Stop all Zks
> 5) Wait 60 sec
> 6) Kill AM
> 7) wait for 30 sec
> 8) Start all ZKs
> 9) Wait for application to finish
> 10) Validate expected containers of the app
> Expected behavior:
> New attempt of AM should start and docker containers launched by 1st attempt
> should be recovered by new attempt.
> Actual behavior:
> New AM attempt starts. It can not recover 1st attempt docker containers. It
> can not read component details from ZK.
> Thus, it starts new attempt for all containers.
> {code}
> 2018-07-19 22:42:47,595 [main] INFO service.ServiceScheduler - Registering
> appattempt_1531977563978_0015_02, fault-test-zkrm-httpd-docker into
> registry
> 2018-07-19 22:42:47,611 [main] INFO service.ServiceScheduler - Received 1
> containers from previous attempt.
> 2018-07-19 22:42:47,642 [main] INFO service.ServiceScheduler - Could not
> read component paths:
> `/users/hrt-qa/services/yarn-service/fault-test-zkrm-httpd-docker/components':
> No such file or directory: KeeperErrorCode = NoNode for
> /registry/users/hrt-qa/services/yarn-service/fault-test-zkrm-httpd-docker/components
> 2018-07-19 22:42:47,643 [main] INFO service.ServiceScheduler - Handling
> container_e08_1531977563978_0015_01_03 from previous attempt
> 2018-07-19 22:42:47,643 [main] INFO service.ServiceScheduler - Record not
> found in registry for container container_e08_1531977563978_0015_01_03
> from previous attempt, releasing
> 2018-07-19 22:42:47,649 [AMRM Callback Handler Thread] INFO
> impl.TimelineV2ClientImpl - Updated timeline service address to xxx:33019
> 2018-07-19 22:42:47,651 [main] INFO service.ServiceScheduler - Triggering
> initial evaluation of component httpd
> 2018-07-19 22:42:47,652 [main] INFO component.Component - [INIT COMPONENT
> httpd]: 2 instances.
> 2018-07-19 22:42:47,652 [main] INFO component.Component - [COMPONENT httpd]
> Requesting for 2 container(s){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8579) New AM attempt could not retrieve previous attempt component data
[
https://issues.apache.org/jira/browse/YARN-8579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560218#comment-16560218
]
Chandni Singh commented on YARN-8579:
-
[~gsaha] Thanks for debugging the issue. patch 2 looks good to me.
Just a nitpick. Since we use slf4j, we can use it instead of string
concatenation in the log stmt
{code:java}
LOG.info("Containers recovered after AM registered: ", containers);
{code}
to
{code:java}
LOG.info("Containers recovered after AM registered: {} ", containers);
{code}
> New AM attempt could not retrieve previous attempt component data
> -
>
> Key: YARN-8579
> URL: https://issues.apache.org/jira/browse/YARN-8579
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.1.1
>Reporter: Yesha Vora
>Assignee: Gour Saha
>Priority: Critical
> Fix For: 3.2.0, 3.1.2
>
> Attachments: YARN-8579.001.patch, YARN-8579.002.patch
>
>
> Steps:
> 1) Launch httpd-docker
> 2) Wait for app to be in STABLE state
> 3) Run validation for app (It takes around 3 mins)
> 4) Stop all Zks
> 5) Wait 60 sec
> 6) Kill AM
> 7) wait for 30 sec
> 8) Start all ZKs
> 9) Wait for application to finish
> 10) Validate expected containers of the app
> Expected behavior:
> New attempt of AM should start and docker containers launched by 1st attempt
> should be recovered by new attempt.
> Actual behavior:
> New AM attempt starts. It can not recover 1st attempt docker containers. It
> can not read component details from ZK.
> Thus, it starts new attempt for all containers.
> {code}
> 2018-07-19 22:42:47,595 [main] INFO service.ServiceScheduler - Registering
> appattempt_1531977563978_0015_02, fault-test-zkrm-httpd-docker into
> registry
> 2018-07-19 22:42:47,611 [main] INFO service.ServiceScheduler - Received 1
> containers from previous attempt.
> 2018-07-19 22:42:47,642 [main] INFO service.ServiceScheduler - Could not
> read component paths:
> `/users/hrt-qa/services/yarn-service/fault-test-zkrm-httpd-docker/components':
> No such file or directory: KeeperErrorCode = NoNode for
> /registry/users/hrt-qa/services/yarn-service/fault-test-zkrm-httpd-docker/components
> 2018-07-19 22:42:47,643 [main] INFO service.ServiceScheduler - Handling
> container_e08_1531977563978_0015_01_03 from previous attempt
> 2018-07-19 22:42:47,643 [main] INFO service.ServiceScheduler - Record not
> found in registry for container container_e08_1531977563978_0015_01_03
> from previous attempt, releasing
> 2018-07-19 22:42:47,649 [AMRM Callback Handler Thread] INFO
> impl.TimelineV2ClientImpl - Updated timeline service address to xxx:33019
> 2018-07-19 22:42:47,651 [main] INFO service.ServiceScheduler - Triggering
> initial evaluation of component httpd
> 2018-07-19 22:42:47,652 [main] INFO component.Component - [INIT COMPONENT
> httpd]: 2 instances.
> 2018-07-19 22:42:47,652 [main] INFO component.Component - [COMPONENT httpd]
> Requesting for 2 container(s){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8596) Wrong behavior in SQLFederationStateStore when submitting the same app in the same subcluster
[
https://issues.apache.org/jira/browse/YARN-8596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560210#comment-16560210
]
genericqa commented on YARN-8596:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 29m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 9s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 14s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
18s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 64m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | YARN-8596 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12933415/YARN-8596.v1.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux c2c2b9032060 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / b429f19 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/21407/testReport/ |
| Max. process+thread count | 330 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/21407/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Wrong behavior in SQLFederationStateStore when submitting the same app in the
[jira] [Commented] (YARN-8579) New AM attempt could not retrieve previous attempt component data
[
https://issues.apache.org/jira/browse/YARN-8579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560197#comment-16560197
]
Gour Saha commented on YARN-8579:
-
[~csingh], please review the patch when you get a chance.
> New AM attempt could not retrieve previous attempt component data
> -
>
> Key: YARN-8579
> URL: https://issues.apache.org/jira/browse/YARN-8579
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.1.1
>Reporter: Yesha Vora
>Assignee: Gour Saha
>Priority: Critical
> Fix For: 3.2.0, 3.1.2
>
> Attachments: YARN-8579.001.patch, YARN-8579.002.patch
>
>
> Steps:
> 1) Launch httpd-docker
> 2) Wait for app to be in STABLE state
> 3) Run validation for app (It takes around 3 mins)
> 4) Stop all Zks
> 5) Wait 60 sec
> 6) Kill AM
> 7) wait for 30 sec
> 8) Start all ZKs
> 9) Wait for application to finish
> 10) Validate expected containers of the app
> Expected behavior:
> New attempt of AM should start and docker containers launched by 1st attempt
> should be recovered by new attempt.
> Actual behavior:
> New AM attempt starts. It can not recover 1st attempt docker containers. It
> can not read component details from ZK.
> Thus, it starts new attempt for all containers.
> {code}
> 2018-07-19 22:42:47,595 [main] INFO service.ServiceScheduler - Registering
> appattempt_1531977563978_0015_02, fault-test-zkrm-httpd-docker into
> registry
> 2018-07-19 22:42:47,611 [main] INFO service.ServiceScheduler - Received 1
> containers from previous attempt.
> 2018-07-19 22:42:47,642 [main] INFO service.ServiceScheduler - Could not
> read component paths:
> `/users/hrt-qa/services/yarn-service/fault-test-zkrm-httpd-docker/components':
> No such file or directory: KeeperErrorCode = NoNode for
> /registry/users/hrt-qa/services/yarn-service/fault-test-zkrm-httpd-docker/components
> 2018-07-19 22:42:47,643 [main] INFO service.ServiceScheduler - Handling
> container_e08_1531977563978_0015_01_03 from previous attempt
> 2018-07-19 22:42:47,643 [main] INFO service.ServiceScheduler - Record not
> found in registry for container container_e08_1531977563978_0015_01_03
> from previous attempt, releasing
> 2018-07-19 22:42:47,649 [AMRM Callback Handler Thread] INFO
> impl.TimelineV2ClientImpl - Updated timeline service address to xxx:33019
> 2018-07-19 22:42:47,651 [main] INFO service.ServiceScheduler - Triggering
> initial evaluation of component httpd
> 2018-07-19 22:42:47,652 [main] INFO component.Component - [INIT COMPONENT
> httpd]: 2 instances.
> 2018-07-19 22:42:47,652 [main] INFO component.Component - [COMPONENT httpd]
> Requesting for 2 container(s){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8579) New AM attempt could not retrieve previous attempt component data
[
https://issues.apache.org/jira/browse/YARN-8579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560193#comment-16560193
]
Gour Saha commented on YARN-8579:
-
Uploaded 002 with a few more asserts in the test.
> New AM attempt could not retrieve previous attempt component data
> -
>
> Key: YARN-8579
> URL: https://issues.apache.org/jira/browse/YARN-8579
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.1.1
>Reporter: Yesha Vora
>Assignee: Gour Saha
>Priority: Critical
> Fix For: 3.2.0, 3.1.2
>
> Attachments: YARN-8579.001.patch, YARN-8579.002.patch
>
>
> Steps:
> 1) Launch httpd-docker
> 2) Wait for app to be in STABLE state
> 3) Run validation for app (It takes around 3 mins)
> 4) Stop all Zks
> 5) Wait 60 sec
> 6) Kill AM
> 7) wait for 30 sec
> 8) Start all ZKs
> 9) Wait for application to finish
> 10) Validate expected containers of the app
> Expected behavior:
> New attempt of AM should start and docker containers launched by 1st attempt
> should be recovered by new attempt.
> Actual behavior:
> New AM attempt starts. It can not recover 1st attempt docker containers. It
> can not read component details from ZK.
> Thus, it starts new attempt for all containers.
> {code}
> 2018-07-19 22:42:47,595 [main] INFO service.ServiceScheduler - Registering
> appattempt_1531977563978_0015_02, fault-test-zkrm-httpd-docker into
> registry
> 2018-07-19 22:42:47,611 [main] INFO service.ServiceScheduler - Received 1
> containers from previous attempt.
> 2018-07-19 22:42:47,642 [main] INFO service.ServiceScheduler - Could not
> read component paths:
> `/users/hrt-qa/services/yarn-service/fault-test-zkrm-httpd-docker/components':
> No such file or directory: KeeperErrorCode = NoNode for
> /registry/users/hrt-qa/services/yarn-service/fault-test-zkrm-httpd-docker/components
> 2018-07-19 22:42:47,643 [main] INFO service.ServiceScheduler - Handling
> container_e08_1531977563978_0015_01_03 from previous attempt
> 2018-07-19 22:42:47,643 [main] INFO service.ServiceScheduler - Record not
> found in registry for container container_e08_1531977563978_0015_01_03
> from previous attempt, releasing
> 2018-07-19 22:42:47,649 [AMRM Callback Handler Thread] INFO
> impl.TimelineV2ClientImpl - Updated timeline service address to xxx:33019
> 2018-07-19 22:42:47,651 [main] INFO service.ServiceScheduler - Triggering
> initial evaluation of component httpd
> 2018-07-19 22:42:47,652 [main] INFO component.Component - [INIT COMPONENT
> httpd]: 2 instances.
> 2018-07-19 22:42:47,652 [main] INFO component.Component - [COMPONENT httpd]
> Requesting for 2 container(s){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8579) New AM attempt could not retrieve previous attempt component data
[
https://issues.apache.org/jira/browse/YARN-8579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gour Saha updated YARN-8579:
Attachment: YARN-8579.002.patch
> New AM attempt could not retrieve previous attempt component data
> -
>
> Key: YARN-8579
> URL: https://issues.apache.org/jira/browse/YARN-8579
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.1.1
>Reporter: Yesha Vora
>Assignee: Gour Saha
>Priority: Critical
> Fix For: 3.2.0, 3.1.2
>
> Attachments: YARN-8579.001.patch, YARN-8579.002.patch
>
>
> Steps:
> 1) Launch httpd-docker
> 2) Wait for app to be in STABLE state
> 3) Run validation for app (It takes around 3 mins)
> 4) Stop all Zks
> 5) Wait 60 sec
> 6) Kill AM
> 7) wait for 30 sec
> 8) Start all ZKs
> 9) Wait for application to finish
> 10) Validate expected containers of the app
> Expected behavior:
> New attempt of AM should start and docker containers launched by 1st attempt
> should be recovered by new attempt.
> Actual behavior:
> New AM attempt starts. It can not recover 1st attempt docker containers. It
> can not read component details from ZK.
> Thus, it starts new attempt for all containers.
> {code}
> 2018-07-19 22:42:47,595 [main] INFO service.ServiceScheduler - Registering
> appattempt_1531977563978_0015_02, fault-test-zkrm-httpd-docker into
> registry
> 2018-07-19 22:42:47,611 [main] INFO service.ServiceScheduler - Received 1
> containers from previous attempt.
> 2018-07-19 22:42:47,642 [main] INFO service.ServiceScheduler - Could not
> read component paths:
> `/users/hrt-qa/services/yarn-service/fault-test-zkrm-httpd-docker/components':
> No such file or directory: KeeperErrorCode = NoNode for
> /registry/users/hrt-qa/services/yarn-service/fault-test-zkrm-httpd-docker/components
> 2018-07-19 22:42:47,643 [main] INFO service.ServiceScheduler - Handling
> container_e08_1531977563978_0015_01_03 from previous attempt
> 2018-07-19 22:42:47,643 [main] INFO service.ServiceScheduler - Record not
> found in registry for container container_e08_1531977563978_0015_01_03
> from previous attempt, releasing
> 2018-07-19 22:42:47,649 [AMRM Callback Handler Thread] INFO
> impl.TimelineV2ClientImpl - Updated timeline service address to xxx:33019
> 2018-07-19 22:42:47,651 [main] INFO service.ServiceScheduler - Triggering
> initial evaluation of component httpd
> 2018-07-19 22:42:47,652 [main] INFO component.Component - [INIT COMPONENT
> httpd]: 2 instances.
> 2018-07-19 22:42:47,652 [main] INFO component.Component - [COMPONENT httpd]
> Requesting for 2 container(s){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8596) Wrong behavior in SQLFederationStateStore when submitting the same app in the same subcluster
[ https://issues.apache.org/jira/browse/YARN-8596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560164#comment-16560164 ] Giovanni Matteo Fumarola commented on YARN-8596: Thanks [~elgoiri] for the feedback. 1) The application was not inserted into the statestore because was already present. The message should be fine like that. 2) I can move the else if next to the } but I will have to move the 2 comments within the if block. That will reduce the readability of the code. > Wrong behavior in SQLFederationStateStore when submitting the same app in the > same subcluster > - > > Key: YARN-8596 > URL: https://issues.apache.org/jira/browse/YARN-8596 > Project: Hadoop YARN > Issue Type: Bug > Components: federation >Reporter: Íñigo Goiri >Assignee: Giovanni Matteo Fumarola >Priority: Major > Attachments: YARN-8596.v1.patch > > > Wrong behavior in SQLFederationStateStore when we want to submit the same > application in the same subcluster -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8571) Validate service principal format prior to launching yarn service
[
https://issues.apache.org/jira/browse/YARN-8571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Billie Rinaldi updated YARN-8571:
-
Affects Version/s: 3.1.1
> Validate service principal format prior to launching yarn service
> -
>
> Key: YARN-8571
> URL: https://issues.apache.org/jira/browse/YARN-8571
> Project: Hadoop YARN
> Issue Type: Bug
> Components: security, yarn
>Affects Versions: 3.1.0, 3.1.1
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Fix For: 3.2.0, 3.1.2
>
> Attachments: YARN-8571.001.patch, YARN-8571.002.patch
>
>
> Hadoop client and server interaction is designed to validate the service
> principal before RPC request is permitted. In YARN service, the same
> security model is enforced to prevent replay attack. However, end user
> might submit JSON that looks like this to YARN service REST API:
> {code}
> {
> "name": "sleeper-service",
> "version": "1.0.0",
> "components" :
> [
> {
> "name": "sleeper",
> "number_of_containers": 2,
> "launch_command": "sleep 90",
> "resource": {
> "cpus": 1,
> "memory": "256"
> }
> }
> ],
> "kerberos_principal" : {
> "principal_name" : "ambari...@example.com",
> "keytab" : "file:///etc/security/keytabs/smokeuser.headless.keytab"
> }
> }
> {code}
> The kerberos principal is end user kerberos principal instead of service
> principal. This does not work properly because YARN service application
> master requires to run with a service principal to communicate with YARN CLI
> client via Hadoop RPC. Without breaking Hadoop security design in this JIRA,
> it might be in our best interest to validate principal_name during
> submission, and report error message when someone tries to run YARN service
> with user principal.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8596) Wrong behavior in SQLFederationStateStore when submitting the same app in the same subcluster
[
https://issues.apache.org/jira/browse/YARN-8596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560158#comment-16560158
]
Íñigo Goiri commented on YARN-8596:
---
Thanks [~giovanni.fumarola] for the patch.
A couple comments before Yetus gives a thumbs up:
* The message is not very clear, it should be something like: Application appId
was not present in the State Store. Already present in subcluster X.
* Can we move the } else if { into a single line?
> Wrong behavior in SQLFederationStateStore when submitting the same app in the
> same subcluster
> -
>
> Key: YARN-8596
> URL: https://issues.apache.org/jira/browse/YARN-8596
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Reporter: Íñigo Goiri
>Assignee: Giovanni Matteo Fumarola
>Priority: Major
> Attachments: YARN-8596.v1.patch
>
>
> Wrong behavior in SQLFederationStateStore when we want to submit the same
> application in the same subcluster
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8571) Validate service principal format prior to launching yarn service
[
https://issues.apache.org/jira/browse/YARN-8571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560149#comment-16560149
]
Hudson commented on YARN-8571:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14655 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/14655/])
YARN-8571. Validate service principal format prior to launching yarn (billie:
rev b429f19d32d8f60a3535e047ef10cfb3edeb54c8)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/exceptions/RestApiErrorMessages.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/TestServiceApiUtil.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/utils/ServiceApiUtil.java
> Validate service principal format prior to launching yarn service
> -
>
> Key: YARN-8571
> URL: https://issues.apache.org/jira/browse/YARN-8571
> Project: Hadoop YARN
> Issue Type: Bug
> Components: security, yarn
>Affects Versions: 3.1.0
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN-8571.001.patch, YARN-8571.002.patch
>
>
> Hadoop client and server interaction is designed to validate the service
> principal before RPC request is permitted. In YARN service, the same
> security model is enforced to prevent replay attack. However, end user
> might submit JSON that looks like this to YARN service REST API:
> {code}
> {
> "name": "sleeper-service",
> "version": "1.0.0",
> "components" :
> [
> {
> "name": "sleeper",
> "number_of_containers": 2,
> "launch_command": "sleep 90",
> "resource": {
> "cpus": 1,
> "memory": "256"
> }
> }
> ],
> "kerberos_principal" : {
> "principal_name" : "ambari...@example.com",
> "keytab" : "file:///etc/security/keytabs/smokeuser.headless.keytab"
> }
> }
> {code}
> The kerberos principal is end user kerberos principal instead of service
> principal. This does not work properly because YARN service application
> master requires to run with a service principal to communicate with YARN CLI
> client via Hadoop RPC. Without breaking Hadoop security design in this JIRA,
> it might be in our best interest to validate principal_name during
> submission, and report error message when someone tries to run YARN service
> with user principal.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8407) Container launch exception in AM log should be printed in ERROR level
[
https://issues.apache.org/jira/browse/YARN-8407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560142#comment-16560142
]
genericqa commented on YARN-8407:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} YARN-8407 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-8407 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12933287/YARN-8407.002.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/21408/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Container launch exception in AM log should be printed in ERROR level
> -
>
> Key: YARN-8407
> URL: https://issues.apache.org/jira/browse/YARN-8407
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Yesha Vora
>Assignee: Yesha Vora
>Priority: Major
> Attachments: YARN-8407.001.patch, YARN-8407.002.patch
>
>
> when a container launch is failing due to docker image not available is
> logged as INFO level in AM log.
> Container launch failure should be logged as ERROR.
> Steps:
> launch httpd yarn-service application with invalid docker image
>
> {code:java}
> 2018-06-07 01:51:32,966 [Component dispatcher] INFO
> instance.ComponentInstance - [COMPINSTANCE httpd-0 :
> container_e05_1528335963594_0001_01_02]:
> container_e05_1528335963594_0001_01_02 completed. Reinsert back to
> pending list and requested a new container.
> exitStatus=-1, diagnostics=[2018-06-07 01:51:02.363]Exception from
> container-launch.
> Container id: container_e05_1528335963594_0001_01_02
> Exit code: 7
> Exception message: Launch container failed
> Shell error output: Unable to find image 'xxx/httpd:0.1' locally
> Trying to pull repository xxx/httpd ...
> /usr/bin/docker-current: Get https://xxx/v1/_ping: dial tcp: lookup xxx on
> yyy: no such host.
> See '/usr/bin/docker-current run --help'.
> Shell output: main : command provided 4
> main : run as user is hbase
> main : requested yarn user is hbase
> Creating script paths...
> Creating local dirs...
> Getting exit code file...
> Changing effective user to root...
> Wrote the exit code 7 to
> /grid/0/hadoop/yarn/local/nmPrivate/application_1528335963594_0001/container_e05_1528335963594_0001_01_02/container_e05_1528335963594_0001_01_02.pid.exitcode
> [2018-06-07 01:51:02.393]Diagnostic message from attempt :
> [2018-06-07 01:51:02.394]Container exited with a non-zero exit code 7. Last
> 4096 bytes of stderr.txt :
> [2018-06-07 01:51:32.428]Could not find
> nmPrivate/application_1528335963594_0001/container_e05_1528335963594_0001_01_02//container_e05_1528335963594_0001_01_02.pid
> in any of the directories
> 2018-06-07 01:51:32,966 [Component dispatcher] INFO
> instance.ComponentInstance - [COMPINSTANCE httpd-0 :
> container_e05_1528335963594_0001_01_02] Transitioned from STARTED to INIT
> on STOP event{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8596) Wrong behavior in SQLFederationStateStore when submitting the same app in the same subcluster
[ https://issues.apache.org/jira/browse/YARN-8596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated YARN-8596: --- Component/s: federation > Wrong behavior in SQLFederationStateStore when submitting the same app in the > same subcluster > - > > Key: YARN-8596 > URL: https://issues.apache.org/jira/browse/YARN-8596 > Project: Hadoop YARN > Issue Type: Bug > Components: federation >Reporter: Íñigo Goiri >Assignee: Giovanni Matteo Fumarola >Priority: Major > Attachments: YARN-8596.v1.patch > > > Wrong behavior in SQLFederationStateStore when we want to submit the same > application in the same subcluster -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-8599) Build Master module for MaWo app
[ https://issues.apache.org/jira/browse/YARN-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yesha Vora reassigned YARN-8599: Assignee: Yesha Vora > Build Master module for MaWo app > > > Key: YARN-8599 > URL: https://issues.apache.org/jira/browse/YARN-8599 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Yesha Vora >Assignee: Yesha Vora >Priority: Major > > Master component for MaWo application is responsible for driving end-to-end > job execution. Its responsibility is > * Get Job definition and create a Queue of Tasks > * Assign Tasks to Worker > * Manage Workers lifecycle -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8596) Wrong behavior in SQLFederationStateStore when submitting the same app in the same subcluster
[ https://issues.apache.org/jira/browse/YARN-8596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated YARN-8596: --- Attachment: YARN-8596.v1.patch > Wrong behavior in SQLFederationStateStore when submitting the same app in the > same subcluster > - > > Key: YARN-8596 > URL: https://issues.apache.org/jira/browse/YARN-8596 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Íñigo Goiri >Assignee: Giovanni Matteo Fumarola >Priority: Major > Attachments: YARN-8596.v1.patch > > > Wrong behavior in SQLFederationStateStore when we want to submit the same > application in the same subcluster -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8599) Build Master module for MaWo app
Yesha Vora created YARN-8599: Summary: Build Master module for MaWo app Key: YARN-8599 URL: https://issues.apache.org/jira/browse/YARN-8599 Project: Hadoop YARN Issue Type: Sub-task Reporter: Yesha Vora Master component for MaWo application is responsible for driving end-to-end job execution. Its responsibility is * Get Job definition and create a Queue of Tasks * Assign Tasks to Worker * Manage Workers lifecycle -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8598) Build Master Job Module for MaWo Application
Yesha Vora created YARN-8598: Summary: Build Master Job Module for MaWo Application Key: YARN-8598 URL: https://issues.apache.org/jira/browse/YARN-8598 Project: Hadoop YARN Issue Type: Sub-task Reporter: Yesha Vora A job in MaWo application is a collection of Tasks. A Job consists of a setup task, a list of tasks and a teardown task. * JobBuilder ** SimpleTaskJobBuilder : SimpleJobBuilder should be able to parse simpleJobdescription file. In this file format, each line is considered as Task ** SimpleTaskJsonJobBuilder: Utility to parse json job description file. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-8598) Build Master Job Module for MaWo Application
[ https://issues.apache.org/jira/browse/YARN-8598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yesha Vora reassigned YARN-8598: Assignee: Yesha Vora > Build Master Job Module for MaWo Application > > > Key: YARN-8598 > URL: https://issues.apache.org/jira/browse/YARN-8598 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Yesha Vora >Assignee: Yesha Vora >Priority: Major > > A job in MaWo application is a collection of Tasks. A Job consists of a setup > task, a list of tasks and a teardown task. > * JobBuilder > ** SimpleTaskJobBuilder : SimpleJobBuilder should be able to parse > simpleJobdescription file. In this file format, each line is considered as > Task > ** SimpleTaskJsonJobBuilder: Utility to parse json job description file. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8407) Container launch exception in AM log should be printed in ERROR level
[
https://issues.apache.org/jira/browse/YARN-8407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560120#comment-16560120
]
Suma Shivaprasad edited comment on YARN-8407 at 7/27/18 6:40 PM:
-
Thanks [~yeshavora] Patch LGTM . Can you pls submit patch for jenkins?
was (Author: suma.shivaprasad):
Thanks [~yeshavora] Patch LGTM
> Container launch exception in AM log should be printed in ERROR level
> -
>
> Key: YARN-8407
> URL: https://issues.apache.org/jira/browse/YARN-8407
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Yesha Vora
>Assignee: Yesha Vora
>Priority: Major
> Attachments: YARN-8407.001.patch, YARN-8407.002.patch
>
>
> when a container launch is failing due to docker image not available is
> logged as INFO level in AM log.
> Container launch failure should be logged as ERROR.
> Steps:
> launch httpd yarn-service application with invalid docker image
>
> {code:java}
> 2018-06-07 01:51:32,966 [Component dispatcher] INFO
> instance.ComponentInstance - [COMPINSTANCE httpd-0 :
> container_e05_1528335963594_0001_01_02]:
> container_e05_1528335963594_0001_01_02 completed. Reinsert back to
> pending list and requested a new container.
> exitStatus=-1, diagnostics=[2018-06-07 01:51:02.363]Exception from
> container-launch.
> Container id: container_e05_1528335963594_0001_01_02
> Exit code: 7
> Exception message: Launch container failed
> Shell error output: Unable to find image 'xxx/httpd:0.1' locally
> Trying to pull repository xxx/httpd ...
> /usr/bin/docker-current: Get https://xxx/v1/_ping: dial tcp: lookup xxx on
> yyy: no such host.
> See '/usr/bin/docker-current run --help'.
> Shell output: main : command provided 4
> main : run as user is hbase
> main : requested yarn user is hbase
> Creating script paths...
> Creating local dirs...
> Getting exit code file...
> Changing effective user to root...
> Wrote the exit code 7 to
> /grid/0/hadoop/yarn/local/nmPrivate/application_1528335963594_0001/container_e05_1528335963594_0001_01_02/container_e05_1528335963594_0001_01_02.pid.exitcode
> [2018-06-07 01:51:02.393]Diagnostic message from attempt :
> [2018-06-07 01:51:02.394]Container exited with a non-zero exit code 7. Last
> 4096 bytes of stderr.txt :
> [2018-06-07 01:51:32.428]Could not find
> nmPrivate/application_1528335963594_0001/container_e05_1528335963594_0001_01_02//container_e05_1528335963594_0001_01_02.pid
> in any of the directories
> 2018-06-07 01:51:32,966 [Component dispatcher] INFO
> instance.ComponentInstance - [COMPINSTANCE httpd-0 :
> container_e05_1528335963594_0001_01_02] Transitioned from STARTED to INIT
> on STOP event{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8407) Container launch exception in AM log should be printed in ERROR level
[
https://issues.apache.org/jira/browse/YARN-8407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560120#comment-16560120
]
Suma Shivaprasad commented on YARN-8407:
Thanks [~yeshavora] Patch LGTM
> Container launch exception in AM log should be printed in ERROR level
> -
>
> Key: YARN-8407
> URL: https://issues.apache.org/jira/browse/YARN-8407
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Yesha Vora
>Assignee: Yesha Vora
>Priority: Major
> Attachments: YARN-8407.001.patch, YARN-8407.002.patch
>
>
> when a container launch is failing due to docker image not available is
> logged as INFO level in AM log.
> Container launch failure should be logged as ERROR.
> Steps:
> launch httpd yarn-service application with invalid docker image
>
> {code:java}
> 2018-06-07 01:51:32,966 [Component dispatcher] INFO
> instance.ComponentInstance - [COMPINSTANCE httpd-0 :
> container_e05_1528335963594_0001_01_02]:
> container_e05_1528335963594_0001_01_02 completed. Reinsert back to
> pending list and requested a new container.
> exitStatus=-1, diagnostics=[2018-06-07 01:51:02.363]Exception from
> container-launch.
> Container id: container_e05_1528335963594_0001_01_02
> Exit code: 7
> Exception message: Launch container failed
> Shell error output: Unable to find image 'xxx/httpd:0.1' locally
> Trying to pull repository xxx/httpd ...
> /usr/bin/docker-current: Get https://xxx/v1/_ping: dial tcp: lookup xxx on
> yyy: no such host.
> See '/usr/bin/docker-current run --help'.
> Shell output: main : command provided 4
> main : run as user is hbase
> main : requested yarn user is hbase
> Creating script paths...
> Creating local dirs...
> Getting exit code file...
> Changing effective user to root...
> Wrote the exit code 7 to
> /grid/0/hadoop/yarn/local/nmPrivate/application_1528335963594_0001/container_e05_1528335963594_0001_01_02/container_e05_1528335963594_0001_01_02.pid.exitcode
> [2018-06-07 01:51:02.393]Diagnostic message from attempt :
> [2018-06-07 01:51:02.394]Container exited with a non-zero exit code 7. Last
> 4096 bytes of stderr.txt :
> [2018-06-07 01:51:32.428]Could not find
> nmPrivate/application_1528335963594_0001/container_e05_1528335963594_0001_01_02//container_e05_1528335963594_0001_01_02.pid
> in any of the directories
> 2018-06-07 01:51:32,966 [Component dispatcher] INFO
> instance.ComponentInstance - [COMPINSTANCE httpd-0 :
> container_e05_1528335963594_0001_01_02] Transitioned from STARTED to INIT
> on STOP event{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7708) [GPG] Load based policy generator
[
https://issues.apache.org/jira/browse/YARN-7708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560112#comment-16560112
]
genericqa commented on YARN-7708:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m
9s{color} | {color:red} Docker failed to build yetus/hadoop:abb62dd. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-7708 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12933411/YARN-7708-YARN-7402.04.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/21405/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> [GPG] Load based policy generator
> -
>
> Key: YARN-7708
> URL: https://issues.apache.org/jira/browse/YARN-7708
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Carlo Curino
>Assignee: Young Chen
>Priority: Major
> Attachments: YARN-7708-YARN-7402.01.cumulative.patch,
> YARN-7708-YARN-7402.01.patch, YARN-7708-YARN-7402.02.cumulative.patch,
> YARN-7708-YARN-7402.02.patch, YARN-7708-YARN-7402.03.cumulative.patch,
> YARN-7708-YARN-7402.03.patch, YARN-7708-YARN-7402.03.patch,
> YARN-7708-YARN-7402.04.cumulative.patch, YARN-7708-YARN-7402.04.patch,
> YARN-7708-YARN-7402.05.cumulative.patch,
> YARN-7708-YARN-7402.06.cumulative.patch,
> YARN-7708-YARN-7402.07.cumulative.patch
>
>
> This policy reads load from the "pendingQueueLength" metrics and provides
> scaling into a set of weights that influence the AMRMProxy and Router
> behaviors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-8597) Build Worker utility for MaWo Application
[ https://issues.apache.org/jira/browse/YARN-8597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yesha Vora reassigned YARN-8597: Assignee: Yesha Vora > Build Worker utility for MaWo Application > - > > Key: YARN-8597 > URL: https://issues.apache.org/jira/browse/YARN-8597 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Yesha Vora >Assignee: Yesha Vora >Priority: Major > > The worker is responsible for executing Tasks. > * Worker > ** Create a worker class which drives worker life cycle > ** Create WorkAssignment Protocol. It should be handle Register/deregister > worker, send heartbeat > ** Lifecycle: Register worker, Run Setup Task, Get Task from master and > execute it using TaskRunner, Run Teardown Task > * TaskRunner > ** Simple Task Runner : This runner should be able to execute a simple task > ** Composite Task Runner: This runner should be able to execute composite > task > * TaskWallTimeLimiter > ** Create a utility which can abort the task if the execution time exceeds > task timeout. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8597) Build Worker utility for MaWo Application
Yesha Vora created YARN-8597: Summary: Build Worker utility for MaWo Application Key: YARN-8597 URL: https://issues.apache.org/jira/browse/YARN-8597 Project: Hadoop YARN Issue Type: Sub-task Reporter: Yesha Vora The worker is responsible for executing Tasks. * Worker ** Create a worker class which drives worker life cycle ** Create WorkAssignment Protocol. It should be handle Register/deregister worker, send heartbeat ** Lifecycle: Register worker, Run Setup Task, Get Task from master and execute it using TaskRunner, Run Teardown Task * TaskRunner ** Simple Task Runner : This runner should be able to execute a simple task ** Composite Task Runner: This runner should be able to execute composite task * TaskWallTimeLimiter ** Create a utility which can abort the task if the execution time exceeds task timeout. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8596) Wrong behavior in SQLFederationStateStore when submitting the same app in the same subcluster
Íñigo Goiri created YARN-8596: - Summary: Wrong behavior in SQLFederationStateStore when submitting the same app in the same subcluster Key: YARN-8596 URL: https://issues.apache.org/jira/browse/YARN-8596 Project: Hadoop YARN Issue Type: Bug Reporter: Íñigo Goiri Assignee: Giovanni Matteo Fumarola Wrong behavior in SQLFederationStateStore when we want to submit the same application in the same subcluster -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8429) Improve diagnostic message when artifact is not set properly
[
https://issues.apache.org/jira/browse/YARN-8429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560105#comment-16560105
]
Gour Saha commented on YARN-8429:
-
Awesome. Thanks again [~eyang].
> Improve diagnostic message when artifact is not set properly
>
>
> Key: YARN-8429
> URL: https://issues.apache.org/jira/browse/YARN-8429
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.1.1
>Reporter: Yesha Vora
>Assignee: Gour Saha
>Priority: Major
> Fix For: 3.2.0, 3.1.2
>
> Attachments: YARN-8429.001.patch, YARN-8429.002.patch,
> YARN-8429.003.patch, YARN-8429.004.patch
>
>
> Steps:
> 1) Create launch json file. Replace "artifact" with "artifacts"
> 2) launch yarn service app with cli
> The application launch fails with below error
> {code}
> [xxx xxx]$ yarn app -launch test2-2 test.json
> 18/06/14 17:08:00 INFO client.AHSProxy: Connecting to Application History
> server at xxx/xxx:10200
> 18/06/14 17:08:00 INFO client.AHSProxy: Connecting to Application History
> server at xxx/xxx:10200
> 18/06/14 17:08:00 INFO client.ApiServiceClient: Loading service definition
> from local FS: /xxx/test.json
> 18/06/14 17:08:01 INFO util.log: Logging initialized @2782ms
> 18/06/14 17:08:01 ERROR client.ApiServiceClient: Dest_file must not be
> absolute path: /xxx/xxx
> {code}
> artifact field is not mandatory. However, If that field is specified
> incorrectly, launch cmd should fail with proper error.
> Here, The error message regarding Dest file is misleading.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8571) Validate service principal format prior to launching yarn service
[
https://issues.apache.org/jira/browse/YARN-8571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560106#comment-16560106
]
Billie Rinaldi commented on YARN-8571:
--
+1 for patch 002. Thanks, [~eyang]!
> Validate service principal format prior to launching yarn service
> -
>
> Key: YARN-8571
> URL: https://issues.apache.org/jira/browse/YARN-8571
> Project: Hadoop YARN
> Issue Type: Bug
> Components: security, yarn
>Affects Versions: 3.1.0
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN-8571.001.patch, YARN-8571.002.patch
>
>
> Hadoop client and server interaction is designed to validate the service
> principal before RPC request is permitted. In YARN service, the same
> security model is enforced to prevent replay attack. However, end user
> might submit JSON that looks like this to YARN service REST API:
> {code}
> {
> "name": "sleeper-service",
> "version": "1.0.0",
> "components" :
> [
> {
> "name": "sleeper",
> "number_of_containers": 2,
> "launch_command": "sleep 90",
> "resource": {
> "cpus": 1,
> "memory": "256"
> }
> }
> ],
> "kerberos_principal" : {
> "principal_name" : "ambari...@example.com",
> "keytab" : "file:///etc/security/keytabs/smokeuser.headless.keytab"
> }
> }
> {code}
> The kerberos principal is end user kerberos principal instead of service
> principal. This does not work properly because YARN service application
> master requires to run with a service principal to communicate with YARN CLI
> client via Hadoop RPC. Without breaking Hadoop security design in this JIRA,
> it might be in our best interest to validate principal_name during
> submission, and report error message when someone tries to run YARN service
> with user principal.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7708) [GPG] Load based policy generator
[ https://issues.apache.org/jira/browse/YARN-7708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Young Chen updated YARN-7708: - Attachment: YARN-7708-YARN-7402.04.patch > [GPG] Load based policy generator > - > > Key: YARN-7708 > URL: https://issues.apache.org/jira/browse/YARN-7708 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Carlo Curino >Assignee: Young Chen >Priority: Major > Attachments: YARN-7708-YARN-7402.01.cumulative.patch, > YARN-7708-YARN-7402.01.patch, YARN-7708-YARN-7402.02.cumulative.patch, > YARN-7708-YARN-7402.02.patch, YARN-7708-YARN-7402.03.cumulative.patch, > YARN-7708-YARN-7402.03.patch, YARN-7708-YARN-7402.03.patch, > YARN-7708-YARN-7402.04.cumulative.patch, YARN-7708-YARN-7402.04.patch, > YARN-7708-YARN-7402.05.cumulative.patch, > YARN-7708-YARN-7402.06.cumulative.patch, > YARN-7708-YARN-7402.07.cumulative.patch > > > This policy reads load from the "pendingQueueLength" metrics and provides > scaling into a set of weights that influence the AMRMProxy and Router > behaviors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7833) [PERF/TEST] Extend SLS to support simulation of a Federated Environment
[ https://issues.apache.org/jira/browse/YARN-7833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tanuj Nayak updated YARN-7833: -- Attachment: YARN-7833.v3.patch > [PERF/TEST] Extend SLS to support simulation of a Federated Environment > --- > > Key: YARN-7833 > URL: https://issues.apache.org/jira/browse/YARN-7833 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Carlo Curino >Assignee: Tanuj Nayak >Priority: Major > Attachments: YARN-7833.v1.patch, YARN-7833.v2.patch, > YARN-7833.v3.patch > > > To develop algorithms for federation, it would be of great help to have a > version of SLS that supports multi RMs and GPG. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4175) Example of use YARN-1197
[ https://issues.apache.org/jira/browse/YARN-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560087#comment-16560087 ] Arun Suresh commented on YARN-4175: --- Thanks for taking this up.. yup makes sense . Do post your patch.. I'll try to take a look over the weekend. > Example of use YARN-1197 > > > Key: YARN-4175 > URL: https://issues.apache.org/jira/browse/YARN-4175 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: MENG DING >Priority: Major > Attachments: YARN-4175.003.patch, YARN-4175.1.patch, YARN-4175.2.patch > > > Like YARN-2609, we need a example program to demonstrate how to use YARN-1197 > from end-to-end. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7833) [PERF/TEST] Extend SLS to support simulation of a Federated Environment
[ https://issues.apache.org/jira/browse/YARN-7833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tanuj Nayak updated YARN-7833: -- Attachment: YARN-7833.v2.patch > [PERF/TEST] Extend SLS to support simulation of a Federated Environment > --- > > Key: YARN-7833 > URL: https://issues.apache.org/jira/browse/YARN-7833 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Carlo Curino >Assignee: Tanuj Nayak >Priority: Major > Attachments: YARN-7833.v1.patch, YARN-7833.v2.patch > > > To develop algorithms for federation, it would be of great help to have a > version of SLS that supports multi RMs and GPG. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7833) [PERF/TEST] Extend SLS to support simulation of a Federated Environment
[ https://issues.apache.org/jira/browse/YARN-7833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tanuj Nayak updated YARN-7833: -- Attachment: (was: YARN-7833.v2.patch) > [PERF/TEST] Extend SLS to support simulation of a Federated Environment > --- > > Key: YARN-7833 > URL: https://issues.apache.org/jira/browse/YARN-7833 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Carlo Curino >Assignee: Tanuj Nayak >Priority: Major > Attachments: YARN-7833.v1.patch > > > To develop algorithms for federation, it would be of great help to have a > version of SLS that supports multi RMs and GPG. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7833) [PERF/TEST] Extend SLS to support simulation of a Federated Environment
[ https://issues.apache.org/jira/browse/YARN-7833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tanuj Nayak updated YARN-7833: -- Attachment: YARN-7833.v2.patch > [PERF/TEST] Extend SLS to support simulation of a Federated Environment > --- > > Key: YARN-7833 > URL: https://issues.apache.org/jira/browse/YARN-7833 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Carlo Curino >Assignee: Tanuj Nayak >Priority: Major > Attachments: YARN-7833.v1.patch, YARN-7833.v2.patch > > > To develop algorithms for federation, it would be of great help to have a > version of SLS that supports multi RMs and GPG. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8590) Fair scheduler promotion does not update container execution type and token
[
https://issues.apache.org/jira/browse/YARN-8590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560060#comment-16560060
]
genericqa commented on YARN-8590:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m
8s{color} | {color:red} Docker failed to build yetus/hadoop:abb62dd. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-8590 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12933401/YARN-8590-YARN-1011.01.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/21403/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Fair scheduler promotion does not update container execution type and token
> ---
>
> Key: YARN-8590
> URL: https://issues.apache.org/jira/browse/YARN-8590
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Affects Versions: YARN-1011
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>Priority: Major
> Attachments: YARN-8590-YARN-1011.00.patch,
> YARN-8590-YARN-1011.01.patch
>
>
> Fair Scheduler promotion of opportunistic containers does not update
> container execution type and token. This leads to incorrect resource
> accounting when the promoted containers are released.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4175) Example of use YARN-1197
[
https://issues.apache.org/jira/browse/YARN-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560054#comment-16560054
]
Manikandan R commented on YARN-4175:
Can you please review and share your comments?
{quote}Ideally, it has to be converted based on server side RM config
(Something similar to YARN-7159) . For example, If resource type 'resource1'
unit is 'Gi' configured at RM config and clients are passing values in
different units, it has to be converted to 'Gi' not to 'Mi'. Thoughts? I can
raise a separate JIRA to handle this separately based on comments.
{quote}
Does it makes sense? Can I start this new JIRA?
> Example of use YARN-1197
>
>
> Key: YARN-4175
> URL: https://issues.apache.org/jira/browse/YARN-4175
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, nodemanager, resourcemanager
>Reporter: Wangda Tan
>Assignee: MENG DING
>Priority: Major
> Attachments: YARN-4175.003.patch, YARN-4175.1.patch, YARN-4175.2.patch
>
>
> Like YARN-2609, we need a example program to demonstrate how to use YARN-1197
> from end-to-end.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7518) Node manager should allow resource units to be lower cased
[ https://issues.apache.org/jira/browse/YARN-7518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560047#comment-16560047 ] Manikandan R commented on YARN-7518: Since units like "m" and "M" have different meanings, Is it ok to ignore case while doing unit checks and allow? or Am i missing anything? > Node manager should allow resource units to be lower cased > -- > > Key: YARN-7518 > URL: https://issues.apache.org/jira/browse/YARN-7518 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-beta1, 3.1.0 >Reporter: Daniel Templeton >Priority: Major > > When we do units checks, we should ignore case. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8590) Fair scheduler promotion does not update container execution type and token
[ https://issues.apache.org/jira/browse/YARN-8590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8590: - Attachment: YARN-8590-YARN-1011.01.patch > Fair scheduler promotion does not update container execution type and token > --- > > Key: YARN-8590 > URL: https://issues.apache.org/jira/browse/YARN-8590 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Affects Versions: YARN-1011 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Major > Attachments: YARN-8590-YARN-1011.00.patch, > YARN-8590-YARN-1011.01.patch > > > Fair Scheduler promotion of opportunistic containers does not update > container execution type and token. This leads to incorrect resource > accounting when the promoted containers are released. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8590) Fair scheduler promotion does not update container execution type and token
[ https://issues.apache.org/jira/browse/YARN-8590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8590: - Attachment: (was: YARN-8590-YARN-1011.01.patch) > Fair scheduler promotion does not update container execution type and token > --- > > Key: YARN-8590 > URL: https://issues.apache.org/jira/browse/YARN-8590 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Affects Versions: YARN-1011 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Major > Attachments: YARN-8590-YARN-1011.00.patch, > YARN-8590-YARN-1011.01.patch > > > Fair Scheduler promotion of opportunistic containers does not update > container execution type and token. This leads to incorrect resource > accounting when the promoted containers are released. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8590) Fair scheduler promotion does not update container execution type and token
[ https://issues.apache.org/jira/browse/YARN-8590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560030#comment-16560030 ] Haibo Chen commented on YARN-8590: -- Updated the patch to increment container token identifier version so that NM is willing to accept and handle the promotion. > Fair scheduler promotion does not update container execution type and token > --- > > Key: YARN-8590 > URL: https://issues.apache.org/jira/browse/YARN-8590 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Affects Versions: YARN-1011 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Major > Attachments: YARN-8590-YARN-1011.00.patch, > YARN-8590-YARN-1011.01.patch > > > Fair Scheduler promotion of opportunistic containers does not update > container execution type and token. This leads to incorrect resource > accounting when the promoted containers are released. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8590) Fair scheduler promotion does not update container execution type and token
[
https://issues.apache.org/jira/browse/YARN-8590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560031#comment-16560031
]
genericqa commented on YARN-8590:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m
8s{color} | {color:red} Docker failed to build yetus/hadoop:abb62dd. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-8590 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12933400/YARN-8590-YARN-1011.01.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/21402/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Fair scheduler promotion does not update container execution type and token
> ---
>
> Key: YARN-8590
> URL: https://issues.apache.org/jira/browse/YARN-8590
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Affects Versions: YARN-1011
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>Priority: Major
> Attachments: YARN-8590-YARN-1011.00.patch,
> YARN-8590-YARN-1011.01.patch
>
>
> Fair Scheduler promotion of opportunistic containers does not update
> container execution type and token. This leads to incorrect resource
> accounting when the promoted containers are released.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8590) Fair scheduler promotion does not update container execution type and token
[ https://issues.apache.org/jira/browse/YARN-8590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8590: - Attachment: YARN-8590-YARN-1011.01.patch > Fair scheduler promotion does not update container execution type and token > --- > > Key: YARN-8590 > URL: https://issues.apache.org/jira/browse/YARN-8590 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Affects Versions: YARN-1011 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Major > Attachments: YARN-8590-YARN-1011.00.patch, > YARN-8590-YARN-1011.01.patch > > > Fair Scheduler promotion of opportunistic containers does not update > container execution type and token. This leads to incorrect resource > accounting when the promoted containers are released. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8587) Delays are noticed to launch docker container
[
https://issues.apache.org/jira/browse/YARN-8587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560026#comment-16560026
]
Eric Yang commented on YARN-8587:
-
[~yeshavora] YARN state machine transition from SCHEDULED to RUNNING then run
container-executor with the docker run command. SSH command tests happen in
parallel while container-executor launch docker run command. This may report
incorrect result from v1 API when container-executor and docker run command
take more time to start. We might want to query for container sub-state
RUNNING_BUT_NOT_READY from YARN service REST API (or yarn app -status
[appname]) to determine if docker run command is actually started. Only run
docker ps -a after container sub-state has changed to RUNNING.
> Delays are noticed to launch docker container
> -
>
> Key: YARN-8587
> URL: https://issues.apache.org/jira/browse/YARN-8587
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.1.1
>Reporter: Yesha Vora
>Priority: Major
> Labels: Docker
>
> Launch dshell application. Wait for application to go in RUNNING state.
> {code:java}
> yarn jar /xx/hadoop-yarn-applications-distributedshell-*.jar -shell_command
> "sleep 300" -num_containers 1 -shell_env YARN_CONTAINER_RUNTIME_TYPE=docker
> -shell_env YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=httpd:0.1 -shell_env
> YARN_CONTAINER_RUNTIME_DOCKER_DELAYED_REMOVAL=true -jar
> /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell-xx.jar
> {code}
> Find out container allocation. Run docker inspect command for docker
> containers launched by app.
> Sometimes, the container is allocated to NM but docker PID is not up.
> {code:java}
> Command ssh -q -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
> xxx "sudo su - -c \"docker ps -a | grep
> container_e02_1531189225093_0003_01_02\" root" failed after 0 retries
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8595) [UI2] Container diagnostic missing from container page
[
https://issues.apache.org/jira/browse/YARN-8595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559982#comment-16559982
]
genericqa commented on YARN-8595:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
37m 52s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 26s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 51m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | YARN-8595 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12933384/YARN-8595.001.patch |
| Optional Tests | asflicense shadedclient |
| uname | Linux 0568d1a08f0b 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 3cc7ce8 |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 302 (vs. ulimit of 1) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/21400/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> [UI2] Container diagnostic missing from container page
> --
>
> Key: YARN-8595
> URL: https://issues.apache.org/jira/browse/YARN-8595
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-ui-v2
>Reporter: Akhil PB
>Assignee: Akhil PB
>Priority: Major
> Attachments: YARN-8595.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8591) [ATSv2] NPE error while checking for entity acl in non-secure cluster
[
https://issues.apache.org/jira/browse/YARN-8591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559947#comment-16559947
]
Sunil Govindan commented on YARN-8591:
--
Thanks [~rohithsharma]. This is a straight forward patch and thanks for adding
test also.
Looks good. I could commit this tomo if there are no objections.
> [ATSv2] NPE error while checking for entity acl in non-secure cluster
> -
>
> Key: YARN-8591
> URL: https://issues.apache.org/jira/browse/YARN-8591
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelinereader, timelineserver
>Reporter: Akhil PB
>Assignee: Rohith Sharma K S
>Priority: Major
> Attachments: YARN-8591.01.patch
>
>
> {code:java}
> GET
> http://ctr-e138-1518143905142-417433-01-04.hwx.site:8198/ws/v2/timeline/apps/application_1532578985272_0002/entities/YARN_CONTAINER?fields=ALL&_=1532670071899{code}
> {code:java}
> 2018-07-27 05:32:03,468 WARN webapp.GenericExceptionHandler
> (GenericExceptionHandler.java:toResponse(98)) - INTERNAL_SERVER_ERROR
> javax.ws.rs.WebApplicationException: java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.handleException(TimelineReaderWebServices.java:196)
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.getEntities(TimelineReaderWebServices.java:624)
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.getEntities(TimelineReaderWebServices.java:474)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.security.TimelineReaderWhitelistAuthorizationFilter.doFilter(TimelineReaderWhitelistAuthorizationFilter.java:85)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:644)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.security.http.CrossOriginFilter.doFilter(CrossOriginFilter.java:98)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
>
[jira] [Comment Edited] (YARN-8587) Delays are noticed to launch docker container
[
https://issues.apache.org/jira/browse/YARN-8587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558907#comment-16558907
]
Eric Yang edited comment on YARN-8587 at 7/27/18 4:00 PM:
--
There is backward incompatibility concern with distributed shell, where we
allow user to specify multiple unix command and output redirection of log file.
For fixing this transient false positive, logging mechanism behavior will
change. stderr, stdout will contain command output. stderr.txt and stdout.txt
will contain more information including command launched, and docker errors.
Hence, this can only be fixed if we agree that the incompatible change is
negligible.
was (Author: eyang):
There is backward incompatibility concern with distributed shell, where we
allow user to specify multiple unix command and output redirection of log file.
For fixing this transient false positive, logging mechanism behavior will
change. stderr, stdout will contain command output. stderr.txt and stdout.txt
will container more information including command launched, and docker errors.
Hence, this can only be fixed if we agree that the incompatible change is
negligible.
> Delays are noticed to launch docker container
> -
>
> Key: YARN-8587
> URL: https://issues.apache.org/jira/browse/YARN-8587
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.1.1
>Reporter: Yesha Vora
>Priority: Major
> Labels: Docker
>
> Launch dshell application. Wait for application to go in RUNNING state.
> {code:java}
> yarn jar /xx/hadoop-yarn-applications-distributedshell-*.jar -shell_command
> "sleep 300" -num_containers 1 -shell_env YARN_CONTAINER_RUNTIME_TYPE=docker
> -shell_env YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=httpd:0.1 -shell_env
> YARN_CONTAINER_RUNTIME_DOCKER_DELAYED_REMOVAL=true -jar
> /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell-xx.jar
> {code}
> Find out container allocation. Run docker inspect command for docker
> containers launched by app.
> Sometimes, the container is allocated to NM but docker PID is not up.
> {code:java}
> Command ssh -q -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
> xxx "sudo su - -c \"docker ps -a | grep
> container_e02_1531189225093_0003_01_02\" root" failed after 0 retries
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8592) [UI2] Resource Manager Web UI V2 shows a blank page in windows OS And Chrome browser
[
https://issues.apache.org/jira/browse/YARN-8592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akhil PB updated YARN-8592:
---
Summary: [UI2] Resource Manager Web UI V2 shows a blank page in windows OS
And Chrome browser (was: [ui2]Resource Manager Web UI V2 shows a blank page in
windows OS And Chrome browser)
> [UI2] Resource Manager Web UI V2 shows a blank page in windows OS And Chrome
> browser
>
>
> Key: YARN-8592
> URL: https://issues.apache.org/jira/browse/YARN-8592
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Akhil S Naik
>Assignee: Akhil PB
>Priority: Major
>
> after successfully installing ambari 2.7v and hdp 3.0 in Windows Operating
> Sytem .
> But, When we connect to Resource Manager UI(v2), web browser shows a blank
> page.
> In a chrome development tool, this error is shown
> Stack trace..
> {code:java}
> TypeError: Cannot read property 'join's null
> at new i (hostname:8088/ui2/assets/vendor.js:5598:40988)
> at l (hostname:8088/ui2/assets/vendor.js:5598:41338)
> at p (hostname:8088/ui2/assets/vendor.js:5598:42035)
> at Function.q [as guess] (hostname:8088/ui2/assets/vendor.js:5598:42235)
> at getDefaultTimezone (hostname:8088/ui2/assets/yarn-ui.js:366:445)
> at convertTimestampWithT (hostname:8088/ui2/assets/yarn-ui.js:366:656)
> at Object.timeStampToDate (hostname:8088/ui2/assets/yarn-ui.js:348:127)
> at Class.internalNormalizeSingleResponse
> (hostname:8088/ui2/assets/yarn-ui.js:287:73)
> at Class. (hostname:8088/ui2/assets/yarn-ui.js:288:2024)
> at Array.map (native)
> {code}
> The Error is hit in the underlying framework that Yarn-UI uses :
> moment-timezone-with-data-2010-2020.min.js
> this is the discussion happening in framework github for the same :
> https://github.com/moment/moment-timezone/issues/294
> Please find the discussion on this bug here :
> https://community.hortonworks.com/questions/208887/resource-manager-web-ui-v2-shows-a-blank-page.html
> It works well in firefox browser in windows Operating systems.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-8592) [ui2]Resource Manager Web UI V2 shows a blank page in windows OS And Chrome browser
[
https://issues.apache.org/jira/browse/YARN-8592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akhil PB reassigned YARN-8592:
--
Assignee: Akhil PB
> [ui2]Resource Manager Web UI V2 shows a blank page in windows OS And Chrome
> browser
> ---
>
> Key: YARN-8592
> URL: https://issues.apache.org/jira/browse/YARN-8592
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Akhil S Naik
>Assignee: Akhil PB
>Priority: Major
>
> after successfully installing ambari 2.7v and hdp 3.0 in Windows Operating
> Sytem .
> But, When we connect to Resource Manager UI(v2), web browser shows a blank
> page.
> In a chrome development tool, this error is shown
> Stack trace..
> {code:java}
> TypeError: Cannot read property 'join's null
> at new i (hostname:8088/ui2/assets/vendor.js:5598:40988)
> at l (hostname:8088/ui2/assets/vendor.js:5598:41338)
> at p (hostname:8088/ui2/assets/vendor.js:5598:42035)
> at Function.q [as guess] (hostname:8088/ui2/assets/vendor.js:5598:42235)
> at getDefaultTimezone (hostname:8088/ui2/assets/yarn-ui.js:366:445)
> at convertTimestampWithT (hostname:8088/ui2/assets/yarn-ui.js:366:656)
> at Object.timeStampToDate (hostname:8088/ui2/assets/yarn-ui.js:348:127)
> at Class.internalNormalizeSingleResponse
> (hostname:8088/ui2/assets/yarn-ui.js:287:73)
> at Class. (hostname:8088/ui2/assets/yarn-ui.js:288:2024)
> at Array.map (native)
> {code}
> The Error is hit in the underlying framework that Yarn-UI uses :
> moment-timezone-with-data-2010-2020.min.js
> this is the discussion happening in framework github for the same :
> https://github.com/moment/moment-timezone/issues/294
> Please find the discussion on this bug here :
> https://community.hortonworks.com/questions/208887/resource-manager-web-ui-v2-shows-a-blank-page.html
> It works well in firefox browser in windows Operating systems.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8590) Fair scheduler promotion does not update container execution type and token
[
https://issues.apache.org/jira/browse/YARN-8590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559933#comment-16559933
]
genericqa commented on YARN-8590:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m
9s{color} | {color:red} Docker failed to build yetus/hadoop:abb62dd. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-8590 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12933282/YARN-8590-YARN-1011.00.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/21401/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Fair scheduler promotion does not update container execution type and token
> ---
>
> Key: YARN-8590
> URL: https://issues.apache.org/jira/browse/YARN-8590
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Affects Versions: YARN-1011
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>Priority: Major
> Attachments: YARN-8590-YARN-1011.00.patch
>
>
> Fair Scheduler promotion of opportunistic containers does not update
> container execution type and token. This leads to incorrect resource
> accounting when the promoted containers are released.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8595) [UI2] Container diagnostic missing from container page
Akhil PB created YARN-8595: -- Summary: [UI2] Container diagnostic missing from container page Key: YARN-8595 URL: https://issues.apache.org/jira/browse/YARN-8595 Project: Hadoop YARN Issue Type: Bug Components: yarn-ui-v2 Reporter: Akhil PB Assignee: Akhil PB -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8429) Improve diagnostic message when artifact is not set properly
[
https://issues.apache.org/jira/browse/YARN-8429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559896#comment-16559896
]
Eric Yang commented on YARN-8429:
-
[~gsaha] Thanks for catching my mistake. Patch 4 is merged to branch-3.1 now.
> Improve diagnostic message when artifact is not set properly
>
>
> Key: YARN-8429
> URL: https://issues.apache.org/jira/browse/YARN-8429
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.1.1
>Reporter: Yesha Vora
>Assignee: Gour Saha
>Priority: Major
> Fix For: 3.2.0, 3.1.2
>
> Attachments: YARN-8429.001.patch, YARN-8429.002.patch,
> YARN-8429.003.patch, YARN-8429.004.patch
>
>
> Steps:
> 1) Create launch json file. Replace "artifact" with "artifacts"
> 2) launch yarn service app with cli
> The application launch fails with below error
> {code}
> [xxx xxx]$ yarn app -launch test2-2 test.json
> 18/06/14 17:08:00 INFO client.AHSProxy: Connecting to Application History
> server at xxx/xxx:10200
> 18/06/14 17:08:00 INFO client.AHSProxy: Connecting to Application History
> server at xxx/xxx:10200
> 18/06/14 17:08:00 INFO client.ApiServiceClient: Loading service definition
> from local FS: /xxx/test.json
> 18/06/14 17:08:01 INFO util.log: Logging initialized @2782ms
> 18/06/14 17:08:01 ERROR client.ApiServiceClient: Dest_file must not be
> absolute path: /xxx/xxx
> {code}
> artifact field is not mandatory. However, If that field is specified
> incorrectly, launch cmd should fail with proper error.
> Here, The error message regarding Dest file is misleading.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8591) [ATSv2] NPE error while checking for entity acl in non-secure cluster
[
https://issues.apache.org/jira/browse/YARN-8591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559813#comment-16559813
]
genericqa commented on YARN-8591:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 39s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 26s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
59s{color} | {color:green} hadoop-yarn-server-timelineservice in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 56m 49s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | YARN-8591 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12933358/YARN-8591.01.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 07c1d962dbe2 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15
17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 64e739e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/21399/testReport/ |
| Max. process+thread count | 336 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/21399/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> [ATSv2] NPE error while checking for entity acl
[jira] [Commented] (YARN-8574) Allow dot in attribute values
[
https://issues.apache.org/jira/browse/YARN-8574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559798#comment-16559798
]
genericqa commented on YARN-8574:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
31s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} YARN-3409 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 31m
9s{color} | {color:green} YARN-3409 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} YARN-3409 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} YARN-3409 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} YARN-3409 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 13s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
26s{color} | {color:green} YARN-3409 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} YARN-3409 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 25s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
6s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 68m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | YARN-8574 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12933356/YARN-8574-YARN-3409.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 99b05b48efab 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | YARN-3409 / acd7729 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/21397/testReport/ |
| Max. process+thread count | 305 (vs. ulimit of 1) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/21397/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Allow dot in attribute values
> --
>
> Key:
[jira] [Commented] (YARN-8593) Add new RM web service endpoint to get cluster user info
[ https://issues.apache.org/jira/browse/YARN-8593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559760#comment-16559760 ] Rohith Sharma K S commented on YARN-8593: - End user required to know only RM login ugi right? Why requested user need to be send back? Requested user means client. Client user is same as requested user. > Add new RM web service endpoint to get cluster user info > > > Key: YARN-8593 > URL: https://issues.apache.org/jira/browse/YARN-8593 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Akhil PB >Assignee: Akhil PB >Priority: Major > Attachments: YARN-8593.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8155) Improve ATSv2 client logging in RM and NM publisher
[
https://issues.apache.org/jira/browse/YARN-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559746#comment-16559746
]
genericqa commented on YARN-8155:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 16m
38s{color} | {color:red} Docker failed to build yetus/hadoop:f667ef1. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-8155 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12933357/YARN-8155-branch-2.002.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/21398/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Improve ATSv2 client logging in RM and NM publisher
> ---
>
> Key: YARN-8155
> URL: https://issues.apache.org/jira/browse/YARN-8155
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Rohith Sharma K S
>Assignee: Abhishek Modi
>Priority: Major
> Fix For: 3.2.0, 3.1.1, 3.0.4
>
> Attachments: YARN-8155-branch-2.002.patch,
> YARN-8155-branch-2.v1.patch, YARN-8155.001.patch, YARN-8155.002.patch,
> YARN-8155.003.patch, YARN-8155.004.patch, YARN-8155.005.patch,
> YARN-8155.006.patch
>
>
> We see that NM logs are filled with larger stack trace of NotFoundException
> if collector is removed from one of the NM and other NMs are still publishing
> the entities.
>
> This Jira is to improve the logging in NM so that we log with informative
> message.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8591) [ATSv2] NPE error while checking for entity acl in non-secure cluster
[
https://issues.apache.org/jira/browse/YARN-8591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559735#comment-16559735
]
Rohith Sharma K S commented on YARN-8591:
-
Attached the simple patch. [~sunil.gov...@gmail.com] could you take a look at
patch please?
> [ATSv2] NPE error while checking for entity acl in non-secure cluster
> -
>
> Key: YARN-8591
> URL: https://issues.apache.org/jira/browse/YARN-8591
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelinereader, timelineserver
>Reporter: Akhil PB
>Assignee: Rohith Sharma K S
>Priority: Major
> Attachments: YARN-8591.01.patch
>
>
> {code:java}
> GET
> http://ctr-e138-1518143905142-417433-01-04.hwx.site:8198/ws/v2/timeline/apps/application_1532578985272_0002/entities/YARN_CONTAINER?fields=ALL&_=1532670071899{code}
> {code:java}
> 2018-07-27 05:32:03,468 WARN webapp.GenericExceptionHandler
> (GenericExceptionHandler.java:toResponse(98)) - INTERNAL_SERVER_ERROR
> javax.ws.rs.WebApplicationException: java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.handleException(TimelineReaderWebServices.java:196)
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.getEntities(TimelineReaderWebServices.java:624)
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.getEntities(TimelineReaderWebServices.java:474)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.security.TimelineReaderWhitelistAuthorizationFilter.doFilter(TimelineReaderWhitelistAuthorizationFilter.java:85)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:644)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.security.http.CrossOriginFilter.doFilter(CrossOriginFilter.java:98)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1604)
>
[jira] [Commented] (YARN-8591) [ATSv2] NPE error while checking for entity acl in non-secure cluster
[
https://issues.apache.org/jira/browse/YARN-8591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559732#comment-16559732
]
Rohith Sharma K S commented on YARN-8591:
-
In non-secure cluster, incoming UGI will be null if we don't use
static-web-app-filter. In such cases, ugi need to be validated against null
and return true.
> [ATSv2] NPE error while checking for entity acl in non-secure cluster
> -
>
> Key: YARN-8591
> URL: https://issues.apache.org/jira/browse/YARN-8591
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelinereader, timelineserver
>Reporter: Akhil PB
>Assignee: Rohith Sharma K S
>Priority: Major
> Attachments: YARN-8591.01.patch
>
>
> {code:java}
> GET
> http://ctr-e138-1518143905142-417433-01-04.hwx.site:8198/ws/v2/timeline/apps/application_1532578985272_0002/entities/YARN_CONTAINER?fields=ALL&_=1532670071899{code}
> {code:java}
> 2018-07-27 05:32:03,468 WARN webapp.GenericExceptionHandler
> (GenericExceptionHandler.java:toResponse(98)) - INTERNAL_SERVER_ERROR
> javax.ws.rs.WebApplicationException: java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.handleException(TimelineReaderWebServices.java:196)
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.getEntities(TimelineReaderWebServices.java:624)
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.getEntities(TimelineReaderWebServices.java:474)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.security.TimelineReaderWhitelistAuthorizationFilter.doFilter(TimelineReaderWhitelistAuthorizationFilter.java:85)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:644)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.security.http.CrossOriginFilter.doFilter(CrossOriginFilter.java:98)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
>
[jira] [Updated] (YARN-8591) [ATSv2] NPE error while checking for entity acl in non-secure cluster
[
https://issues.apache.org/jira/browse/YARN-8591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated YARN-8591:
Summary: [ATSv2] NPE error while checking for entity acl in non-secure
cluster (was: [ATSv2] YARN_CONTAINER API throws 500 INTERNAL_SERVER_ERROR)
> [ATSv2] NPE error while checking for entity acl in non-secure cluster
> -
>
> Key: YARN-8591
> URL: https://issues.apache.org/jira/browse/YARN-8591
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelinereader, timelineserver
>Reporter: Akhil PB
>Assignee: Rohith Sharma K S
>Priority: Major
> Attachments: YARN-8591.01.patch
>
>
> {code:java}
> GET
> http://ctr-e138-1518143905142-417433-01-04.hwx.site:8198/ws/v2/timeline/apps/application_1532578985272_0002/entities/YARN_CONTAINER?fields=ALL&_=1532670071899{code}
> {code:java}
> 2018-07-27 05:32:03,468 WARN webapp.GenericExceptionHandler
> (GenericExceptionHandler.java:toResponse(98)) - INTERNAL_SERVER_ERROR
> javax.ws.rs.WebApplicationException: java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.handleException(TimelineReaderWebServices.java:196)
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.getEntities(TimelineReaderWebServices.java:624)
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.getEntities(TimelineReaderWebServices.java:474)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.security.TimelineReaderWhitelistAuthorizationFilter.doFilter(TimelineReaderWhitelistAuthorizationFilter.java:85)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:644)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.security.http.CrossOriginFilter.doFilter(CrossOriginFilter.java:98)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
>
[jira] [Updated] (YARN-8591) [ATSv2] YARN_CONTAINER API throws 500 INTERNAL_SERVER_ERROR
[
https://issues.apache.org/jira/browse/YARN-8591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated YARN-8591:
Attachment: YARN-8591.01.patch
> [ATSv2] YARN_CONTAINER API throws 500 INTERNAL_SERVER_ERROR
> ---
>
> Key: YARN-8591
> URL: https://issues.apache.org/jira/browse/YARN-8591
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelinereader, timelineserver
>Reporter: Akhil PB
>Assignee: Rohith Sharma K S
>Priority: Major
> Attachments: YARN-8591.01.patch
>
>
> {code:java}
> GET
> http://ctr-e138-1518143905142-417433-01-04.hwx.site:8198/ws/v2/timeline/apps/application_1532578985272_0002/entities/YARN_CONTAINER?fields=ALL&_=1532670071899{code}
> {code:java}
> 2018-07-27 05:32:03,468 WARN webapp.GenericExceptionHandler
> (GenericExceptionHandler.java:toResponse(98)) - INTERNAL_SERVER_ERROR
> javax.ws.rs.WebApplicationException: java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.handleException(TimelineReaderWebServices.java:196)
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.getEntities(TimelineReaderWebServices.java:624)
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.getEntities(TimelineReaderWebServices.java:474)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.security.TimelineReaderWhitelistAuthorizationFilter.doFilter(TimelineReaderWhitelistAuthorizationFilter.java:85)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:644)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.security.http.CrossOriginFilter.doFilter(CrossOriginFilter.java:98)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1604)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
>
[jira] [Commented] (YARN-8593) Add new RM web service endpoint to get cluster user info
[
https://issues.apache.org/jira/browse/YARN-8593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559712#comment-16559712
]
genericqa commented on YARN-8593:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 32s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 7s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 68m 51s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
26s{color} | {color:green} hadoop-yarn-server-router in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}136m 53s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | YARN-8593 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12933342/YARN-8593.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 18c203603c8f 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 8d3c068 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| unit |
[jira] [Commented] (YARN-8593) Add new RM web service endpoint to get cluster user info
[
https://issues.apache.org/jira/browse/YARN-8593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559713#comment-16559713
]
Sunil Govindan commented on YARN-8593:
--
Yes. I also thought the same. But existing api {{public ClusterInfo
getClusterInfo()}} doesnt take *hsr* as input to get incoming request. Thoughts?
> Add new RM web service endpoint to get cluster user info
>
>
> Key: YARN-8593
> URL: https://issues.apache.org/jira/browse/YARN-8593
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: Akhil PB
>Assignee: Akhil PB
>Priority: Major
> Attachments: YARN-8593.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8593) Add new RM web service endpoint to get cluster user info
[ https://issues.apache.org/jira/browse/YARN-8593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559705#comment-16559705 ] Rohith Sharma K S commented on YARN-8593: - To me, it make sense more putting as part of ClusterInfo instead as new REST end point. cc:/ [~sunilg] > Add new RM web service endpoint to get cluster user info > > > Key: YARN-8593 > URL: https://issues.apache.org/jira/browse/YARN-8593 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Akhil PB >Assignee: Akhil PB >Priority: Major > Attachments: YARN-8593.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8155) Improve ATSv2 client logging in RM and NM publisher
[ https://issues.apache.org/jira/browse/YARN-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-8155: Attachment: YARN-8155-branch-2.002.patch > Improve ATSv2 client logging in RM and NM publisher > --- > > Key: YARN-8155 > URL: https://issues.apache.org/jira/browse/YARN-8155 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Rohith Sharma K S >Assignee: Abhishek Modi >Priority: Major > Fix For: 3.2.0, 3.1.1, 3.0.4 > > Attachments: YARN-8155-branch-2.002.patch, > YARN-8155-branch-2.v1.patch, YARN-8155.001.patch, YARN-8155.002.patch, > YARN-8155.003.patch, YARN-8155.004.patch, YARN-8155.005.patch, > YARN-8155.006.patch > > > We see that NM logs are filled with larger stack trace of NotFoundException > if collector is removed from one of the NM and other NMs are still publishing > the entities. > > This Jira is to improve the logging in NM so that we log with informative > message. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8574) Allow dot in attribute values
[ https://issues.apache.org/jira/browse/YARN-8574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt updated YARN-8574: --- Attachment: YARN-8574-YARN-3409.002.patch > Allow dot in attribute values > -- > > Key: YARN-8574 > URL: https://issues.apache.org/jira/browse/YARN-8574 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Minor > Attachments: YARN-8574-YARN-3409.001.patch, > YARN-8574-YARN-3409.002.patch > > > Currently "." is considered as invalid value. Enable the same; -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8574) Allow dot in attribute values
[
https://issues.apache.org/jira/browse/YARN-8574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559544#comment-16559544
]
Bibin A Chundatt edited comment on YARN-8574 at 7/27/18 12:31 PM:
--
Hi [~Naganarasimha]
Branch needs rebase to run CI.
Minor update in patch is required since currently values allows \
Could you help with rebase ??
was (Author: bibinchundatt):
Hi [~Naganarasimha]
Branch needs rebase to run CI.
Minor update in patch is required since currently values allows {{\}}.
Could you help with rebase ??
> Allow dot in attribute values
> --
>
> Key: YARN-8574
> URL: https://issues.apache.org/jira/browse/YARN-8574
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Minor
> Attachments: YARN-8574-YARN-3409.001.patch
>
>
> Currently "." is considered as invalid value. Enable the same;
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
