[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage
[
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15015332#comment-15015332
]
Hadoop QA commented on YARN-4053:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
44s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} feature-YARN-2928 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s
{color} | {color:green} feature-YARN-2928 passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 37s
{color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
54s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 23s
{color} | {color:red} hadoop-yarn-server-timelineservice in feature-YARN-2928
failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s
{color} | {color:green} feature-YARN-2928 passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
51s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 20s
{color} | {color:red} hadoop-yarn-server-timelineservice in the patch failed
with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 46s
{color} | {color:green} hadoop-yarn-server-timelineservice in the patch passed
with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 46s
{color} | {color:green} hadoop-yarn-server-timelineservice in the patch passed
with JDK v1.7.0_85. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 35m 3s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:date2015-11-20 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12773453/YARN-4053-feature-YARN-2928.08.patch
|
| JIRA Issue | YARN-4053 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 82b95b6d1a2e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86
[jira] [Updated] (YARN-4053) Change the way metric values are stored in HBase Storage
[ https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-4053: --- Attachment: YARN-4053-feature-YARN-2928.08.patch > Change the way metric values are stored in HBase Storage > > > Key: YARN-4053 > URL: https://issues.apache.org/jira/browse/YARN-4053 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-2928-1st-milestone > Attachments: YARN-4053-YARN-2928.01.patch, > YARN-4053-YARN-2928.02.patch, YARN-4053-feature-YARN-2928.03.patch, > YARN-4053-feature-YARN-2928.04.patch, YARN-4053-feature-YARN-2928.05.patch, > YARN-4053-feature-YARN-2928.06.patch, YARN-4053-feature-YARN-2928.07.patch, > YARN-4053-feature-YARN-2928.08.patch > > > Currently HBase implementation uses GenericObjectMapper to convert and store > values in backend HBase storage. This converts everything into a string > representation(ASCII/UTF-8 encoded byte array). > While this is fine in most cases, it does not quite serve our use case for > metrics. > So we need to decide how are we going to encode and decode metric values and > store them in HBase. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4376) Memory Timeline Store return incorrect results on fromId paging
Jonathan Eagles created YARN-4376: - Summary: Memory Timeline Store return incorrect results on fromId paging Key: YARN-4376 URL: https://issues.apache.org/jira/browse/YARN-4376 Project: Hadoop YARN Issue Type: Bug Reporter: Jonathan Eagles Assignee: Jonathan Eagles As pointed out correctly by [~jlowe]. https://issues.apache.org/jira/browse/TEZ-2628?focusedCommentId=14715831&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14715831 The MemoryTimelineStore cannot page correctly when using fromId. This is due switching between data structures that apparently have different natural sorting. In addition, the approach of creating a new data structure every time from scratch is costly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3623) We should have a config to indicate the Timeline Service version
[ https://issues.apache.org/jira/browse/YARN-3623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15015061#comment-15015061 ] Li Lu commented on YARN-3623: - Patch LGTM. The findbugs warnings appears to be irrelevant to the fix here. > We should have a config to indicate the Timeline Service version > > > Key: YARN-3623 > URL: https://issues.apache.org/jira/browse/YARN-3623 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Naganarasimha G R > Attachments: YARN-3623-2015-11-19.1.patch > > > So far RM, MR AM, DA AM added/changed new config to enable the feature to > write the timeline data to v2 server. It's good to have a YARN > timeline-service.version config like timeline-service.enable to indicate the > version of the running timeline service with the given YARN cluster. It's > beneficial for users to more smoothly move from v1 to v2, as they don't need > to change the existing config, but switch this config from v1 to v2. And each > framework doesn't need to have their own v1/v2 config. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4334) Ability to avoid ResourceManager recovery if state store is "too old"
[
https://issues.apache.org/jira/browse/YARN-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15015025#comment-15015025
]
Hadoop QA commented on YARN-4334:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 9s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
4s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 39s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 36s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 55s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
49s {color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 34s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in
trunk has 3 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 55s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 12s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 35s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 36s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 36s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 32s
{color} | {color:red} Patch generated 24 new checkstyle issues in
hadoop-yarn-project/hadoop-yarn (total was 601, now 622). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 58s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 39s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
introduced 2 new FindBugs issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 52s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 14s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 31s {color}
| {color:red} hadoop-yarn-api in the patch failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 28s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 68m 51s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}
[jira] [Commented] (YARN-3623) We should have a config to indicate the Timeline Service version
[
https://issues.apache.org/jira/browse/YARN-3623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014948#comment-15014948
]
Hadoop QA commented on YARN-3623:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
44s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 42s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 34s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 8s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 30s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in
trunk has 3 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 18s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 37s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
7s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 20s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 29s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 29s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 34s
{color} | {color:red} Patch generated 1 new checkstyle issues in
hadoop-yarn-project/hadoop-yarn (total was 212, now 212). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 12s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 26s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 49s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 31s
{color} | {color:green} hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 21s {color}
| {color:red} hadoop-yarn-common in the patch failed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 29s
{color} | {color:green} hadoop-yarn-api in the patch passed with JDK v1.7.0_85.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 26s {color}
| {color:red} hadoop-yarn-common in the patch failed with JDK v1.7.0_85.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:
[jira] [Commented] (YARN-3980) Plumb resource-utilization info in node heartbeat through to the scheduler
[ https://issues.apache.org/jira/browse/YARN-3980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014944#comment-15014944 ] Jason Kace commented on YARN-3980: -- The newly added unit tests (TestMiniYarnClusterNodeUtilization) are generating timeout errors. The logs do not have any information/exceptions. I have tested locally and can complete these tests in 17-18 seconds. Other unit mini yarn cluster unit tests also require 15-20 seconds (TestMiniYarnCluster) to complete. Are there any time limits for unit tests? Is it possible to see additional jenkins logs to understand where the tests may have stopped? > Plumb resource-utilization info in node heartbeat through to the scheduler > -- > > Key: YARN-3980 > URL: https://issues.apache.org/jira/browse/YARN-3980 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, scheduler >Affects Versions: 2.7.1 >Reporter: Karthik Kambatla >Assignee: Inigo Goiri > Attachments: YARN-3980-v0.patch, YARN-3980-v1.patch, > YARN-3980-v2.patch, YARN-3980-v3.patch, YARN-3980-v4.patch, > YARN-3980-v5.patch, YARN-3980-v6.patch, YARN-3980-v7.patch > > > YARN-1012 and YARN-3534 collect resource utilization information for all > containers and the node respectively and send it to the RM on node heartbeat. > We should plumb it through to the scheduler so the scheduler can make use of > it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4372) Cannot enable system-metrics-publisher inside MiniYARNCluster
[
https://issues.apache.org/jira/browse/YARN-4372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014894#comment-15014894
]
Hadoop QA commented on YARN-4372:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 6s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 7s
{color} | {color:blue} The patch file was not named according to hadoop's
naming conventions. Please see https://wiki.apache.org/hadoop/HowToContribute
for instructions. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
55s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 55s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 7s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 32s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
44s {color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 16s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in
trunk has 3 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 17s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 41s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 3s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 3s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 18s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 52s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 58m 27s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 5m 35s {color}
| {color:red} hadoop-yarn-server-tests in the patch failed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 7s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.7.0_85. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 59m 35s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.7.0_85. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 5m
[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage
[ https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014822#comment-15014822 ] Sangjin Lee commented on YARN-4053: --- Could you address that new checkstyle violation? Then I think it's good to go. Thanks! > Change the way metric values are stored in HBase Storage > > > Key: YARN-4053 > URL: https://issues.apache.org/jira/browse/YARN-4053 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-2928-1st-milestone > Attachments: YARN-4053-YARN-2928.01.patch, > YARN-4053-YARN-2928.02.patch, YARN-4053-feature-YARN-2928.03.patch, > YARN-4053-feature-YARN-2928.04.patch, YARN-4053-feature-YARN-2928.05.patch, > YARN-4053-feature-YARN-2928.06.patch, YARN-4053-feature-YARN-2928.07.patch > > > Currently HBase implementation uses GenericObjectMapper to convert and store > values in backend HBase storage. This converts everything into a string > representation(ASCII/UTF-8 encoded byte array). > While this is fine in most cases, it does not quite serve our use case for > metrics. > So we need to decide how are we going to encode and decode metric values and > store them in HBase. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage
[
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014815#comment-15014815
]
Hadoop QA commented on YARN-4053:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 8s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
27s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} feature-YARN-2928 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s
{color} | {color:green} feature-YARN-2928 passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 28s
{color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 16s
{color} | {color:red} hadoop-yarn-server-timelineservice in feature-YARN-2928
failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} feature-YARN-2928 passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 11s
{color} | {color:red} Patch generated 1 new checkstyle issues in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice
(total was 42, now 38). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 15s
{color} | {color:red} hadoop-yarn-server-timelineservice in the patch failed
with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 5s
{color} | {color:green} hadoop-yarn-server-timelineservice in the patch passed
with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 55s
{color} | {color:green} hadoop-yarn-server-timelineservice in the patch passed
with JDK v1.7.0_85. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 17s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:date2015-11-19 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12773366/YARN-4053-feature-YARN-2928.07.patch
|
| JIRA Issue | YARN-4053 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit
[jira] [Commented] (YARN-3623) We should have a config to indicate the Timeline Service version
[ https://issues.apache.org/jira/browse/YARN-3623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014756#comment-15014756 ] Xuan Gong commented on YARN-3623: - Attached a simple patch which added a new config to indicate the Timeline Service version. > We should have a config to indicate the Timeline Service version > > > Key: YARN-3623 > URL: https://issues.apache.org/jira/browse/YARN-3623 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Naganarasimha G R > Attachments: YARN-3623-2015-11-19.1.patch > > > So far RM, MR AM, DA AM added/changed new config to enable the feature to > write the timeline data to v2 server. It's good to have a YARN > timeline-service.version config like timeline-service.enable to indicate the > version of the running timeline service with the given YARN cluster. It's > beneficial for users to more smoothly move from v1 to v2, as they don't need > to change the existing config, but switch this config from v1 to v2. And each > framework doesn't need to have their own v1/v2 config. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3623) We should have a config to indicate the Timeline Service version
[ https://issues.apache.org/jira/browse/YARN-3623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-3623: Attachment: YARN-3623-2015-11-19.1.patch > We should have a config to indicate the Timeline Service version > > > Key: YARN-3623 > URL: https://issues.apache.org/jira/browse/YARN-3623 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Naganarasimha G R > Attachments: YARN-3623-2015-11-19.1.patch > > > So far RM, MR AM, DA AM added/changed new config to enable the feature to > write the timeline data to v2 server. It's good to have a YARN > timeline-service.version config like timeline-service.enable to indicate the > version of the running timeline service with the given YARN cluster. It's > beneficial for users to more smoothly move from v1 to v2, as they don't need > to change the existing config, but switch this config from v1 to v2. And each > framework doesn't need to have their own v1/v2 config. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4234) New put APIs in TimelineClient for ats v1.5
[ https://issues.apache.org/jira/browse/YARN-4234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014731#comment-15014731 ] Vinod Kumar Vavilapalli commented on YARN-4234: --- As I [commented on YARN-3623|https://issues.apache.org/jira/browse/YARN-3623?focusedCommentId=15014637&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15014637], we should split this patch and put the common code at YARN-3623. Once we do it, this JIRA depends on YARN-3623, linking the tickets to reflect that. > New put APIs in TimelineClient for ats v1.5 > --- > > Key: YARN-4234 > URL: https://issues.apache.org/jira/browse/YARN-4234 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-4234-2015-11-13.1.patch, > YARN-4234-2015-11-16.1.patch, YARN-4234-2015-11-16.2.patch, > YARN-4234-2015.2.patch, YARN-4234.1.patch, YARN-4234.2.patch, > YARN-4234.2015-11-12.1.patch, YARN-4234.2015-11-12.1.patch, > YARN-4234.2015-11-18.1.patch, YARN-4234.2015-11-18.2.patch, > YARN-4234.20151109.patch, YARN-4234.20151110.1.patch, > YARN-4234.2015.1.patch, YARN-4234.3.patch > > > In this ticket, we will add new put APIs in timelineClient to let > clients/applications have the option to use ATS v1.5 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4183) Enabling generic application history forces every job to get a timeline service delegation token
[ https://issues.apache.org/jira/browse/YARN-4183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014726#comment-15014726 ] Vinod Kumar Vavilapalli commented on YARN-4183: --- In the interest of focus, we should move all the version related discussion to YARN-3623. I'll make this JIRA depend on YARN-3623. > Enabling generic application history forces every job to get a timeline > service delegation token > > > Key: YARN-4183 > URL: https://issues.apache.org/jira/browse/YARN-4183 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-4183.1.patch > > > When enabling just the Generic History Server and not the timeline server, > the system metrics publisher will not publish the events to the timeline > store as it checks if the timeline server and system metrics publisher are > enabled before creating a timeline client. > To make it work, if the timeline service flag is turned on, it will force > every yarn application to get a delegation token. > Instead of checking if timeline service is enabled, we should be checking if > application history server is enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4183) Enabling generic application history forces every job to get a timeline service delegation token
[ https://issues.apache.org/jira/browse/YARN-4183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014685#comment-15014685 ] Sangjin Lee commented on YARN-4183: --- Thanks [~vinodkv] for the suggestion. The following is my proposal. - yarn.timeline-service.enabled should be interpreted as the config that indicates the timeline service daemon is up *both* on the client side and the server side - specifically for the RM's system metrics publisher, it should continue to check both yarn.timeline-service.enabled and yarn.resourcemanager.system-metrics-publisher.enabled (which is the current code); if not, and the timeline service is not up (which is a totally valid situation), the system metrics publisher will have a continuous stream of write errors - there needs to be a strong client-side config for each framework (MR or tez) that controls whether it wants to use the timeline service; the client can use the timeline service only if *both* yarn.timeline-service.enabled is true and its own config is true I hope the last point should address the original issue of this JIRA (if MR does not want to use the timeline service, it should be able to do so). The decision to use the timeline service and get the delegation token should not hinge on which version is enabled IMO, as the version is another global property. Let me know if that sounds reasonable. > Enabling generic application history forces every job to get a timeline > service delegation token > > > Key: YARN-4183 > URL: https://issues.apache.org/jira/browse/YARN-4183 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-4183.1.patch > > > When enabling just the Generic History Server and not the timeline server, > the system metrics publisher will not publish the events to the timeline > store as it checks if the timeline server and system metrics publisher are > enabled before creating a timeline client. > To make it work, if the timeline service flag is turned on, it will force > every yarn application to get a delegation token. > Instead of checking if timeline service is enabled, we should be checking if > application history server is enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4053) Change the way metric values are stored in HBase Storage
[ https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-4053: --- Attachment: YARN-4053-feature-YARN-2928.07.patch > Change the way metric values are stored in HBase Storage > > > Key: YARN-4053 > URL: https://issues.apache.org/jira/browse/YARN-4053 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-2928-1st-milestone > Attachments: YARN-4053-YARN-2928.01.patch, > YARN-4053-YARN-2928.02.patch, YARN-4053-feature-YARN-2928.03.patch, > YARN-4053-feature-YARN-2928.04.patch, YARN-4053-feature-YARN-2928.05.patch, > YARN-4053-feature-YARN-2928.06.patch, YARN-4053-feature-YARN-2928.07.patch > > > Currently HBase implementation uses GenericObjectMapper to convert and store > values in backend HBase storage. This converts everything into a string > representation(ASCII/UTF-8 encoded byte array). > While this is fine in most cases, it does not quite serve our use case for > metrics. > So we need to decide how are we going to encode and decode metric values and > store them in HBase. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4368) Support Multiple versions of the timeline service at the same time
[ https://issues.apache.org/jira/browse/YARN-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014680#comment-15014680 ] Vinod Kumar Vavilapalli commented on YARN-4368: --- This depends on YARN-3623, linking tickets. > Support Multiple versions of the timeline service at the same time > -- > > Key: YARN-4368 > URL: https://issues.apache.org/jira/browse/YARN-4368 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Naganarasimha G R > > During rolling updgrade it will be helpfull to have the older version of the > timeline server to be also running so that the existing apps can submit to > the older version of ATS . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage
[
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014673#comment-15014673
]
Varun Saxena commented on YARN-4053:
This patch is similar to version 5 of the patch. No longer using generics.
Removed ValueConverterImpl.
And changed add method to {{Number add(Number num1, Number num2,
Number...numbers)}}
> Change the way metric values are stored in HBase Storage
>
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Labels: yarn-2928-1st-milestone
> Attachments: YARN-4053-YARN-2928.01.patch,
> YARN-4053-YARN-2928.02.patch, YARN-4053-feature-YARN-2928.03.patch,
> YARN-4053-feature-YARN-2928.04.patch, YARN-4053-feature-YARN-2928.05.patch,
> YARN-4053-feature-YARN-2928.06.patch, YARN-4053-feature-YARN-2928.07.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store
> values in backend HBase storage. This converts everything into a string
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for
> metrics.
> So we need to decide how are we going to encode and decode metric values and
> store them in HBase.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3623) We should have a config to indicate the Timeline Service version
[ https://issues.apache.org/jira/browse/YARN-3623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-3623: -- Issue Type: Bug (was: Sub-task) Parent: (was: YARN-2928) > We should have a config to indicate the Timeline Service version > > > Key: YARN-3623 > URL: https://issues.apache.org/jira/browse/YARN-3623 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Naganarasimha G R > > So far RM, MR AM, DA AM added/changed new config to enable the feature to > write the timeline data to v2 server. It's good to have a YARN > timeline-service.version config like timeline-service.enable to indicate the > version of the running timeline service with the given YARN cluster. It's > beneficial for users to more smoothly move from v1 to v2, as they don't need > to change the existing config, but switch this config from v1 to v2. And each > framework doesn't need to have their own v1/v2 config. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4225) Add preemption status to yarn queue -status for capacity scheduler
[ https://issues.apache.org/jira/browse/YARN-4225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014666#comment-15014666 ] Eric Payne commented on YARN-4225: -- All of the tests listed in the failure section above work for me in my local build environment. [~jlowe], would you have time to review this patch? > Add preemption status to yarn queue -status for capacity scheduler > -- > > Key: YARN-4225 > URL: https://issues.apache.org/jira/browse/YARN-4225 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, yarn >Affects Versions: 2.7.1 >Reporter: Eric Payne >Assignee: Eric Payne >Priority: Minor > Attachments: YARN-4225.001.patch, YARN-4225.002.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3623) We should have a config to indicate the Timeline Service version
[ https://issues.apache.org/jira/browse/YARN-3623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014661#comment-15014661 ] Vinod Kumar Vavilapalli commented on YARN-3623: --- Copying comments from YARN-4183 w.r.t versioning: By Me - There should be an explicit yarn.timeline-service.version which tells YarnClient to get tokens or not - yes for non-present version (default), v1, v2 but no for v1.5. - We should also use the same property in the new API calls proposed for V1.5 YARN-4233 / V2 YARN-2928, lest the users think they can call any API independent of what is supported on server side. - The version field has semantics on both client and server side at the same time - it's picking a solution end-to-end. By [~sjlee0] - The version is a list or enum - https://issues.apache.org/jira/browse/YARN-4183?focusedCommentId=15004954&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15004954. Related JIRA: YARN-4368. > We should have a config to indicate the Timeline Service version > > > Key: YARN-3623 > URL: https://issues.apache.org/jira/browse/YARN-3623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Naganarasimha G R > > So far RM, MR AM, DA AM added/changed new config to enable the feature to > write the timeline data to v2 server. It's good to have a YARN > timeline-service.version config like timeline-service.enable to indicate the > version of the running timeline service with the given YARN cluster. It's > beneficial for users to more smoothly move from v1 to v2, as they don't need > to change the existing config, but switch this config from v1 to v2. And each > framework doesn't need to have their own v1/v2 config. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3623) We should have a config to indicate the Timeline Service version
[ https://issues.apache.org/jira/browse/YARN-3623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-3623: -- Summary: We should have a config to indicate the Timeline Service version (was: Having the config to indicate the timeline service version) > We should have a config to indicate the Timeline Service version > > > Key: YARN-3623 > URL: https://issues.apache.org/jira/browse/YARN-3623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Naganarasimha G R > > So far RM, MR AM, DA AM added/changed new config to enable the feature to > write the timeline data to v2 server. It's good to have a YARN > timeline-service.version config like timeline-service.enable to indicate the > version of the running timeline service with the given YARN cluster. It's > beneficial for users to more smoothly move from v1 to v2, as they don't need > to change the existing config, but switch this config from v1 to v2. And each > framework doesn't need to have their own v1/v2 config. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3623) Having the config to indicate the timeline service version
[ https://issues.apache.org/jira/browse/YARN-3623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014637#comment-15014637 ] Vinod Kumar Vavilapalli commented on YARN-3623: --- [~xgong] / [~Naganarasimha] / [~sjlee0] / [~gtCarrera9], there are too many JIRAs about this versioning topic and the discussion is split all over. Let me try to consolidate the tickets. Let's use this one as the base JIRA for introducing versioning. [~xgong], can you post part of your patch at YARN-4234 here? I'll move this to be a common JIRA across v1, v2 and v1.5 and make all the other JIRAs depend on this. > Having the config to indicate the timeline service version > -- > > Key: YARN-3623 > URL: https://issues.apache.org/jira/browse/YARN-3623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Naganarasimha G R > > So far RM, MR AM, DA AM added/changed new config to enable the feature to > write the timeline data to v2 server. It's good to have a YARN > timeline-service.version config like timeline-service.enable to indicate the > version of the running timeline service with the given YARN cluster. It's > beneficial for users to more smoothly move from v1 to v2, as they don't need > to change the existing config, but switch this config from v1 to v2. And each > framework doesn't need to have their own v1/v2 config. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4374) RM scheduler UI rounds user limit factor
[ https://issues.apache.org/jira/browse/YARN-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014573#comment-15014573 ] Chang Li commented on YARN-4374: [~jlowe], please help review the latest patch > RM scheduler UI rounds user limit factor > > > Key: YARN-4374 > URL: https://issues.apache.org/jira/browse/YARN-4374 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Chang Li >Assignee: Chang Li > Attachments: Screenshot1.png, YARN-4374.2.patch, YARN-4374.patch > > > RM scheduler UI rounds user limit factor, such as from 0.25 up to 0.3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4183) Enabling generic application history forces every job to get a timeline service delegation token
[ https://issues.apache.org/jira/browse/YARN-4183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014576#comment-15014576 ] Vinod Kumar Vavilapalli commented on YARN-4183: --- It's a wall of text, but to facilitate progress, [~sjlee0] / [~Naganarasimha], can one of you summarize your (proposal + open-questions) similar to what I did in my comment above [under _The right thing to do_ | https://issues.apache.org/jira/browse/YARN-4183?focusedCommentId=14999670&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14999670]? Tx. > Enabling generic application history forces every job to get a timeline > service delegation token > > > Key: YARN-4183 > URL: https://issues.apache.org/jira/browse/YARN-4183 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-4183.1.patch > > > When enabling just the Generic History Server and not the timeline server, > the system metrics publisher will not publish the events to the timeline > store as it checks if the timeline server and system metrics publisher are > enabled before creating a timeline client. > To make it work, if the timeline service flag is turned on, it will force > every yarn application to get a delegation token. > Instead of checking if timeline service is enabled, we should be checking if > application history server is enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4334) Ability to avoid ResourceManager recovery if state store is "too old"
[ https://issues.apache.org/jira/browse/YARN-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chang Li updated YARN-4334: --- Attachment: YARN-4334.patch added unit test > Ability to avoid ResourceManager recovery if state store is "too old" > - > > Key: YARN-4334 > URL: https://issues.apache.org/jira/browse/YARN-4334 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Jason Lowe >Assignee: Chang Li > Attachments: YARN-4334.patch, YARN-4334.wip.2.patch, > YARN-4334.wip.3.patch, YARN-4334.wip.4.patch, YARN-4334.wip.patch > > > There are times when a ResourceManager has been down long enough that > ApplicationMasters and potentially external client-side monitoring mechanisms > have given up completely. If the ResourceManager starts back up and tries to > recover we can get into situations where the RM launches new application > attempts for the AMs that gave up, but then the client _also_ launches > another instance of the app because it assumed everything was dead. > It would be nice if the RM could be optionally configured to avoid trying to > recover if the state store was "too old." The RM would come up without any > applications recovered, but we would avoid a double-submission situation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2859) ApplicationHistoryServer binds to default port 8188 in MiniYARNCluster
[
https://issues.apache.org/jira/browse/YARN-2859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014565#comment-15014565
]
Hudson commented on YARN-2859:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #620 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/620/])
YARN-2859.addendum: fix the remaining issue from the previous patch (xgong: rev
f114e728da6e19f3d35ff0cfef9fceea26aa5d46)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java
* hadoop-yarn-project/CHANGES.txt
YARN-2859 addendum: move the entry from 2.6.2 to 2.6.3 in (xgong: rev
866dce4e8e105d0e2c7e041b5d30213536a61702)
* hadoop-yarn-project/CHANGES.txt
> ApplicationHistoryServer binds to default port 8188 in MiniYARNCluster
> --
>
> Key: YARN-2859
> URL: https://issues.apache.org/jira/browse/YARN-2859
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Reporter: Hitesh Shah
>Assignee: Vinod Kumar Vavilapalli
>Priority: Critical
> Fix For: 2.8.0, 2.7.2, 2.6.3
>
> Attachments: YARN-2859-addendum.txt, YARN-2859.txt
>
>
> In mini cluster, a random port should be used.
> Also, the config is not updated to the host that the process got bound to.
> {code}
> 2014-11-13 13:07:01,905 INFO [main] server.MiniYARNCluster
> (MiniYARNCluster.java:serviceStart(722)) - MiniYARN ApplicationHistoryServer
> address: localhost:10200
> 2014-11-13 13:07:01,905 INFO [main] server.MiniYARNCluster
> (MiniYARNCluster.java:serviceStart(724)) - MiniYARN ApplicationHistoryServer
> web address: 0.0.0.0:8188
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4340) Add "list" API to reservation system
[ https://issues.apache.org/jira/browse/YARN-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014555#comment-15014555 ] Sean Po commented on YARN-4340: --- For the following, the fields are required JSON serialization: Unread field:ReservationDefinitionInfo.java:[line 36] Unread field:ReservationDefinitionInfo.java:[line 37] Unread field:ReservationDefinitionInfo.java:[line 38] Unread field:ReservationDefinitionInfo.java:[line 45] Unread field:ReservationInfo.java:[line 48] Unread field:ReservationInfo.java:[line 49] Unread public/protected field:At ReservationInfo.java:[line 45] Unread public/protected field:At ReservationRequestInfo.java:[line 35] Unread public/protected field:At ReservationRequestInfo.java:[line 36] Unread public/protected field:At ReservationRequestInfo.java:[line 37] Unread public/protected field:At ReservationRequestInfo.java:[line 38] Unread public/protected field:At ReservationRequestsInfo.java:[line 37] Unread public/protected field:At ReservationRequestsInfo.java:[line 38] The following test might be flaky - I am not able to reproduce this test failure: hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService I do not think the following are tests that are caused by my changes as they also occur in the latest version of trunk. hadoop.yarn.server.resourcemanager.TestAMAuthorization hadoop.yarn.server.resourcemanager.TestClientRMTokens hadoop.mapreduce.v2.TestMRJobsWithProfiler > Add "list" API to reservation system > > > Key: YARN-4340 > URL: https://issues.apache.org/jira/browse/YARN-4340 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler, fairscheduler, resourcemanager >Reporter: Carlo Curino >Assignee: Sean Po > Attachments: YARN-4340.v1.patch, YARN-4340.v2.patch > > > This JIRA tracks changes to the APIs of the reservation system, and enables > querying the reservation system on which reservation exists by "time-range, > reservation-id, username". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage
[ https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014525#comment-15014525 ] Varun Saxena commented on YARN-4053: Ok. > Change the way metric values are stored in HBase Storage > > > Key: YARN-4053 > URL: https://issues.apache.org/jira/browse/YARN-4053 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-2928-1st-milestone > Attachments: YARN-4053-YARN-2928.01.patch, > YARN-4053-YARN-2928.02.patch, YARN-4053-feature-YARN-2928.03.patch, > YARN-4053-feature-YARN-2928.04.patch, YARN-4053-feature-YARN-2928.05.patch, > YARN-4053-feature-YARN-2928.06.patch > > > Currently HBase implementation uses GenericObjectMapper to convert and store > values in backend HBase storage. This converts everything into a string > representation(ASCII/UTF-8 encoded byte array). > While this is fine in most cases, it does not quite serve our use case for > metrics. > So we need to decide how are we going to encode and decode metric values and > store them in HBase. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage
[
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014518#comment-15014518
]
Varun Saxena commented on YARN-4053:
Just to clarify, by mixing I meant I was thinking of a method like {{T add(T
num1, T num2, T...numbers)}} as minimum 2 numbers are required to add. Anyways
if we change the method to original suggestion i.e. {{T
add(Number...numbers)}}, it would work. But then we are not really utilizing
generics here, atleast for add method. Let me know your thoughts on this.
> Change the way metric values are stored in HBase Storage
>
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Labels: yarn-2928-1st-milestone
> Attachments: YARN-4053-YARN-2928.01.patch,
> YARN-4053-YARN-2928.02.patch, YARN-4053-feature-YARN-2928.03.patch,
> YARN-4053-feature-YARN-2928.04.patch, YARN-4053-feature-YARN-2928.05.patch,
> YARN-4053-feature-YARN-2928.06.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store
> values in backend HBase storage. This converts everything into a string
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for
> metrics.
> So we need to decide how are we going to encode and decode metric values and
> store them in HBase.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage
[ https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014509#comment-15014509 ] Sangjin Lee commented on YARN-4053: --- Also, please look at the checkstyle violations. There are new ones that are introduced by this patch. > Change the way metric values are stored in HBase Storage > > > Key: YARN-4053 > URL: https://issues.apache.org/jira/browse/YARN-4053 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-2928-1st-milestone > Attachments: YARN-4053-YARN-2928.01.patch, > YARN-4053-YARN-2928.02.patch, YARN-4053-feature-YARN-2928.03.patch, > YARN-4053-feature-YARN-2928.04.patch, YARN-4053-feature-YARN-2928.05.patch, > YARN-4053-feature-YARN-2928.06.patch > > > Currently HBase implementation uses GenericObjectMapper to convert and store > values in backend HBase storage. This converts everything into a string > representation(ASCII/UTF-8 encoded byte array). > While this is fine in most cases, it does not quite serve our use case for > metrics. > So we need to decide how are we going to encode and decode metric values and > store them in HBase. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2859) ApplicationHistoryServer binds to default port 8188 in MiniYARNCluster
[
https://issues.apache.org/jira/browse/YARN-2859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014504#comment-15014504
]
Hudson commented on YARN-2859:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2558 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2558/])
YARN-2859.addendum: fix the remaining issue from the previous patch (xgong: rev
f114e728da6e19f3d35ff0cfef9fceea26aa5d46)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java
* hadoop-yarn-project/CHANGES.txt
YARN-2859 addendum: move the entry from 2.6.2 to 2.6.3 in (xgong: rev
866dce4e8e105d0e2c7e041b5d30213536a61702)
* hadoop-yarn-project/CHANGES.txt
> ApplicationHistoryServer binds to default port 8188 in MiniYARNCluster
> --
>
> Key: YARN-2859
> URL: https://issues.apache.org/jira/browse/YARN-2859
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Reporter: Hitesh Shah
>Assignee: Vinod Kumar Vavilapalli
>Priority: Critical
> Fix For: 2.8.0, 2.7.2, 2.6.3
>
> Attachments: YARN-2859-addendum.txt, YARN-2859.txt
>
>
> In mini cluster, a random port should be used.
> Also, the config is not updated to the host that the process got bound to.
> {code}
> 2014-11-13 13:07:01,905 INFO [main] server.MiniYARNCluster
> (MiniYARNCluster.java:serviceStart(722)) - MiniYARN ApplicationHistoryServer
> address: localhost:10200
> 2014-11-13 13:07:01,905 INFO [main] server.MiniYARNCluster
> (MiniYARNCluster.java:serviceStart(724)) - MiniYARN ApplicationHistoryServer
> web address: 0.0.0.0:8188
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage
[
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014497#comment-15014497
]
Sangjin Lee commented on YARN-4053:
---
As mentioned above the generic version is good only if {{add()}} and
{{compare()}} would operate only on return values of {{decodeValue()}}. I'm not
sure if that's always a reliable assumption although it happens to be true
right now. [~jrottinghuis]?
Can we go back to non-generic version for the ValueConverter hierarchy? Sorry
for going back and forth on this one!
Just one more minor nit. In {{LongConverter}}, {{decodeValue()}} can simply
return {{Bytes.toLong()}} without using {{Long.valueOf()}}. The same goes for
{{add()}}. Autoboxing will take care of that for you.
> Change the way metric values are stored in HBase Storage
>
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Labels: yarn-2928-1st-milestone
> Attachments: YARN-4053-YARN-2928.01.patch,
> YARN-4053-YARN-2928.02.patch, YARN-4053-feature-YARN-2928.03.patch,
> YARN-4053-feature-YARN-2928.04.patch, YARN-4053-feature-YARN-2928.05.patch,
> YARN-4053-feature-YARN-2928.06.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store
> values in backend HBase storage. This converts everything into a string
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for
> metrics.
> So we need to decide how are we going to encode and decode metric values and
> store them in HBase.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage
[
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014487#comment-15014487
]
Hadoop QA commented on YARN-4053:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 9s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
12s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s
{color} | {color:green} feature-YARN-2928 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s
{color} | {color:green} feature-YARN-2928 passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 38s
{color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 28s
{color} | {color:red} hadoop-yarn-server-timelineservice in feature-YARN-2928
failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} feature-YARN-2928 passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 15s
{color} | {color:red} Patch generated 6 new checkstyle issues in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice
(total was 43, now 45). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
56s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 27s
{color} | {color:red} hadoop-yarn-server-timelineservice in the patch failed
with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 25s
{color} | {color:green} hadoop-yarn-server-timelineservice in the patch passed
with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 56s
{color} | {color:green} hadoop-yarn-server-timelineservice in the patch passed
with JDK v1.7.0_85. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 56s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:date2015-11-19 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12773338/YARN-4053-feature-YARN-2928.06.patch
|
| JIRA Issue | YARN-4053 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit
[jira] [Commented] (YARN-3769) Preemption occurring unnecessarily because preemption doesn't consider user limit
[
https://issues.apache.org/jira/browse/YARN-3769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014488#comment-15014488
]
Eric Payne commented on YARN-3769:
--
Thanks, [~leftnoteasy], but I'm not seeing failures in {{TestLeafQueue}}:
-
-
[hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt|https://builds.apache.org/job/PreCommit-YARN-Build/9726/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt]
{noformat}
Running
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue
Tests run: 26, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.435 sec -
in
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue
{noformat}
-
-
[hadoop-yarn-server-resourcemanager-jdk1.7.0_85.txt|https://builds.apache.org/job/PreCommit-YARN-Build/9726/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_85.txt]
{noformat}
Running
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue
Tests run: 26, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.957 sec -
in
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue
{noformat}
> Preemption occurring unnecessarily because preemption doesn't consider user
> limit
> -
>
> Key: YARN-3769
> URL: https://issues.apache.org/jira/browse/YARN-3769
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Affects Versions: 2.6.0, 2.7.0, 2.8.0
>Reporter: Eric Payne
>Assignee: Eric Payne
> Attachments: YARN-3769-branch-2.002.patch,
> YARN-3769-branch-2.7.002.patch, YARN-3769-branch-2.7.003.patch,
> YARN-3769-branch-2.7.005.patch, YARN-3769-branch-2.7.006.patch,
> YARN-3769-branch-2.7.007.patch, YARN-3769.001.branch-2.7.patch,
> YARN-3769.001.branch-2.8.patch, YARN-3769.003.patch, YARN-3769.004.patch,
> YARN-3769.005.patch
>
>
> We are seeing the preemption monitor preempting containers from queue A and
> then seeing the capacity scheduler giving them immediately back to queue A.
> This happens quite often and causes a lot of churn.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4372) Cannot enable system-metrics-publisher inside MiniYARNCluster
[ https://issues.apache.org/jira/browse/YARN-4372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-4372: -- Attachment: YARN-4372-20151119.1.txt Attaching a patch that should fix this. The patch does the following: - Reordered the service creation in MiniYARNCluster to be AHS -> RM -> NMs. - Moved URI creation in TimelineClient to service-start so that RM can safely start sending events. This should be fine for existing users of TimelineClient also as they cannot do anything for real before client.start(). - Added a simple test in TestMiniYARNCluster to validate the service order. [~Naganarasimha] Responding to your comments on this JIRA as well as YARN-2859. bq. the timeline service was having some issue starting the web service though the port was correctly set (was not an expert with jersey and guice, hence had stopped further analysis there) Can you try my uploaded patch here together with YARN-4350 and see if you still find issues? bq. In MiniYARNCluster RM servicewrapper is first added and then AHSwrapper, and also actual AHS service is started in a thread, so RM's will be using the wrong timelineclient address(port is zero) as AHS service is not yet initialized. This should be fixed in the patch - I changed the order to be AHS -> RM -> NM. The separate thread is not an issue as RM will only start after AHS successfully starts. bq. In Timeline client Impl's serviceInit URI for timeline REST service is set. So even though we create the correct service order (as per previous step), RM's SMP will fail to publish, as timelineweb address is got only after the AHS service is started. Fixed TimelineClient to delay the URI creation. bq. Also one more point(not related to this jira) to note in MiniYARNCluster, we no more support old AHS interfce so basically yarn.timeline-service.generic-application-history.store-class should not be configured in ApplicationHistoryServerWrapper so that levelDBtimelinestore is created. which i feel is correct atleast for existing 2.7.x versions. And if you agree with it, can we fix it along with this jira as its a small thing ?, Even if that store is specified, TimelineDataManager will continue to be instantiated in the server, so your assumption is wrong that levelDB is not created. Agree? > Cannot enable system-metrics-publisher inside MiniYARNCluster > - > > Key: YARN-4372 > URL: https://issues.apache.org/jira/browse/YARN-4372 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Attachments: YARN-4372-20151119.1.txt > > > [~Naganarasimha] found this at YARN-2859, see [this > comment|https://issues.apache.org/jira/browse/YARN-2859?focusedCommentId=15005746&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15005746]. > The way daemons are started inside MiniYARNCluster, RM is not setup correctly > to send information to TimelineService. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage
[ https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014432#comment-15014432 ] Varun Saxena commented on YARN-4053: This patch removes ValueConverterImpl and uses generics as suggested by [~sjlee0]. Haven't added varargs in add method as varargs don't go well with generics and can lead to class cast exceptions. Maybe concrete class types can be used as varargs but that would mean mixing generic params with concrete type param(Number...) in the same method. So haven't added them > Change the way metric values are stored in HBase Storage > > > Key: YARN-4053 > URL: https://issues.apache.org/jira/browse/YARN-4053 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-2928-1st-milestone > Attachments: YARN-4053-YARN-2928.01.patch, > YARN-4053-YARN-2928.02.patch, YARN-4053-feature-YARN-2928.03.patch, > YARN-4053-feature-YARN-2928.04.patch, YARN-4053-feature-YARN-2928.05.patch, > YARN-4053-feature-YARN-2928.06.patch > > > Currently HBase implementation uses GenericObjectMapper to convert and store > values in backend HBase storage. This converts everything into a string > representation(ASCII/UTF-8 encoded byte array). > While this is fine in most cases, it does not quite serve our use case for > metrics. > So we need to decide how are we going to encode and decode metric values and > store them in HBase. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4053) Change the way metric values are stored in HBase Storage
[ https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-4053: --- Attachment: YARN-4053-feature-YARN-2928.06.patch > Change the way metric values are stored in HBase Storage > > > Key: YARN-4053 > URL: https://issues.apache.org/jira/browse/YARN-4053 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-2928-1st-milestone > Attachments: YARN-4053-YARN-2928.01.patch, > YARN-4053-YARN-2928.02.patch, YARN-4053-feature-YARN-2928.03.patch, > YARN-4053-feature-YARN-2928.04.patch, YARN-4053-feature-YARN-2928.05.patch, > YARN-4053-feature-YARN-2928.06.patch > > > Currently HBase implementation uses GenericObjectMapper to convert and store > values in backend HBase storage. This converts everything into a string > representation(ASCII/UTF-8 encoded byte array). > While this is fine in most cases, it does not quite serve our use case for > metrics. > So we need to decide how are we going to encode and decode metric values and > store them in HBase. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3454) RLESparseResourceAllocation does not handle removal of partial intervals (+ introducing support for efficient "merge" operations)
[
https://issues.apache.org/jira/browse/YARN-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014398#comment-15014398
]
Hadoop QA commented on YARN-3454:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 6s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
11s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 49s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
34s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 62m 16s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 62m 28s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.7.0_85. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 147m 0s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.yarn.server.resourcemanager.TestAMAuthorization |
| | hadoop.yarn.server.resourcemanager.rmapp.TestRMAppTransitions |
| | hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| JDK v1.7.0_85 Failed junit tests |
hadoop.yarn.server.resourcemanager.TestAMAuthorization |
| | hadoop.yarn.server.resourcemanager.TestClientRMTokens |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:date2015-11-19 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12773303/YARN-3454.4.pat
[jira] [Commented] (YARN-3769) Preemption occurring unnecessarily because preemption doesn't consider user limit
[ https://issues.apache.org/jira/browse/YARN-3769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014372#comment-15014372 ] Wangda Tan commented on YARN-3769: -- [~eepayne], thanks for update, could you check test failures of latest patch? It seems TestLeafQueue is still failing. > Preemption occurring unnecessarily because preemption doesn't consider user > limit > - > > Key: YARN-3769 > URL: https://issues.apache.org/jira/browse/YARN-3769 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Affects Versions: 2.6.0, 2.7.0, 2.8.0 >Reporter: Eric Payne >Assignee: Eric Payne > Attachments: YARN-3769-branch-2.002.patch, > YARN-3769-branch-2.7.002.patch, YARN-3769-branch-2.7.003.patch, > YARN-3769-branch-2.7.005.patch, YARN-3769-branch-2.7.006.patch, > YARN-3769-branch-2.7.007.patch, YARN-3769.001.branch-2.7.patch, > YARN-3769.001.branch-2.8.patch, YARN-3769.003.patch, YARN-3769.004.patch, > YARN-3769.005.patch > > > We are seeing the preemption monitor preempting containers from queue A and > then seeing the capacity scheduler giving them immediately back to queue A. > This happens quite often and causes a lot of churn. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2859) ApplicationHistoryServer binds to default port 8188 in MiniYARNCluster
[
https://issues.apache.org/jira/browse/YARN-2859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014314#comment-15014314
]
Hudson commented on YARN-2859:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk #1423 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/1423/])
YARN-2859.addendum: fix the remaining issue from the previous patch (xgong: rev
f114e728da6e19f3d35ff0cfef9fceea26aa5d46)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java
* hadoop-yarn-project/CHANGES.txt
YARN-2859 addendum: move the entry from 2.6.2 to 2.6.3 in (xgong: rev
866dce4e8e105d0e2c7e041b5d30213536a61702)
* hadoop-yarn-project/CHANGES.txt
> ApplicationHistoryServer binds to default port 8188 in MiniYARNCluster
> --
>
> Key: YARN-2859
> URL: https://issues.apache.org/jira/browse/YARN-2859
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Reporter: Hitesh Shah
>Assignee: Vinod Kumar Vavilapalli
>Priority: Critical
> Fix For: 2.8.0, 2.7.2, 2.6.3
>
> Attachments: YARN-2859-addendum.txt, YARN-2859.txt
>
>
> In mini cluster, a random port should be used.
> Also, the config is not updated to the host that the process got bound to.
> {code}
> 2014-11-13 13:07:01,905 INFO [main] server.MiniYARNCluster
> (MiniYARNCluster.java:serviceStart(722)) - MiniYARN ApplicationHistoryServer
> address: localhost:10200
> 2014-11-13 13:07:01,905 INFO [main] server.MiniYARNCluster
> (MiniYARNCluster.java:serviceStart(724)) - MiniYARN ApplicationHistoryServer
> web address: 0.0.0.0:8188
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4368) Support Multiple versions of the timeline service at the same time
[ https://issues.apache.org/jira/browse/YARN-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014282#comment-15014282 ] Vrushali C commented on YARN-4368: -- This is a good discussion, thanks [~Naganarasimha] for the jira.. Another view for a +1 on this: When there are lightweight users who don't really need an hbase-scale timeline data, it would be great to have both options available. As yarn adoption is growing, this would be a very useful client side feature to have users up and running quickly. But, I would rather not have to support both at runtime, would prefer to set up one versus the other for the entire cluster. That would help in accounting since with a hybrid setup else we can't account for all jobs on the clusters. I would lean towards not having it as a client side setting since adhoc jobs may choose to not emit stats to ATS v2 and that will disrupt the accountability of users on the cluster. > Support Multiple versions of the timeline service at the same time > -- > > Key: YARN-4368 > URL: https://issues.apache.org/jira/browse/YARN-4368 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Naganarasimha G R > > During rolling updgrade it will be helpfull to have the older version of the > timeline server to be also running so that the existing apps can submit to > the older version of ATS . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4368) Support Multiple versions of the timeline service at the same time
[ https://issues.apache.org/jira/browse/YARN-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014243#comment-15014243 ] Sangjin Lee commented on YARN-4368: --- Thanks for opening this JIRA and providing your thoughts [~Naganarasimha]! The benefit is pretty clear because, as you mentioned, with this ability one can test both v.1 and v.2 with a single cluster without restarting it (which would not be practical with non-test clusters) and compare performance, data, etc. On the other hand, there are a lot of details that need to be worked out for us to define what we mean by this. Also, it would be a fair amount of work as the current code assumes a binary choice between v.1 and v.2 in numerous places. For example, what would we allow in terms of client behavior? Assuming both v.1 and v.2 are enabled on the cluster, should clients still have ability to pick which version it would write to (e.g. v.1 only, v.2 only, v.1 + v.2)? Would that choice be defined at the level of the framework (e.g. MR, tez, etc.), or could it be for individual jobs for a given framework? > Support Multiple versions of the timeline service at the same time > -- > > Key: YARN-4368 > URL: https://issues.apache.org/jira/browse/YARN-4368 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Naganarasimha G R > > During rolling updgrade it will be helpfull to have the older version of the > timeline server to be also running so that the existing apps can submit to > the older version of ATS . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4374) RM scheduler UI rounds user limit factor
[
https://issues.apache.org/jira/browse/YARN-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014237#comment-15014237
]
Hadoop QA commented on YARN-4374:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 8s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 44s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 68m 18s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 48s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.7.0_85. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 157m 48s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
| JDK v1.7.0_85 Failed junit tests |
hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:date2015-11-19 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/127732
[jira] [Commented] (YARN-2859) ApplicationHistoryServer binds to default port 8188 in MiniYARNCluster
[
https://issues.apache.org/jira/browse/YARN-2859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014220#comment-15014220
]
Hudson commented on YARN-2859:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #685 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/685/])
YARN-2859 addendum: move the entry from 2.6.2 to 2.6.3 in (xgong: rev
866dce4e8e105d0e2c7e041b5d30213536a61702)
* hadoop-yarn-project/CHANGES.txt
> ApplicationHistoryServer binds to default port 8188 in MiniYARNCluster
> --
>
> Key: YARN-2859
> URL: https://issues.apache.org/jira/browse/YARN-2859
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Reporter: Hitesh Shah
>Assignee: Vinod Kumar Vavilapalli
>Priority: Critical
> Fix For: 2.8.0, 2.7.2, 2.6.3
>
> Attachments: YARN-2859-addendum.txt, YARN-2859.txt
>
>
> In mini cluster, a random port should be used.
> Also, the config is not updated to the host that the process got bound to.
> {code}
> 2014-11-13 13:07:01,905 INFO [main] server.MiniYARNCluster
> (MiniYARNCluster.java:serviceStart(722)) - MiniYARN ApplicationHistoryServer
> address: localhost:10200
> 2014-11-13 13:07:01,905 INFO [main] server.MiniYARNCluster
> (MiniYARNCluster.java:serviceStart(724)) - MiniYARN ApplicationHistoryServer
> web address: 0.0.0.0:8188
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2859) ApplicationHistoryServer binds to default port 8188 in MiniYARNCluster
[
https://issues.apache.org/jira/browse/YARN-2859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014221#comment-15014221
]
Hudson commented on YARN-2859:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2626 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2626/])
YARN-2859.addendum: fix the remaining issue from the previous patch (xgong: rev
f114e728da6e19f3d35ff0cfef9fceea26aa5d46)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java
* hadoop-yarn-project/CHANGES.txt
YARN-2859 addendum: move the entry from 2.6.2 to 2.6.3 in (xgong: rev
866dce4e8e105d0e2c7e041b5d30213536a61702)
* hadoop-yarn-project/CHANGES.txt
> ApplicationHistoryServer binds to default port 8188 in MiniYARNCluster
> --
>
> Key: YARN-2859
> URL: https://issues.apache.org/jira/browse/YARN-2859
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Reporter: Hitesh Shah
>Assignee: Vinod Kumar Vavilapalli
>Priority: Critical
> Fix For: 2.8.0, 2.7.2, 2.6.3
>
> Attachments: YARN-2859-addendum.txt, YARN-2859.txt
>
>
> In mini cluster, a random port should be used.
> Also, the config is not updated to the host that the process got bound to.
> {code}
> 2014-11-13 13:07:01,905 INFO [main] server.MiniYARNCluster
> (MiniYARNCluster.java:serviceStart(722)) - MiniYARN ApplicationHistoryServer
> address: localhost:10200
> 2014-11-13 13:07:01,905 INFO [main] server.MiniYARNCluster
> (MiniYARNCluster.java:serviceStart(724)) - MiniYARN ApplicationHistoryServer
> web address: 0.0.0.0:8188
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2859) ApplicationHistoryServer binds to default port 8188 in MiniYARNCluster
[
https://issues.apache.org/jira/browse/YARN-2859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014139#comment-15014139
]
Hudson commented on YARN-2859:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #697 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/697/])
YARN-2859.addendum: fix the remaining issue from the previous patch (xgong: rev
f114e728da6e19f3d35ff0cfef9fceea26aa5d46)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java
YARN-2859 addendum: move the entry from 2.6.2 to 2.6.3 in (xgong: rev
866dce4e8e105d0e2c7e041b5d30213536a61702)
* hadoop-yarn-project/CHANGES.txt
> ApplicationHistoryServer binds to default port 8188 in MiniYARNCluster
> --
>
> Key: YARN-2859
> URL: https://issues.apache.org/jira/browse/YARN-2859
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Reporter: Hitesh Shah
>Assignee: Vinod Kumar Vavilapalli
>Priority: Critical
> Fix For: 2.8.0, 2.7.2, 2.6.3
>
> Attachments: YARN-2859-addendum.txt, YARN-2859.txt
>
>
> In mini cluster, a random port should be used.
> Also, the config is not updated to the host that the process got bound to.
> {code}
> 2014-11-13 13:07:01,905 INFO [main] server.MiniYARNCluster
> (MiniYARNCluster.java:serviceStart(722)) - MiniYARN ApplicationHistoryServer
> address: localhost:10200
> 2014-11-13 13:07:01,905 INFO [main] server.MiniYARNCluster
> (MiniYARNCluster.java:serviceStart(724)) - MiniYARN ApplicationHistoryServer
> web address: 0.0.0.0:8188
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3454) RLESparseResourceAllocation does not handle removal of partial intervals (+ introducing support for efficient "merge" operations)
[ https://issues.apache.org/jira/browse/YARN-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014118#comment-15014118 ] Carlo Curino commented on YARN-3454: updated patch fixes checkstyle/whitespace/javadoc... unit test failures seem not related... > RLESparseResourceAllocation does not handle removal of partial intervals (+ > introducing support for efficient "merge" operations) > -- > > Key: YARN-3454 > URL: https://issues.apache.org/jira/browse/YARN-3454 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.8.0, 2.7.1, 2.6.2 >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: YARN-3454.1.patch, YARN-3454.2.patch, YARN-3454.3.patch, > YARN-3454.4.patch, YARN-3454.patch > > > The RLESparseResourceAllocation.removeInterval(...) method handles well exact > match interval removals, but does not handles correctly partial overlaps. > In the context of this fix, we also introduced static methods to "merge" two > RLESparseResourceAllocation, while applying an operator in the process > (add/subtract/min/max/subtractTestPositive) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3454) RLESparseResourceAllocation does not handle removal of partial intervals (+ introducing support for efficient "merge" operations)
[ https://issues.apache.org/jira/browse/YARN-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlo Curino updated YARN-3454: --- Attachment: YARN-3454.4.patch > RLESparseResourceAllocation does not handle removal of partial intervals (+ > introducing support for efficient "merge" operations) > -- > > Key: YARN-3454 > URL: https://issues.apache.org/jira/browse/YARN-3454 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.8.0, 2.7.1, 2.6.2 >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: YARN-3454.1.patch, YARN-3454.2.patch, YARN-3454.3.patch, > YARN-3454.4.patch, YARN-3454.patch > > > The RLESparseResourceAllocation.removeInterval(...) method handles well exact > match interval removals, but does not handles correctly partial overlaps. > In the context of this fix, we also introduced static methods to "merge" two > RLESparseResourceAllocation, while applying an operator in the process > (add/subtract/min/max/subtractTestPositive) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2859) ApplicationHistoryServer binds to default port 8188 in MiniYARNCluster
[
https://issues.apache.org/jira/browse/YARN-2859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014112#comment-15014112
]
Hudson commented on YARN-2859:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #684 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/684/])
YARN-2859.addendum: fix the remaining issue from the previous patch (xgong: rev
f114e728da6e19f3d35ff0cfef9fceea26aa5d46)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java
> ApplicationHistoryServer binds to default port 8188 in MiniYARNCluster
> --
>
> Key: YARN-2859
> URL: https://issues.apache.org/jira/browse/YARN-2859
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Reporter: Hitesh Shah
>Assignee: Vinod Kumar Vavilapalli
>Priority: Critical
> Fix For: 2.8.0, 2.7.2, 2.6.3
>
> Attachments: YARN-2859-addendum.txt, YARN-2859.txt
>
>
> In mini cluster, a random port should be used.
> Also, the config is not updated to the host that the process got bound to.
> {code}
> 2014-11-13 13:07:01,905 INFO [main] server.MiniYARNCluster
> (MiniYARNCluster.java:serviceStart(722)) - MiniYARN ApplicationHistoryServer
> address: localhost:10200
> 2014-11-13 13:07:01,905 INFO [main] server.MiniYARNCluster
> (MiniYARNCluster.java:serviceStart(724)) - MiniYARN ApplicationHistoryServer
> web address: 0.0.0.0:8188
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2859) ApplicationHistoryServer binds to default port 8188 in MiniYARNCluster
[
https://issues.apache.org/jira/browse/YARN-2859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014106#comment-15014106
]
Xuan Gong commented on YARN-2859:
-
Committed into trunk/branch-2/branch-2.7/branch-2.6. Thanks, vinod for the
patch. Thanks, [~Naganarasimha] for the review.
> ApplicationHistoryServer binds to default port 8188 in MiniYARNCluster
> --
>
> Key: YARN-2859
> URL: https://issues.apache.org/jira/browse/YARN-2859
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Reporter: Hitesh Shah
>Assignee: Vinod Kumar Vavilapalli
>Priority: Critical
> Fix For: 2.8.0, 2.7.2, 2.6.3
>
> Attachments: YARN-2859-addendum.txt, YARN-2859.txt
>
>
> In mini cluster, a random port should be used.
> Also, the config is not updated to the host that the process got bound to.
> {code}
> 2014-11-13 13:07:01,905 INFO [main] server.MiniYARNCluster
> (MiniYARNCluster.java:serviceStart(722)) - MiniYARN ApplicationHistoryServer
> address: localhost:10200
> 2014-11-13 13:07:01,905 INFO [main] server.MiniYARNCluster
> (MiniYARNCluster.java:serviceStart(724)) - MiniYARN ApplicationHistoryServer
> web address: 0.0.0.0:8188
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4248) REST API for submit/update/delete Reservations
[ https://issues.apache.org/jira/browse/YARN-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014092#comment-15014092 ] Carlo Curino commented on YARN-4248: Updated patch: * Fixed 2/5 of the asflicense issues (the last 3 are .json files for examples) * javac warnings are on AMLivenessMonitor (untouched in this patch) * fixed as many checkstyle as possible, few are left because I think the code is easier to read as is (e.g., the hidden-fields for setters that do this.a = a; ) * fixed javadoc warnings related to this patch (param misreferenced in comment) * unit-test errors seem unrelated > REST API for submit/update/delete Reservations > -- > > Key: YARN-4248 > URL: https://issues.apache.org/jira/browse/YARN-4248 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: YARN-4248.2.patch, YARN-4248.3.patch, YARN-4248.patch > > > This JIRA tracks work to extend the RMWebService to support REST APIs to > submit/update/delete reservations. This will ease integration with external > tools that are not java-based. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4248) REST API for submit/update/delete Reservations
[ https://issues.apache.org/jira/browse/YARN-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlo Curino updated YARN-4248: --- Attachment: YARN-4248.3.patch > REST API for submit/update/delete Reservations > -- > > Key: YARN-4248 > URL: https://issues.apache.org/jira/browse/YARN-4248 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: YARN-4248.2.patch, YARN-4248.3.patch, YARN-4248.patch > > > This JIRA tracks work to extend the RMWebService to support REST APIs to > submit/update/delete reservations. This will ease integration with external > tools that are not java-based. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2859) ApplicationHistoryServer binds to default port 8188 in MiniYARNCluster
[
https://issues.apache.org/jira/browse/YARN-2859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014086#comment-15014086
]
Hudson commented on YARN-2859:
--
FAILURE: Integrated in Hadoop-trunk-Commit #8828 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8828/])
YARN-2859 addendum: move the entry from 2.6.2 to 2.6.3 in (xgong: rev
866dce4e8e105d0e2c7e041b5d30213536a61702)
* hadoop-yarn-project/CHANGES.txt
> ApplicationHistoryServer binds to default port 8188 in MiniYARNCluster
> --
>
> Key: YARN-2859
> URL: https://issues.apache.org/jira/browse/YARN-2859
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Reporter: Hitesh Shah
>Assignee: Vinod Kumar Vavilapalli
>Priority: Critical
> Fix For: 2.8.0, 2.7.2, 2.6.3
>
> Attachments: YARN-2859-addendum.txt, YARN-2859.txt
>
>
> In mini cluster, a random port should be used.
> Also, the config is not updated to the host that the process got bound to.
> {code}
> 2014-11-13 13:07:01,905 INFO [main] server.MiniYARNCluster
> (MiniYARNCluster.java:serviceStart(722)) - MiniYARN ApplicationHistoryServer
> address: localhost:10200
> 2014-11-13 13:07:01,905 INFO [main] server.MiniYARNCluster
> (MiniYARNCluster.java:serviceStart(724)) - MiniYARN ApplicationHistoryServer
> web address: 0.0.0.0:8188
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2859) ApplicationHistoryServer binds to default port 8188 in MiniYARNCluster
[
https://issues.apache.org/jira/browse/YARN-2859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014068#comment-15014068
]
Hudson commented on YARN-2859:
--
FAILURE: Integrated in Hadoop-trunk-Commit #8827 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8827/])
YARN-2859.addendum: fix the remaining issue from the previous patch (xgong: rev
f114e728da6e19f3d35ff0cfef9fceea26aa5d46)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java
* hadoop-yarn-project/CHANGES.txt
> ApplicationHistoryServer binds to default port 8188 in MiniYARNCluster
> --
>
> Key: YARN-2859
> URL: https://issues.apache.org/jira/browse/YARN-2859
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Reporter: Hitesh Shah
>Assignee: Vinod Kumar Vavilapalli
>Priority: Critical
> Fix For: 2.8.0, 2.7.2, 2.6.3
>
> Attachments: YARN-2859-addendum.txt, YARN-2859.txt
>
>
> In mini cluster, a random port should be used.
> Also, the config is not updated to the host that the process got bound to.
> {code}
> 2014-11-13 13:07:01,905 INFO [main] server.MiniYARNCluster
> (MiniYARNCluster.java:serviceStart(722)) - MiniYARN ApplicationHistoryServer
> address: localhost:10200
> 2014-11-13 13:07:01,905 INFO [main] server.MiniYARNCluster
> (MiniYARNCluster.java:serviceStart(724)) - MiniYARN ApplicationHistoryServer
> web address: 0.0.0.0:8188
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2859) ApplicationHistoryServer binds to default port 8188 in MiniYARNCluster
[
https://issues.apache.org/jira/browse/YARN-2859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014044#comment-15014044
]
Xuan Gong commented on YARN-2859:
-
+1 LGTM. Checking this in
> ApplicationHistoryServer binds to default port 8188 in MiniYARNCluster
> --
>
> Key: YARN-2859
> URL: https://issues.apache.org/jira/browse/YARN-2859
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Reporter: Hitesh Shah
>Assignee: Vinod Kumar Vavilapalli
>Priority: Critical
> Fix For: 2.8.0, 2.7.2, 2.6.3
>
> Attachments: YARN-2859-addendum.txt, YARN-2859.txt
>
>
> In mini cluster, a random port should be used.
> Also, the config is not updated to the host that the process got bound to.
> {code}
> 2014-11-13 13:07:01,905 INFO [main] server.MiniYARNCluster
> (MiniYARNCluster.java:serviceStart(722)) - MiniYARN ApplicationHistoryServer
> address: localhost:10200
> 2014-11-13 13:07:01,905 INFO [main] server.MiniYARNCluster
> (MiniYARNCluster.java:serviceStart(724)) - MiniYARN ApplicationHistoryServer
> web address: 0.0.0.0:8188
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4374) RM scheduler UI rounds user limit factor
[
https://issues.apache.org/jira/browse/YARN-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014001#comment-15014001
]
Hadoop QA commented on YARN-4374:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 6s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
24s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 43s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
21s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 60m 52s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 62m 23s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.7.0_85. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 143m 16s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.yarn.server.resourcemanager.TestAMAuthorization |
| | hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| JDK v1.7.0_85 Failed junit tests |
hadoop.yarn.server.resourcemanager.TestAMAuthorization |
| | hadoop.yarn.server.resourcemanager.TestClientRMTokens |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:date2015-11-19 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/127732
[jira] [Commented] (YARN-4304) AM max resource configuration per partition to be displayed/updated correctly in UI and in various partition related metrics
[ https://issues.apache.org/jira/browse/YARN-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15013976#comment-15013976 ] Sunil G commented on YARN-4304: --- Thanks [~leftnoteasy]. Yes, this looks fine for me. Please let me make changes and write some more testcases to see metrics are coming as desired. I will share the screen shots /REST api with the updated patch shortly. > AM max resource configuration per partition to be displayed/updated correctly > in UI and in various partition related metrics > > > Key: YARN-4304 > URL: https://issues.apache.org/jira/browse/YARN-4304 > Project: Hadoop YARN > Issue Type: Sub-task > Components: webapp >Affects Versions: 2.7.1 >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-YARN-4304.patch > > > As we are supporting per-partition level max AM resource percentage > configuration, UI and various metrics also need to display correct > configurations related to same. > For eg: Current UI still shows am-resource percentage per queue level. This > is to be updated correctly when label config is used. > - Display max-am-percentage per-partition in Scheduler UI (label also) and in > ClusterMetrics page > - Update queue/partition related metrics w.r.t per-partition > am-resource-percentage -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4374) RM scheduler UI rounds user limit factor
[ https://issues.apache.org/jira/browse/YARN-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chang Li updated YARN-4374: --- Attachment: YARN-4374.2.patch Thanks [~jlowe] for review. Agree. update .2 patch with %.3f precision. In addition, .2 patch handles trailing 0s. > RM scheduler UI rounds user limit factor > > > Key: YARN-4374 > URL: https://issues.apache.org/jira/browse/YARN-4374 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Chang Li >Assignee: Chang Li > Attachments: Screenshot1.png, YARN-4374.2.patch, YARN-4374.patch > > > RM scheduler UI rounds user limit factor, such as from 0.25 up to 0.3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4375) CapacityScheduler needs more debug logging for why queues don't get containers
[ https://issues.apache.org/jira/browse/YARN-4375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15013820#comment-15013820 ] Sunil G commented on YARN-4375: --- YARN-4091 tries to address this issue with a REST query result model to see what happened in certain node heartbeats for a queue/app/containers. Going to upload a patch there soon. This one will definitely help in adding more debug informations, also need to see excessive logging wont happen. A little trickier, but will be a very good effort if we bring up the same. > CapacityScheduler needs more debug logging for why queues don't get containers > -- > > Key: YARN-4375 > URL: https://issues.apache.org/jira/browse/YARN-4375 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Chang Li >Assignee: Chang Li > > CapacityScheduler needs more debug logging for why queues don't get containers -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4375) CapacityScheduler needs more debug logging for why queues don't get containers
Chang Li created YARN-4375: -- Summary: CapacityScheduler needs more debug logging for why queues don't get containers Key: YARN-4375 URL: https://issues.apache.org/jira/browse/YARN-4375 Project: Hadoop YARN Issue Type: Bug Reporter: Chang Li Assignee: Chang Li CapacityScheduler needs more debug logging for why queues don't get containers -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4374) RM scheduler UI rounds user limit factor
[
https://issues.apache.org/jira/browse/YARN-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15013742#comment-15013742
]
Jason Lowe commented on YARN-4374:
--
I don't know the full history behind why %.1f was used, but I think it may be
to help hide single-precision floating point precision problems. For example:
{noformat}
$ python -c "print repr(0.1)"
0.10001
{noformat}
So we may still want to have some reasonable limit on the precision we will
attempt to show on the UI to hide the annoying effects like the example above.
It looks like %.1f is hiding too much. Maybe %.3f is more reasonable? We're
already treating other percentages rendered in the UI effectively as %.3f once
we account for the scale by 100.
> RM scheduler UI rounds user limit factor
>
>
> Key: YARN-4374
> URL: https://issues.apache.org/jira/browse/YARN-4374
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Chang Li
>Assignee: Chang Li
> Attachments: Screenshot1.png, YARN-4374.patch
>
>
> RM scheduler UI rounds user limit factor, such as from 0.25 up to 0.3
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4374) RM scheduler UI rounds user limit factor
[ https://issues.apache.org/jira/browse/YARN-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chang Li updated YARN-4374: --- Attachment: Screenshot1.png Attach a screenshot to show the patch fix the problem. [~jlowe] please help review > RM scheduler UI rounds user limit factor > > > Key: YARN-4374 > URL: https://issues.apache.org/jira/browse/YARN-4374 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Chang Li >Assignee: Chang Li > Attachments: Screenshot1.png, YARN-4374.patch > > > RM scheduler UI rounds user limit factor, such as from 0.25 up to 0.3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4374) RM scheduler UI rounds user limit factor
[ https://issues.apache.org/jira/browse/YARN-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chang Li updated YARN-4374: --- Attachment: YARN-4374.patch > RM scheduler UI rounds user limit factor > > > Key: YARN-4374 > URL: https://issues.apache.org/jira/browse/YARN-4374 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Chang Li >Assignee: Chang Li > Attachments: YARN-4374.patch > > > RM scheduler UI rounds user limit factor, such as from 0.25 up to 0.3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4374) RM scheduler UI rounds user limit factor
Chang Li created YARN-4374: -- Summary: RM scheduler UI rounds user limit factor Key: YARN-4374 URL: https://issues.apache.org/jira/browse/YARN-4374 Project: Hadoop YARN Issue Type: Bug Reporter: Chang Li Assignee: Chang Li RM scheduler UI rounds user limit factor, such as from 0.25 up to 0.3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4373) Jobs can be temporarily forgotten during recovery
Daniel Templeton created YARN-4373: -- Summary: Jobs can be temporarily forgotten during recovery Key: YARN-4373 URL: https://issues.apache.org/jira/browse/YARN-4373 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.7.1 Reporter: Daniel Templeton Assignee: Daniel Templeton Priority: Critical The RM becomes available to service requests before state store recovery is started. Before recovery and during the recovery period, it's possible for a client to request an application report for a running application to which the RM will respond that the application in unknown. I'm seeing this issue with Oozie during an RM failover. Until the active finishes recovery, it reports erroneous information to Oozie, which doesn't have context to know that it should just try again later. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3477) TimelineClientImpl swallows exceptions
[
https://issues.apache.org/jira/browse/YARN-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15013503#comment-15013503
]
Steve Loughran commented on YARN-3477:
--
test runner host looks a mess; something is picking up what appears to be ipv6
addresses as hostnames
{code}
Invalid host name: local host is: (unknown); destination host is:
"6b5b453fea76":8033; java.net.UnknownHostException; For more details see:
http://wiki.apache.org/hadoop/UnknownHost
{code}
> TimelineClientImpl swallows exceptions
> --
>
> Key: YARN-3477
> URL: https://issues.apache.org/jira/browse/YARN-3477
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Affects Versions: 2.6.0, 2.7.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: YARN-3477-001.patch, YARN-3477-002.patch
>
>
> If timeline client fails more than the retry count, the original exception is
> not thrown. Instead some runtime exception is raised saying "retries run out"
> # the failing exception should be rethrown, ideally via
> NetUtils.wrapException to include URL of the failing endpoing
> # Otherwise, the raised RTE should (a) state that URL and (b) set the
> original fault as the inner cause
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3477) TimelineClientImpl swallows exceptions
[
https://issues.apache.org/jira/browse/YARN-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15013430#comment-15013430
]
Hadoop QA commented on YARN-3477:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
7s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 20s
{color} | {color:green} branch-2 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 27s
{color} | {color:green} branch-2 passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 5s
{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
35s {color} | {color:green} branch-2 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 33s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in
branch-2 has 3 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s
{color} | {color:green} branch-2 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s
{color} | {color:green} branch-2 passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 18s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 33s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 31s
{color} | {color:red} Patch generated 3 new checkstyle issues in
hadoop-yarn-project/hadoop-yarn (total was 54, now 56). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 49m 39s {color}
| {color:red} hadoop-yarn-client in the patch failed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 9s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 49m 45s {color}
| {color:red} hadoop-yarn-client in the patch failed with JDK v1.7.0_85.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 21s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.7.0_85. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 142m 7s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests | hadoop.yarn.client.TestGetGroups |
| JDK v1.8.0_66 Timed out junit tests |
org.apa
[jira] [Commented] (YARN-3784) Indicate preemption timout along with the list of containers to AM (preemption message)
[ https://issues.apache.org/jira/browse/YARN-3784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15013390#comment-15013390 ] Sunil G commented on YARN-3784: --- Hi [~Naganarasimha Garla] Thank you for bringing in the thoughts, nothing is too late. :) Yes, preemption is now only happening from PCPP. After YARN-3224, preemption will also happen from Decommissioning cases. So timeout b/w forceful killing of container is a varying entity. And in ideal cases, we assume to be greater than AM heartbeat interval (usual cases, it will be). Still we can hit with this corner cases. In existing system, AM skips such requests for preemption as containers might have been killed. This ticket is adding one more information which is timeout, along with list of containers. So its an added information for AM, but can skipped if containers are preempted already by AM. I will see whether I can mark a timeout as 0 or -1 to indicate that those are already passed timeout. Hope this clarify the question. Pls let me know if its ok. > Indicate preemption timout along with the list of containers to AM > (preemption message) > --- > > Key: YARN-3784 > URL: https://issues.apache.org/jira/browse/YARN-3784 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-YARN-3784.patch, 0002-YARN-3784.patch, > 0003-YARN-3784.patch, 0004-YARN-3784.patch > > > Currently during preemption, AM is notified with a list of containers which > are marked for preemption. Introducing a timeout duration also along with > this container list so that AM can know how much time it will get to do a > graceful shutdown to its containers (assuming one of preemption policy is > loaded in AM). > This will help in decommissioning NM scenarios, where NM will be > decommissioned after a timeout (also killing containers on it). This timeout > will be helpful to indicate AM that those containers can be killed by RM > forcefully after the timeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3226) UI changes for decommissioning node
[ https://issues.apache.org/jira/browse/YARN-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15013359#comment-15013359 ] Sunil G commented on YARN-3226: --- Thanks [~djp]. Yes, I also feel that's a good option. We can have "Cluster Metrics on Nodes" as a new table. I will make necessary changes. > UI changes for decommissioning node > --- > > Key: YARN-3226 > URL: https://issues.apache.org/jira/browse/YARN-3226 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Sunil G > Attachments: 0001-YARN-3226.patch, Decommissioning_MetricsPge.png > > > Some initial thought is: > decommissioning nodes should still show up in the active nodes list since > they are still running containers. > A separate decommissioning tab to filter for those nodes would be nice, > although I suppose users can also just use the jquery table to sort/search for > nodes in that state from the active nodes list if it's too crowded to add yet > another node > state tab (or maybe get rid of some effectively dead tabs like the reboot > state tab). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4314) Adding container wait time as a metric at queue level and application level.
[ https://issues.apache.org/jira/browse/YARN-4314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15013344#comment-15013344 ] Lavkesh Lahngir commented on YARN-4314: --- Initial thoughts: An AM sends resource requests with heartbeat and RM tries to fulfil the requests and sends back the response. We can maintain a data structure called ContainerWaitTime in the AppSchedulingInfo to keep track of the last timestamp of the heartbeat and number of pending containers. Resource requests and resource allocations change the containerWaitTime object to increase or decrease pending containers. With every heartbeat, the total wait time for this attempt will be increased by (pending_containers *(current_timestamp - last_timestamp). At this moment last_timestamp will be updated to the current timestamp. Every attempt will maintain this data structure similar to memory-seconds and vcores-seconds. In the AppImpl class, there is a method called getAppMetrics() where we will aggregate the wait time from all the attempts and return it back. For AM container wait time, we need to add an additional parameter called scheduledTime. In getAppMetrics() method, we can get total AM container wait time by summing up (attempt_scheduledTime- attempt_startedTime) for all attempts. If the attempt is not yet scheduled, scheduledTime will be replaced by current time. For adding these new metrics to the queue, we need to just update the queue_metrics object.. it will be aggregated at the queue level. For RM recovery we will need to save these metrics to the state store similar to other metrics of the attempt.(memory-seconds and vcore-seconds) Few more classes to be touched for implementing above, but the core idea remains the same. Most of the code is independent of the scheduler apart from few line addition in the different implementation of the scheduler. I have implemented an initial version. I will put out the patch once I have tested it completely. Feedback? > Adding container wait time as a metric at queue level and application level. > > > Key: YARN-4314 > URL: https://issues.apache.org/jira/browse/YARN-4314 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Lavkesh Lahngir >Assignee: Lavkesh Lahngir > > There is a need for adding the container wait-time which can be tracked at > the queue and application level. > An application can have two kinds of wait times. One is AM wait time after > submission and another is total container wait time between AM asking for > containers and getting them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3784) Indicate preemption timout along with the list of containers to AM (preemption message)
[
https://issues.apache.org/jira/browse/YARN-3784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15013343#comment-15013343
]
Naganarasimha G R commented on YARN-3784:
-
Hi [~sunilg], Had one thought, IIUC default value is 15 seconds but not sure in
real cluster same will be used and what heartbeat interval for apps will be
configured, suppose its very low {{WAIT_TIME_BEFORE_KILL}} and high {{AM
heartbeat interval}}, then by the time AM is informed about the preemption
timeout, may be it will not be able to take the decision effectively. So how
would it be to support long timestamp indicating when it will be preempted,
thoughts ?
Sorry to pitch in at the last moment !
> Indicate preemption timout along with the list of containers to AM
> (preemption message)
> ---
>
> Key: YARN-3784
> URL: https://issues.apache.org/jira/browse/YARN-3784
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-YARN-3784.patch, 0002-YARN-3784.patch,
> 0003-YARN-3784.patch, 0004-YARN-3784.patch
>
>
> Currently during preemption, AM is notified with a list of containers which
> are marked for preemption. Introducing a timeout duration also along with
> this container list so that AM can know how much time it will get to do a
> graceful shutdown to its containers (assuming one of preemption policy is
> loaded in AM).
> This will help in decommissioning NM scenarios, where NM will be
> decommissioned after a timeout (also killing containers on it). This timeout
> will be helpful to indicate AM that those containers can be killed by RM
> forcefully after the timeout.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3226) UI changes for decommissioning node
[ https://issues.apache.org/jira/browse/YARN-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1501#comment-1501 ] Junping Du commented on YARN-3226: -- Thanks [~sunilg] for working and uploading a patch! I remember Jason comments in one JIRA (YARN-914 or some sub jira) to say it could be congest after adding decommissioning nodes, and I have the same feeling now. May be we can have another row for Cluster Metrics or separate metrics on nodes out as a new row, something like: Cluster Metrics on Nodes? > UI changes for decommissioning node > --- > > Key: YARN-3226 > URL: https://issues.apache.org/jira/browse/YARN-3226 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Sunil G > Attachments: 0001-YARN-3226.patch, Decommissioning_MetricsPge.png > > > Some initial thought is: > decommissioning nodes should still show up in the active nodes list since > they are still running containers. > A separate decommissioning tab to filter for those nodes would be nice, > although I suppose users can also just use the jquery table to sort/search for > nodes in that state from the active nodes list if it's too crowded to add yet > another node > state tab (or maybe get rid of some effectively dead tabs like the reboot > state tab). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4350) TestDistributedShell fails
[
https://issues.apache.org/jira/browse/YARN-4350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15013315#comment-15013315
]
Naganarasimha G R commented on YARN-4350:
-
Hi [~sjlee0], i think i had handled all these scenarios in YARN-3127, so what i
can do is rebase that patch as per trunk so that you can take a look at it
there and for other issue i think Vinod is planning to handle in YARN-4372, so
i think we can keep this jira open and cross verify after the next merge with
trunk, thoughts?
> TestDistributedShell fails
> --
>
> Key: YARN-4350
> URL: https://issues.apache.org/jira/browse/YARN-4350
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Naganarasimha G R
> Attachments: YARN-4350-feature-YARN-2928.001.patch
>
>
> Currently TestDistributedShell does not pass on the feature-YARN-2928 branch.
> There seem to be 2 distinct issues.
> (1) testDSShellWithoutDomainV2* tests fail sporadically
> These test fail more often than not if tested by themselves:
> {noformat}
> testDSShellWithoutDomainV2DefaultFlow(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
> Time elapsed: 30.998 sec <<< FAILURE!
> java.lang.AssertionError: Application created event should be published
> atleast once expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.checkTimelineV2(TestDistributedShell.java:451)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShell(TestDistributedShell.java:326)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShellWithoutDomainV2DefaultFlow(TestDistributedShell.java:207)
> {noformat}
> They start happening after YARN-4129. I suspect this might have to do with
> some timing issue.
> (2) the whole test times out
> If you run the whole TestDistributedShell test, it times out without fail.
> This may or may not have to do with the port change introduced by YARN-2859
> (just a hunch).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4372) Cannot enable system-metrics-publisher inside MiniYARNCluster
[
https://issues.apache.org/jira/browse/YARN-4372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15013280#comment-15013280
]
Naganarasimha G R commented on YARN-4372:
-
Thanks for raising this issue [~vinodkv], yes it would be better to correct the
order, as in future if we start collecting ATSv2 system events then we will be
having issues later. Issue i faced when i was testing with correcting the order
was : the timeline service was having some issue starting the web service
though the port was correctly set (was not an expert with jersey and guice,
hence had stopped further analysis there)
Also one more point(not related to this jira) to note in MiniYARNCluster, we no
more support old AHS interfce so basically
{{yarn.timeline-service.generic-application-history.store-class}} should not be
configured in {{ApplicationHistoryServerWrapper}} so that levelDBtimelinestore
is created. which i feel is correct atleast for existing 2.7.x versions. And if
you agree with it, can we fix it along with this jira as its a small thing ?,
> Cannot enable system-metrics-publisher inside MiniYARNCluster
> -
>
> Key: YARN-4372
> URL: https://issues.apache.org/jira/browse/YARN-4372
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Vinod Kumar Vavilapalli
>
> [~Naganarasimha] found this at YARN-2859, see [this
> comment|https://issues.apache.org/jira/browse/YARN-2859?focusedCommentId=15005746&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15005746].
> The way daemons are started inside MiniYARNCluster, RM is not setup correctly
> to send information to TimelineService.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (YARN-3556) Yarn REST to submit application fails with NPE if the element has empty entries
[ https://issues.apache.org/jira/browse/YARN-3556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang resolved YARN-3556. --- Resolution: Not A Problem > Yarn REST to submit application fails with NPE if the element > has empty entries > - > > Key: YARN-3556 > URL: https://issues.apache.org/jira/browse/YARN-3556 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 2.6.0 >Reporter: Rajesh Kartha >Assignee: Weiwei Yang >Priority: Minor > Attachments: wc-no-cp-NPE.xml > > > I was trying out the YARN REST api to submit applications and ran into a NPE > where the element has empty tags > example: > > > > > > > hadoop jar > /usr/hadoop/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar wordcount > /tmp/zkSmoke.out /tmp/REST-WC-4 > > > The Exception: > 2015-04-28 09:34:41,725 WARN resourcemanager.RMAuditLogger > (RMAuditLogger.java:logFailure(263)) - USER=dr.who OPERATION=Application > Finished - Failed > TARGET=RMAppManagerRESULT=FAILURE DESCRIPTION=App failed with state: FAILED > PERMISSIONS=Application application_1430176860750_0024 failed > 2 times due to Error launching appattempt_1430176860750_0024_02. > Got exception: java.lang.NullPointerException > at > org.apache.hadoop.yarn.proto.YarnProtos$StringStringMapProto$Builder.setKey(YarnProtos.java:44696) > at > org.apache.hadoop.yarn.api.records.impl.pb.ContainerLaunchContextPBImpl$3$1.next(ContainerLaunchContextPBImpl.java:376) > at > org.apache.hadoop.yarn.api.records.impl.pb.ContainerLaunchContextPBImpl$3$1.next(ContainerLaunchContextPBImpl.java:364) > at > com.google.protobuf.AbstractMessageLite$Builder.checkForNullValues(AbstractMessageLite.java:336) > at > com.google.protobuf.AbstractMessageLite$Builder.addAll(AbstractMessageLite.java:323) > at > org.apache.hadoop.yarn.proto.YarnProtos$ContainerLaunchContextProto$Builder.addAllEnvironment(YarnProtos.java:39623) > at > org.apache.hadoop.yarn.api.records.impl.pb.ContainerLaunchContextPBImpl.addEnvToProto(ContainerLaunchContextPBImpl.java:387) > at > org.apache.hadoop.yarn.api.records.impl.pb.ContainerLaunchContextPBImpl.mergeLocalToBuilder(ContainerLaunchContextPBImpl.java:115) > at > org.apache.hadoop.yarn.api.records.impl.pb.ContainerLaunchContextPBImpl.mergeLocalToProto(ContainerLaunchContextPBImpl.java:128) > at > org.apache.hadoop.yarn.api.records.impl.pb.ContainerLaunchContextPBImpl.getProto(ContainerLaunchContextPBImpl.java:70) > at > org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainerRequestPBImpl.convertToProtoFormat(StartContainerRequestPBImpl.java:156) > at > org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainerRequestPBImpl.mergeLocalToBuilder(StartContainerRequestPBImpl.java:85) > at > org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainerRequestPBImpl.mergeLocalToProto(StartContainerRequestPBImpl.java:95) > at > org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainerRequestPBImpl.getProto(StartContainerRequestPBImpl.java:57) > at > org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainersRequestPBImpl.convertToProtoFormat(StartContainersRequestPBImpl.java:137) > at > org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainersRequestPBImpl.addLocalRequestsToProto(StartContainersRequestPBImpl.java:97) > at > org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainersRequestPBImpl.mergeLocalToBuilder(StartContainersRequestPBImpl.java:79) > at > org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainersRequestPBImpl.mergeLocalToProto(StartContainersRequestPBImpl.java:72) > at > org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainersRequestPBImpl.getProto(StartContainersRequestPBImpl.java:48) > at > org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:93) > at > org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:119) > at > org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:254) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > . Failing the application. APP
[jira] [Commented] (YARN-3556) Yarn REST to submit application fails with NPE if the element has empty entries
[ https://issues.apache.org/jira/browse/YARN-3556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15013263#comment-15013263 ] Weiwei Yang commented on YARN-3556: --- I just spent some time looking at this problem. This is not just for an empty entry in environment, any tag in xml if it is empty, you will get NPE. Yarn leverages protobuf to serialize data, this logic is everywhere, I don't see a good option to fix though I admit the error message is not quite user friendly. > Yarn REST to submit application fails with NPE if the element > has empty entries > - > > Key: YARN-3556 > URL: https://issues.apache.org/jira/browse/YARN-3556 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 2.6.0 >Reporter: Rajesh Kartha >Assignee: Weiwei Yang >Priority: Minor > Attachments: wc-no-cp-NPE.xml > > > I was trying out the YARN REST api to submit applications and ran into a NPE > where the element has empty tags > example: > > > > > > > hadoop jar > /usr/hadoop/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar wordcount > /tmp/zkSmoke.out /tmp/REST-WC-4 > > > The Exception: > 2015-04-28 09:34:41,725 WARN resourcemanager.RMAuditLogger > (RMAuditLogger.java:logFailure(263)) - USER=dr.who OPERATION=Application > Finished - Failed > TARGET=RMAppManagerRESULT=FAILURE DESCRIPTION=App failed with state: FAILED > PERMISSIONS=Application application_1430176860750_0024 failed > 2 times due to Error launching appattempt_1430176860750_0024_02. > Got exception: java.lang.NullPointerException > at > org.apache.hadoop.yarn.proto.YarnProtos$StringStringMapProto$Builder.setKey(YarnProtos.java:44696) > at > org.apache.hadoop.yarn.api.records.impl.pb.ContainerLaunchContextPBImpl$3$1.next(ContainerLaunchContextPBImpl.java:376) > at > org.apache.hadoop.yarn.api.records.impl.pb.ContainerLaunchContextPBImpl$3$1.next(ContainerLaunchContextPBImpl.java:364) > at > com.google.protobuf.AbstractMessageLite$Builder.checkForNullValues(AbstractMessageLite.java:336) > at > com.google.protobuf.AbstractMessageLite$Builder.addAll(AbstractMessageLite.java:323) > at > org.apache.hadoop.yarn.proto.YarnProtos$ContainerLaunchContextProto$Builder.addAllEnvironment(YarnProtos.java:39623) > at > org.apache.hadoop.yarn.api.records.impl.pb.ContainerLaunchContextPBImpl.addEnvToProto(ContainerLaunchContextPBImpl.java:387) > at > org.apache.hadoop.yarn.api.records.impl.pb.ContainerLaunchContextPBImpl.mergeLocalToBuilder(ContainerLaunchContextPBImpl.java:115) > at > org.apache.hadoop.yarn.api.records.impl.pb.ContainerLaunchContextPBImpl.mergeLocalToProto(ContainerLaunchContextPBImpl.java:128) > at > org.apache.hadoop.yarn.api.records.impl.pb.ContainerLaunchContextPBImpl.getProto(ContainerLaunchContextPBImpl.java:70) > at > org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainerRequestPBImpl.convertToProtoFormat(StartContainerRequestPBImpl.java:156) > at > org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainerRequestPBImpl.mergeLocalToBuilder(StartContainerRequestPBImpl.java:85) > at > org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainerRequestPBImpl.mergeLocalToProto(StartContainerRequestPBImpl.java:95) > at > org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainerRequestPBImpl.getProto(StartContainerRequestPBImpl.java:57) > at > org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainersRequestPBImpl.convertToProtoFormat(StartContainersRequestPBImpl.java:137) > at > org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainersRequestPBImpl.addLocalRequestsToProto(StartContainersRequestPBImpl.java:97) > at > org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainersRequestPBImpl.mergeLocalToBuilder(StartContainersRequestPBImpl.java:79) > at > org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainersRequestPBImpl.mergeLocalToProto(StartContainersRequestPBImpl.java:72) > at > org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainersRequestPBImpl.getProto(StartContainersRequestPBImpl.java:48) > at > org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:93) > at > org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:119) > at > org.apache.hadoop.yarn.server.re
[jira] [Commented] (YARN-4140) RM container allocation delayed incase of app submitted to Nodelabel partition
[
https://issues.apache.org/jira/browse/YARN-4140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15013240#comment-15013240
]
Naganarasimha G R commented on YARN-4140:
-
Yes [~wangda] you are right, if 2.8 is in near future then there is no point in
pushing these issues to 2.7.x release. But all our fixes/improvements makes the
NodeLabel more usable hence was trying to push at least the important ones !
> RM container allocation delayed incase of app submitted to Nodelabel partition
> --
>
> Key: YARN-4140
> URL: https://issues.apache.org/jira/browse/YARN-4140
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Fix For: 2.8.0
>
> Attachments: 0001-YARN-4140.patch, 0002-YARN-4140.patch,
> 0003-YARN-4140.patch, 0004-YARN-4140.patch, 0005-YARN-4140.patch,
> 0006-YARN-4140.patch, 0007-YARN-4140.patch, 0008-YARN-4140.patch,
> 0009-YARN-4140.patch, 0010-YARN-4140.patch, 0011-YARN-4140.patch,
> 0012-YARN-4140.patch, 0013-YARN-4140.patch, 0014-YARN-4140.patch
>
>
> Trying to run application on Nodelabel partition I found that the
> application execution time is delayed by 5 – 10 min for 500 containers .
> Total 3 machines 2 machines were in same partition and app submitted to same.
> After enabling debug was able to find the below
> # From AM the container ask is for OFF-SWITCH
> # RM allocating all containers to NODE_LOCAL as shown in logs below.
> # So since I was having about 500 containers time taken was about – 6 minutes
> to allocate 1st map after AM allocation.
> # Tested with about 1K maps using PI job took 17 minutes to allocate next
> container after AM allocation
> Once 500 container allocation on NODE_LOCAL is done the next container
> allocation is done on OFF_SWITCH
> {code}
> 2015-09-09 15:21:58,954 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt:
> showRequests: application=application_1441791998224_0001 request={Priority:
> 20, Capability: , # Containers: 500, Location:
> /default-rack, Relax Locality: true, Node Label Expression: }
> 2015-09-09 15:21:58,954 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt:
> showRequests: application=application_1441791998224_0001 request={Priority:
> 20, Capability: , # Containers: 500, Location: *, Relax
> Locality: true, Node Label Expression: 3}
> 2015-09-09 15:21:58,954 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt:
> showRequests: application=application_1441791998224_0001 request={Priority:
> 20, Capability: , # Containers: 500, Location:
> host-10-19-92-143, Relax Locality: true, Node Label Expression: }
> 2015-09-09 15:21:58,954 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt:
> showRequests: application=application_1441791998224_0001 request={Priority:
> 20, Capability: , # Containers: 500, Location:
> host-10-19-92-117, Relax Locality: true, Node Label Expression: }
> 2015-09-09 15:21:58,954 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
> Assigned to queue: root.b.b1 stats: b1: capacity=1.0, absoluteCapacity=0.5,
> usedResources=, usedCapacity=0.0,
> absoluteUsedCapacity=0.0, numApps=1, numContainers=1 --> vCores:0>, NODE_LOCAL
> {code}
>
> {code}
> 2015-09-09 14:35:45,467 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
> Assigned to queue: root.b.b1 stats: b1: capacity=1.0, absoluteCapacity=0.5,
> usedResources=, usedCapacity=0.0,
> absoluteUsedCapacity=0.0, numApps=1, numContainers=1 --> vCores:0>, NODE_LOCAL
> 2015-09-09 14:35:45,831 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
> Assigned to queue: root.b.b1 stats: b1: capacity=1.0, absoluteCapacity=0.5,
> usedResources=, usedCapacity=0.0,
> absoluteUsedCapacity=0.0, numApps=1, numContainers=1 --> vCores:0>, NODE_LOCAL
> 2015-09-09 14:35:46,469 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
> Assigned to queue: root.b.b1 stats: b1: capacity=1.0, absoluteCapacity=0.5,
> usedResources=, usedCapacity=0.0,
> absoluteUsedCapacity=0.0, numApps=1, numContainers=1 --> vCores:0>, NODE_LOCAL
> 2015-09-09 14:35:46,832 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
> Assigned to queue: root.b.b1 stats: b1: capacity=1.0, absoluteCapacity=0.5,
> usedResources=, usedCapacity=0.0,
> absoluteUsedCapacity=0.0, numApps=1, numContainers=1 --> vCores:0>, NODE_LOCAL
> {code}
> {code}
> dsperf@host-127:/opt/bibin/dsperf/HAINSTALL/install/hadoop/resourcemanage
[jira] [Commented] (YARN-4287) Capacity Scheduler: Rack Locality improvement
[ https://issues.apache.org/jira/browse/YARN-4287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15013120#comment-15013120 ] Hudson commented on YARN-4287: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2555 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2555/]) move fix version of YARN-4287 from 2.8.0 to 2.7.3 (wangda: rev 23a130abd7f26ca95d7e94988c7bc45c6d419d0f) * hadoop-yarn-project/CHANGES.txt > Capacity Scheduler: Rack Locality improvement > - > > Key: YARN-4287 > URL: https://issues.apache.org/jira/browse/YARN-4287 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 2.7.1 >Reporter: Nathan Roberts >Assignee: Nathan Roberts > Fix For: 2.8.0 > > Attachments: YARN-4287-minimal-v2.patch, YARN-4287-minimal-v3.patch, > YARN-4287-minimal-v4-branch-2.7.patch, YARN-4287-minimal-v4.patch, > YARN-4287-minimal.patch, YARN-4287-v2.patch, YARN-4287-v3.patch, > YARN-4287-v4.patch, YARN-4287.patch > > > YARN-4189 does an excellent job describing the issues with the current delay > scheduling algorithms within the capacity scheduler. The design proposal also > seems like a good direction. > This jira proposes a simple interim solution to the key issue we've been > experiencing on a regular basis: > - rackLocal assignments trickle out due to nodeLocalityDelay. This can have > significant impact on things like CombineFileInputFormat which targets very > specific nodes in its split calculations. > I'm not sure when YARN-4189 will become reality so I thought a simple interim > patch might make sense. The basic idea is simple: > 1) Separate delays for rackLocal, and OffSwitch (today there is only 1) > 2) When we're getting rackLocal assignments, subsequent rackLocal assignments > should not be delayed > Patch will be uploaded shortly. No big deal if the consensus is to go > straight to YARN-4189. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4279) Mark ApplicationId and ApplicationAttemptId static methods as @Public, @Unstable
[
https://issues.apache.org/jira/browse/YARN-4279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15013121#comment-15013121
]
Hudson commented on YARN-4279:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2555 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2555/])
YARN-4279. Mark ApplicationId and ApplicationAttemptId static methods as
(stevel: rev 253e0404f37ee7d889b52cc910e51e5fbde8ac8a)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationAttemptId.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationId.java
> Mark ApplicationId and ApplicationAttemptId static methods as @Public,
> @Unstable
>
>
> Key: YARN-4279
> URL: https://issues.apache.org/jira/browse/YARN-4279
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client
>Affects Versions: 2.7.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: YARN-4279-001.patch
>
> Original Estimate: 0.25h
> Remaining Estimate: 0.25h
>
> The classes {{ApplicationId}} and {{ApplicationAttemptId}} both have
> {{newInstance()}} methods tagged as {{@Private}}. Yet they are useful in
> testing, as the alternative is to create and configure the PBImpl classes
> -which are significantly more private.
> The fact that mapreduce's {{MRBuilderUtils}} uses one of the methods shows
> that YARN apps do need access to the methods.
> Marking them as public would make it clear that other YARN apps were using
> them for their production or test code, rather than today, where they are
> used and depended on, yet without the YARN team's knowledge.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
