[jira] [Commented] (YARN-4356) ensure the timeline service v.2 is disabled cleanly and has no impact when it's turned off
[ https://issues.apache.org/jira/browse/YARN-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15369797#comment-15369797 ] Hudson commented on YARN-4356: -- SUCCESS: Integrated in Hadoop-trunk-Commit #10074 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10074/]) YARN-4356. Ensure the timeline service v.2 is disabled cleanly and has (sjlee: rev 89e5c44f9e891a3579384c3fa3766937cd4970f1) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/test/java/org/apache/hadoop/yarn/server/timelineservice/reader/TestTimelineReaderWebServicesHBaseStorage.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/test/java/org/apache/hadoop/yarn/server/timelineservice/reader/TestTimelineReaderServer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatRequestPBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/test/java/org/apache/hadoop/yarn/server/timelineservice/collector/TestPerNodeTimelineCollectorsAuxService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/amlauncher/AMLauncher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/TestTimelineServiceClientIntegration.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/test/java/org/apache/hadoop/yarn/server/timelineservice/reader/TestTimelineReaderWebServices.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMRTimelineEventHandling.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/PerNodeTimelineCollectorsAuxService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServices.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/Context.java *
[jira] [Commented] (YARN-4356) ensure the timeline service v.2 is disabled cleanly and has no impact when it's turned off
[ https://issues.apache.org/jira/browse/YARN-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15083947#comment-15083947 ] Vrushali C commented on YARN-4356: -- Hmm. I am not sure if it is inconsistent. The actual behavior right now checks for an exact 2. The javadoc is more comprehensive and so it still holds true since it's smaller than 3. Now say when we release v2 and then we are working v2.1, we might want to add in a check for 2.1 which does something different from 2 and at the same time, it's not really timeline v3. So IMO, the check is also fine and the javadoc is also fine. > ensure the timeline service v.2 is disabled cleanly and has no impact when > it's turned off > -- > > Key: YARN-4356 > URL: https://issues.apache.org/jira/browse/YARN-4356 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Labels: yarn-2928-1st-milestone > Fix For: YARN-2928 > > Attachments: YARN-4356-feature-YARN-2928.002.patch, > YARN-4356-feature-YARN-2928.003.patch, YARN-4356-feature-YARN-2928.004.patch, > YARN-4356-feature-YARN-2928.005.patch, YARN-4356-feature-YARN-2928.006.patch, > YARN-4356-feature-YARN-2928.poc.001.patch > > > For us to be able to merge the first milestone drop to trunk, we want to > ensure that once disabled the timeline service v.2 has no impact from the > server side to the client side. If the timeline service is not enabled, no > action should be done. If v.1 is enabled but not v.2, v.1 should behave the > same as it does before the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4356) ensure the timeline service v.2 is disabled cleanly and has no impact when it's turned off
[ https://issues.apache.org/jira/browse/YARN-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15083959#comment-15083959 ] Li Lu commented on YARN-4356: - Ah OK. But we do need to update the actual code when we introduce a new version like 2.1, right? > ensure the timeline service v.2 is disabled cleanly and has no impact when > it's turned off > -- > > Key: YARN-4356 > URL: https://issues.apache.org/jira/browse/YARN-4356 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Labels: yarn-2928-1st-milestone > Fix For: YARN-2928 > > Attachments: YARN-4356-feature-YARN-2928.002.patch, > YARN-4356-feature-YARN-2928.003.patch, YARN-4356-feature-YARN-2928.004.patch, > YARN-4356-feature-YARN-2928.005.patch, YARN-4356-feature-YARN-2928.006.patch, > YARN-4356-feature-YARN-2928.poc.001.patch > > > For us to be able to merge the first milestone drop to trunk, we want to > ensure that once disabled the timeline service v.2 has no impact from the > server side to the client side. If the timeline service is not enabled, no > action should be done. If v.1 is enabled but not v.2, v.1 should behave the > same as it does before the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4356) ensure the timeline service v.2 is disabled cleanly and has no impact when it's turned off
[ https://issues.apache.org/jira/browse/YARN-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15083938#comment-15083938 ] Li Lu commented on YARN-4356: - One quick query w.r.t changes in this JIRA: I just noticed that in YarnConfiguration#timelineServiceV2Enabled, the javadoc describes it as: bq. whether the timeline service v.2 is enabled. V.2 refers to a version greater than equal to 2 but smaller than 3. But the actual behavior is to directly check if the version of timeline server equals to 2. I believe this will generate some inconsistency. Are there any special considerations? Shall we quickly put an addendum patch? > ensure the timeline service v.2 is disabled cleanly and has no impact when > it's turned off > -- > > Key: YARN-4356 > URL: https://issues.apache.org/jira/browse/YARN-4356 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Labels: yarn-2928-1st-milestone > Fix For: YARN-2928 > > Attachments: YARN-4356-feature-YARN-2928.002.patch, > YARN-4356-feature-YARN-2928.003.patch, YARN-4356-feature-YARN-2928.004.patch, > YARN-4356-feature-YARN-2928.005.patch, YARN-4356-feature-YARN-2928.006.patch, > YARN-4356-feature-YARN-2928.poc.001.patch > > > For us to be able to merge the first milestone drop to trunk, we want to > ensure that once disabled the timeline service v.2 has no impact from the > server side to the client side. If the timeline service is not enabled, no > action should be done. If v.1 is enabled but not v.2, v.1 should behave the > same as it does before the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4356) ensure the timeline service v.2 is disabled cleanly and has no impact when it's turned off
[ https://issues.apache.org/jira/browse/YARN-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15083994#comment-15083994 ] Vrushali C commented on YARN-4356: -- Yes, if we want the behavior to be different. If the 2.1 behavior simply builds upon 2, then we can leave it as TimelineV2. Or if you'd like, we can have a simple addendum patch to make it clearer, as you had suggested. > ensure the timeline service v.2 is disabled cleanly and has no impact when > it's turned off > -- > > Key: YARN-4356 > URL: https://issues.apache.org/jira/browse/YARN-4356 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Labels: yarn-2928-1st-milestone > Fix For: YARN-2928 > > Attachments: YARN-4356-feature-YARN-2928.002.patch, > YARN-4356-feature-YARN-2928.003.patch, YARN-4356-feature-YARN-2928.004.patch, > YARN-4356-feature-YARN-2928.005.patch, YARN-4356-feature-YARN-2928.006.patch, > YARN-4356-feature-YARN-2928.poc.001.patch > > > For us to be able to merge the first milestone drop to trunk, we want to > ensure that once disabled the timeline service v.2 has no impact from the > server side to the client side. If the timeline service is not enabled, no > action should be done. If v.1 is enabled but not v.2, v.1 should behave the > same as it does before the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4356) ensure the timeline service v.2 is disabled cleanly and has no impact when it's turned off
[
https://issues.apache.org/jira/browse/YARN-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15084415#comment-15084415
]
Sangjin Lee commented on YARN-4356:
---
The current code *does* consider the case where the version is greater than 2
but smaller than 3. Note the cast from float to int:
{code}
2384 public static boolean timelineServiceV2Enabled(Configuration conf) {
2385return timelineServiceEnabled(conf) &&
2386(int)getTimelineServiceVersion(conf) == 2;
2387 }
{code}
> ensure the timeline service v.2 is disabled cleanly and has no impact when
> it's turned off
> --
>
> Key: YARN-4356
> URL: https://issues.apache.org/jira/browse/YARN-4356
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Critical
> Labels: yarn-2928-1st-milestone
> Fix For: YARN-2928
>
> Attachments: YARN-4356-feature-YARN-2928.002.patch,
> YARN-4356-feature-YARN-2928.003.patch, YARN-4356-feature-YARN-2928.004.patch,
> YARN-4356-feature-YARN-2928.005.patch, YARN-4356-feature-YARN-2928.006.patch,
> YARN-4356-feature-YARN-2928.poc.001.patch
>
>
> For us to be able to merge the first milestone drop to trunk, we want to
> ensure that once disabled the timeline service v.2 has no impact from the
> server side to the client side. If the timeline service is not enabled, no
> action should be done. If v.1 is enabled but not v.2, v.1 should behave the
> same as it does before the merge.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4356) ensure the timeline service v.2 is disabled cleanly and has no impact when it's turned off
[ https://issues.apache.org/jira/browse/YARN-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15084419#comment-15084419 ] Li Lu commented on YARN-4356: - Oh I missed this part... Thanks for the reminder! > ensure the timeline service v.2 is disabled cleanly and has no impact when > it's turned off > -- > > Key: YARN-4356 > URL: https://issues.apache.org/jira/browse/YARN-4356 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Labels: yarn-2928-1st-milestone > Fix For: YARN-2928 > > Attachments: YARN-4356-feature-YARN-2928.002.patch, > YARN-4356-feature-YARN-2928.003.patch, YARN-4356-feature-YARN-2928.004.patch, > YARN-4356-feature-YARN-2928.005.patch, YARN-4356-feature-YARN-2928.006.patch, > YARN-4356-feature-YARN-2928.poc.001.patch > > > For us to be able to merge the first milestone drop to trunk, we want to > ensure that once disabled the timeline service v.2 has no impact from the > server side to the client side. If the timeline service is not enabled, no > action should be done. If v.1 is enabled but not v.2, v.1 should behave the > same as it does before the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4356) ensure the timeline service v.2 is disabled cleanly and has no impact when it's turned off
[ https://issues.apache.org/jira/browse/YARN-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15084002#comment-15084002 ] Li Lu commented on YARN-4356: - I'm fine to live with it for now. However, if anyone happen to have time I can certainly help to put it in. > ensure the timeline service v.2 is disabled cleanly and has no impact when > it's turned off > -- > > Key: YARN-4356 > URL: https://issues.apache.org/jira/browse/YARN-4356 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Labels: yarn-2928-1st-milestone > Fix For: YARN-2928 > > Attachments: YARN-4356-feature-YARN-2928.002.patch, > YARN-4356-feature-YARN-2928.003.patch, YARN-4356-feature-YARN-2928.004.patch, > YARN-4356-feature-YARN-2928.005.patch, YARN-4356-feature-YARN-2928.006.patch, > YARN-4356-feature-YARN-2928.poc.001.patch > > > For us to be able to merge the first milestone drop to trunk, we want to > ensure that once disabled the timeline service v.2 has no impact from the > server side to the client side. If the timeline service is not enabled, no > action should be done. If v.1 is enabled but not v.2, v.1 should behave the > same as it does before the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4356) ensure the timeline service v.2 is disabled cleanly and has no impact when it's turned off
[ https://issues.apache.org/jira/browse/YARN-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15052792#comment-15052792 ] Junping Du commented on YARN-4356: -- +1 on 006 patch. [~gtCarrera9], may be you want to commit this in? > ensure the timeline service v.2 is disabled cleanly and has no impact when > it's turned off > -- > > Key: YARN-4356 > URL: https://issues.apache.org/jira/browse/YARN-4356 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Labels: yarn-2928-1st-milestone > Attachments: YARN-4356-feature-YARN-2928.002.patch, > YARN-4356-feature-YARN-2928.003.patch, YARN-4356-feature-YARN-2928.004.patch, > YARN-4356-feature-YARN-2928.005.patch, YARN-4356-feature-YARN-2928.006.patch, > YARN-4356-feature-YARN-2928.poc.001.patch > > > For us to be able to merge the first milestone drop to trunk, we want to > ensure that once disabled the timeline service v.2 has no impact from the > server side to the client side. If the timeline service is not enabled, no > action should be done. If v.1 is enabled but not v.2, v.1 should behave the > same as it does before the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4356) ensure the timeline service v.2 is disabled cleanly and has no impact when it's turned off
[
https://issues.apache.org/jira/browse/YARN-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15053902#comment-15053902
]
Hadoop QA commented on YARN-4356:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 14 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
8s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
54s {color} | {color:green} feature-YARN-2928 passed with JDK v1.8.0_66 {color}
|
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
50s {color} | {color:green} feature-YARN-2928 passed with JDK v1.7.0_91 {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
23s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 7m 1s
{color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m
38s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 52s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in
feature-YARN-2928 has 3 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 54s
{color} | {color:red} hadoop-yarn-common in feature-YARN-2928 failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 44s
{color} | {color:red} hadoop-yarn-server-resourcemanager in feature-YARN-2928
failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m 43s
{color} | {color:green} feature-YARN-2928 passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 25s
{color} | {color:red} hadoop-yarn-server-tests in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
27s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 39m 19s
{color} | {color:red} root-jdk1.8.0_66 with JDK v1.8.0_66 generated 5 new
issues (was 779, now 779). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m 27s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
21s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 52m 41s
{color} | {color:red} root-jdk1.7.0_91 with JDK v1.7.0_91 generated 5 new
issues (was 772, now 772). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m 21s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 30s
{color} | {color:red} Patch generated 7 new checkstyle issues in root (total
was 1967, now 1938). {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 29s
{color} | {color:red} hadoop-yarn-server-tests in the patch failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 22s
{color} | {color:red} hadoop-yarn-server-tests in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 8m 30s
{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-jdk1.8.0_66 with JDK
v1.8.0_66 generated 1 new issues (was 100, now 100). {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 8m 30s
{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66
with JDK v1.8.0_66 generated 1 new issues (was 100, now 100). {color} |
| {color:green}+1{color} | {color:green} javadoc {color} |
[jira] [Commented] (YARN-4356) ensure the timeline service v.2 is disabled cleanly and has no impact when it's turned off
[ https://issues.apache.org/jira/browse/YARN-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15053478#comment-15053478 ] Sangjin Lee commented on YARN-4356: --- Thanks folks for the review! > ensure the timeline service v.2 is disabled cleanly and has no impact when > it's turned off > -- > > Key: YARN-4356 > URL: https://issues.apache.org/jira/browse/YARN-4356 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Labels: yarn-2928-1st-milestone > Fix For: YARN-2928 > > Attachments: YARN-4356-feature-YARN-2928.002.patch, > YARN-4356-feature-YARN-2928.003.patch, YARN-4356-feature-YARN-2928.004.patch, > YARN-4356-feature-YARN-2928.005.patch, YARN-4356-feature-YARN-2928.006.patch, > YARN-4356-feature-YARN-2928.poc.001.patch > > > For us to be able to merge the first milestone drop to trunk, we want to > ensure that once disabled the timeline service v.2 has no impact from the > server side to the client side. If the timeline service is not enabled, no > action should be done. If v.1 is enabled but not v.2, v.1 should behave the > same as it does before the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4356) ensure the timeline service v.2 is disabled cleanly and has no impact when it's turned off
[ https://issues.apache.org/jira/browse/YARN-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15053117#comment-15053117 ] Li Lu commented on YARN-4356: - Sure. If there's no objections, I'll commit this patch later this morning. > ensure the timeline service v.2 is disabled cleanly and has no impact when > it's turned off > -- > > Key: YARN-4356 > URL: https://issues.apache.org/jira/browse/YARN-4356 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Labels: yarn-2928-1st-milestone > Attachments: YARN-4356-feature-YARN-2928.002.patch, > YARN-4356-feature-YARN-2928.003.patch, YARN-4356-feature-YARN-2928.004.patch, > YARN-4356-feature-YARN-2928.005.patch, YARN-4356-feature-YARN-2928.006.patch, > YARN-4356-feature-YARN-2928.poc.001.patch > > > For us to be able to merge the first milestone drop to trunk, we want to > ensure that once disabled the timeline service v.2 has no impact from the > server side to the client side. If the timeline service is not enabled, no > action should be done. If v.1 is enabled but not v.2, v.1 should behave the > same as it does before the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4356) ensure the timeline service v.2 is disabled cleanly and has no impact when it's turned off
[ https://issues.apache.org/jira/browse/YARN-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051498#comment-15051498 ] Sangjin Lee commented on YARN-4356: --- Thanks for the feedback. Now that YARN-3623 has been committed, I'll first cherry-pick that commit over to our branch. I'll also update the patch based on comments from [~djp] and [~Naganarasimha]. > ensure the timeline service v.2 is disabled cleanly and has no impact when > it's turned off > -- > > Key: YARN-4356 > URL: https://issues.apache.org/jira/browse/YARN-4356 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Labels: yarn-2928-1st-milestone > Attachments: YARN-4356-feature-YARN-2928.002.patch, > YARN-4356-feature-YARN-2928.003.patch, YARN-4356-feature-YARN-2928.004.patch, > YARN-4356-feature-YARN-2928.005.patch, > YARN-4356-feature-YARN-2928.poc.001.patch > > > For us to be able to merge the first milestone drop to trunk, we want to > ensure that once disabled the timeline service v.2 has no impact from the > server side to the client side. If the timeline service is not enabled, no > action should be done. If v.1 is enabled but not v.2, v.1 should behave the > same as it does before the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4356) ensure the timeline service v.2 is disabled cleanly and has no impact when it's turned off
[ https://issues.apache.org/jira/browse/YARN-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051966#comment-15051966 ] Li Lu commented on YARN-4356: - I can see in the v.6 patch both concerns are addressed. Any other comments/suggestions/concerns? > ensure the timeline service v.2 is disabled cleanly and has no impact when > it's turned off > -- > > Key: YARN-4356 > URL: https://issues.apache.org/jira/browse/YARN-4356 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Labels: yarn-2928-1st-milestone > Attachments: YARN-4356-feature-YARN-2928.002.patch, > YARN-4356-feature-YARN-2928.003.patch, YARN-4356-feature-YARN-2928.004.patch, > YARN-4356-feature-YARN-2928.005.patch, YARN-4356-feature-YARN-2928.006.patch, > YARN-4356-feature-YARN-2928.poc.001.patch > > > For us to be able to merge the first milestone drop to trunk, we want to > ensure that once disabled the timeline service v.2 has no impact from the > server side to the client side. If the timeline service is not enabled, no > action should be done. If v.1 is enabled but not v.2, v.1 should behave the > same as it does before the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4356) ensure the timeline service v.2 is disabled cleanly and has no impact when it's turned off
[
https://issues.apache.org/jira/browse/YARN-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050645#comment-15050645
]
Junping Du commented on YARN-4356:
--
Thanks [~sjlee0] for updating the patch!
bq. This is the java 7 diamond operator (<>) which is a shorthand for inferring
types. The type information is NOT removed. It's inferred by the compiler, and
the compiler produces the same bytecode as specifying the types.
Got it. Sounds like my coffee is stale and I need a new cup. :)
bq. Got it. Can we proceed with the current patch and get that fix once
YARN-3586 goes in?
Sure. Just a reminder here as I also forget it myself.
bq. This is addressing a javadoc error. The ampersand ("&") is a special
character for javadoc, and it breaks javadoc. It needs to be entity-escaped.
Oh. I see. Let's keep it here.
bq. This is used for the test method launchServer(). This method is invoked
directly by a unit test (thus the @VisibleForTesting annotation). The same for
TimelineReaderServer.
If so, I would prefer not to update configuration in launchServer(), but pass a
updated configuration to the method instead, just as other daemons (RM, NM)
doing. The reason is it could be hard to track the real configurations for
daemons/services if we override them in internal logic. May be we should follow
the same practices?
bq. That's fine. I still put up the patch that includes a version of that
because without it things won't even compile. I will wait until YARN-3623 goes
in before I remove that piece from this patch, then this can get committed.
Sounds good. I just commit YARN-3623 to trunk.
Other looks good to me in 005 patch.
> ensure the timeline service v.2 is disabled cleanly and has no impact when
> it's turned off
> --
>
> Key: YARN-4356
> URL: https://issues.apache.org/jira/browse/YARN-4356
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Critical
> Labels: yarn-2928-1st-milestone
> Attachments: YARN-4356-feature-YARN-2928.002.patch,
> YARN-4356-feature-YARN-2928.003.patch, YARN-4356-feature-YARN-2928.004.patch,
> YARN-4356-feature-YARN-2928.005.patch,
> YARN-4356-feature-YARN-2928.poc.001.patch
>
>
> For us to be able to merge the first milestone drop to trunk, we want to
> ensure that once disabled the timeline service v.2 has no impact from the
> server side to the client side. If the timeline service is not enabled, no
> action should be done. If v.1 is enabled but not v.2, v.1 should behave the
> same as it does before the merge.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4356) ensure the timeline service v.2 is disabled cleanly and has no impact when it's turned off
[
https://issues.apache.org/jira/browse/YARN-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051306#comment-15051306
]
Hadoop QA commented on YARN-4356:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 12 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
28s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 7s
{color} | {color:green} feature-YARN-2928 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
55s {color} | {color:green} feature-YARN-2928 passed with JDK v1.7.0_91 {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
27s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 7m 3s
{color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m
18s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 54s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in
feature-YARN-2928 has 3 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 50s
{color} | {color:red} hadoop-yarn-common in feature-YARN-2928 failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 42s
{color} | {color:red} hadoop-yarn-server-resourcemanager in feature-YARN-2928
failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m 59s
{color} | {color:green} feature-YARN-2928 passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
13s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 36m 15s
{color} | {color:red} root-jdk1.8.0_66 with JDK v1.8.0_66 generated 5 new
issues (was 779, now 779). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
55s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 49m 10s
{color} | {color:red} root-jdk1.7.0_91 with JDK v1.7.0_91 generated 5 new
issues (was 772, now 772). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m 55s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 29s
{color} | {color:red} Patch generated 7 new checkstyle issues in root (total
was 1970, now 1942). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 6m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 14m
9s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 7m 8s
{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-jdk1.8.0_66 with JDK
v1.8.0_66 generated 1 new issues (was 100, now 100). {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 7m 8s
{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66
with JDK v1.8.0_66 generated 1 new issues (was 100, now 100). {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 49s
{color} | {color:green} the patch passed with
[jira] [Commented] (YARN-4356) ensure the timeline service v.2 is disabled cleanly and has no impact when it's turned off
[ https://issues.apache.org/jira/browse/YARN-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048671#comment-15048671 ] Junping Du commented on YARN-4356: -- I would like to review this as many of this patch related to my previous work. [~gtCarrera9], would you hold on the commit? > ensure the timeline service v.2 is disabled cleanly and has no impact when > it's turned off > -- > > Key: YARN-4356 > URL: https://issues.apache.org/jira/browse/YARN-4356 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Labels: yarn-2928-1st-milestone > Attachments: YARN-4356-feature-YARN-2928.002.patch, > YARN-4356-feature-YARN-2928.003.patch, YARN-4356-feature-YARN-2928.004.patch, > YARN-4356-feature-YARN-2928.poc.001.patch > > > For us to be able to merge the first milestone drop to trunk, we want to > ensure that once disabled the timeline service v.2 has no impact from the > server side to the client side. If the timeline service is not enabled, no > action should be done. If v.1 is enabled but not v.2, v.1 should behave the > same as it does before the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4356) ensure the timeline service v.2 is disabled cleanly and has no impact when it's turned off
[ https://issues.apache.org/jira/browse/YARN-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048970#comment-15048970 ] Sangjin Lee commented on YARN-4356: --- The jenkins run above is suspect as it doesn't seem to match v.3. And the next run for v.4 was aborted. Kicking off a new run for v.4. > ensure the timeline service v.2 is disabled cleanly and has no impact when > it's turned off > -- > > Key: YARN-4356 > URL: https://issues.apache.org/jira/browse/YARN-4356 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Labels: yarn-2928-1st-milestone > Attachments: YARN-4356-feature-YARN-2928.002.patch, > YARN-4356-feature-YARN-2928.003.patch, YARN-4356-feature-YARN-2928.004.patch, > YARN-4356-feature-YARN-2928.poc.001.patch > > > For us to be able to merge the first milestone drop to trunk, we want to > ensure that once disabled the timeline service v.2 has no impact from the > server side to the client side. If the timeline service is not enabled, no > action should be done. If v.1 is enabled but not v.2, v.1 should behave the > same as it does before the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4356) ensure the timeline service v.2 is disabled cleanly and has no impact when it's turned off
[
https://issues.apache.org/jira/browse/YARN-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049039#comment-15049039
]
Junping Du commented on YARN-4356:
--
Thanks [~sjlee0] for delivering the patch. 004 patch looks good to me in
overall, but some minor comments:
In JobHistoryEventHandler.java,
{code}
LOG.info("Emitting job history data to the timeline server is enabled");
{code}
server => service, given we don't provide a centralized server in v2.
{code}
+LOG.info("Timeline service is enabled; version: " +
+(timelineServiceV2Enabled? "v2" : "v1"));
{code}
Shall we get configuration version from configuration defined in YARN-3623?
Especially, we have other versions, like: v1.5.
TestMRTimelineEventHandling.java,
{code}
// enable new timeline serivce
{code}
typo: serivce => service
YarnConfiguration.java,
{code}
@return whether the timelien service is enabled.
{code}
typo: timelien => timeline
{code}
+ public static boolean timelineServiceV2Enabled(Configuration conf) {
+return timelineServiceEnabled(conf) && getTimelineServiceVersion(conf) ==
2;
+ }
{code}
Would it be possible to have ATS v2.5 in future? If so, may be we should cast
float number get from version config before comparing with 2?
In NodeHeartbeatRequestPBImpl.java,
{code}
+ this.registeredCollectors = new HashMap<>();
Update new HashMap<>() => new HashMap ()
{code}
and the same problem in NodeHeartbeatResponsePBImpl.java
{code}
this.appCollectorsMap = new HashMap<>();
{code}
In NodeManager.java,
{code}
this.registeredCollectors = new ConcurrentHashMap<>();
{code}
We should also add back types as requirement/convension for generics.
{code}
+ if (this.registeredCollectors != null) {
+this.registeredCollectors.putAll(newRegisteredCollectors);
+ }
{code}
This check of null is unnecessary as the only caller - NMCollectorService is
only running under v2 is enabled. If for some reason, we get NPE here which is
still better than we ignore it silently.
In NodeStatusUpdaterImpl.java,
{code}
- /**
- * Caller should take care of sending non null nodelabels for both
- * arguments
- *
- * @param nodeLabelsNew
- * @param nodeLabelsOld
- * @return if the New node labels are diff from the older one.
- */
- private boolean areNodeLabelsUpdated(Set nodeLabelsNew,
- Set nodeLabelsOld) {
-if (nodeLabelsNew.size() != nodeLabelsOld.size()
-|| !nodeLabelsOld.containsAll(nodeLabelsNew)) {
- return true;
-}
-return false;
- }
{code}
Please remove this unrelated change out for more focus and better tracking.
{code}
+!context.getRegisteredCollectors().containsKey(appId)) {
{code}
I think this logic could be problemtic if collector address get updated due to
NM restart or collector service failure. However, this shouldn't be addressed
in this JIRA. But kindly add a TODO would a good reminder.
In ContainerManagerImpl.java,
{code}
+} else {
+ flowContext = null;
}
{code}
this else branch is not necessary as we can define flowContext to be null at
the beginning.
In ResourceManager.java,
{code}
+ if (version < 2 &&
...
+ } else if (version == 2 &&
{code}
Can we continuously to use YarnConfiguration.timelineServiceV2Enabled() in
every place? Or we could miss these places if we need to change version logic
in future.
In ResourceTrackerService.java,
{code}
List keepAliveApps =
remoteNodeStatus.getKeepAliveApplications();
-if (keepAliveApps != null) {
+if (timelineV2Enabled && keepAliveApps != null) {
{code}
Just a reminder, keepAliveApps is a wrong list to identify running apps on
specific node. YARN-3586 (with patch) is already filed to fix this. We can
either merge that patch in or rebase that patch when this patch done.
In TimelineServiceV2Publisher.java,
{code}
- * This class is responsible for posting application, appattempt & Container
+ * This class is responsible for posting application, appattempt
Container
{code}
Why we need this change?
In PerNodeTimelineCollectorsAuxService.java,
{code}
+ // enable timeline service v.2
+ conf.setBoolean(YarnConfiguration.TIMELINE_SERVICE_ENABLED, true);
+ conf.setFloat(YarnConfiguration.TIMELINE_SERVICE_VERSION, 2.0f);
{code}
We should disable PerNodeTimelineCollectorsAuxService if we don't enable
timeline service v2. Isn't it? If so, I think this is not a necessary change
and we should remove.
In TimelineReaderServer.java,
{code}
+YarnConfiguration conf = new YarnConfiguration();
+// enable timeline service v.2
+conf.setBoolean(YarnConfiguration.TIMELINE_SERVICE_ENABLED, true);
+conf.setFloat(YarnConfiguration.TIMELINE_SERVICE_VERSION, 2.0f);
{code}
The same question with above.
In addition, I think we should split the change that duplicated
[jira] [Commented] (YARN-4356) ensure the timeline service v.2 is disabled cleanly and has no impact when it's turned off
[ https://issues.apache.org/jira/browse/YARN-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049204#comment-15049204 ] Naganarasimha G R commented on YARN-4356: - Hi [~sjlee0] few points from my side # I felt better to extract a interface from NMTimelinePublisher a no op impl and regular impl so that we can avoid null checks by the caller everytime # We would not require two different configs to indicate enabling of the SystemMetricsPublisher we can YarnConfiguration.SYSTEM_METRICS_PUBLISHER_ENABLED and depricate YarnConfiguration.RM_SYSTEM_METRICS_PUBLISHER_ENABLED (at the same time map the value from the prev key to the new key) > ensure the timeline service v.2 is disabled cleanly and has no impact when > it's turned off > -- > > Key: YARN-4356 > URL: https://issues.apache.org/jira/browse/YARN-4356 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Labels: yarn-2928-1st-milestone > Attachments: YARN-4356-feature-YARN-2928.002.patch, > YARN-4356-feature-YARN-2928.003.patch, YARN-4356-feature-YARN-2928.004.patch, > YARN-4356-feature-YARN-2928.poc.001.patch > > > For us to be able to merge the first milestone drop to trunk, we want to > ensure that once disabled the timeline service v.2 has no impact from the > server side to the client side. If the timeline service is not enabled, no > action should be done. If v.1 is enabled but not v.2, v.1 should behave the > same as it does before the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4356) ensure the timeline service v.2 is disabled cleanly and has no impact when it's turned off
[
https://issues.apache.org/jira/browse/YARN-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049392#comment-15049392
]
Hadoop QA commented on YARN-4356:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 12 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
55s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
52s {color} | {color:green} feature-YARN-2928 passed with JDK v1.8.0_66 {color}
|
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
21s {color} | {color:green} feature-YARN-2928 passed with JDK v1.7.0_91 {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
33s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 7m 14s
{color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m
27s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 53s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in
feature-YARN-2928 has 3 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 53s
{color} | {color:red} hadoop-yarn-common in feature-YARN-2928 failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 43s
{color} | {color:red} hadoop-yarn-server-resourcemanager in feature-YARN-2928
failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m 17s
{color} | {color:green} feature-YARN-2928 passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
55s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 37m 33s
{color} | {color:red} root-jdk1.8.0_66 with JDK v1.8.0_66 generated 5 new
issues (was 779, now 779). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m 55s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
24s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 50m 57s
{color} | {color:red} root-jdk1.7.0_91 with JDK v1.7.0_91 generated 5 new
issues (was 772, now 772). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m 24s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 29s
{color} | {color:red} Patch generated 7 new checkstyle issues in root (total
was 1970, now 1942). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 7m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 15m
7s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 7m 36s
{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-jdk1.8.0_66 with JDK
v1.8.0_66 generated 1 new issues (was 100, now 100). {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 7m 36s
{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66
with JDK v1.8.0_66 generated 1 new issues (was 100, now 100). {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m 16s
{color} | {color:green} the patch passed
[jira] [Commented] (YARN-4356) ensure the timeline service v.2 is disabled cleanly and has no impact when it's turned off
[
https://issues.apache.org/jira/browse/YARN-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049817#comment-15049817
]
Sangjin Lee commented on YARN-4356:
---
Thanks [~djp] for your comments. I addressed most of your comments in the new
patch. The following is a response to your comments.
bq. We should also add back types as requirement/convension for generics.
This is the java 7 diamond operator (<>) which is a shorthand for inferring
types. The type information is NOT removed. It's inferred by the compiler, and
the compiler produces the same bytecode as specifying the types.
bq. This check of null is unnecessary as the only caller - NMCollectorService
is only running under v2 is enabled. If for some reason, we get NPE here which
is still better than we ignore it silently.
That's a good catch. I agree that it's a little better not to check for null
here. It's changed in the latest patch.
bq. Please remove this unrelated change out for more focus and better tracking.
Agreed. I originally removed it because it was an unused private method, but I
put it back in.
bq. Just a reminder, keepAliveApps is a wrong list to identify running apps on
specific node. YARN-3586 (with patch) is already filed to fix this. We can
either merge that patch in or rebase that patch when this patch done.
Got it. Can we proceed with the current patch and get that fix once YARN-3586
goes in?
{quote}
In TimelineServiceV2Publisher.java,
- * This class is responsible for posting application, appattempt & Container
+ * This class is responsible for posting application, appattempt
Container
Why we need this change?
{quote}
This is addressing a javadoc error. The ampersand ("&") is a special character
for javadoc, and it breaks javadoc. It needs to be entity-escaped:
{noformat}
[ERROR]
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TimelineServiceV2Publisher.java:61:
error: bad HTML entity
[ERROR] * This class is responsible for posting application, appattempt &
Container
{noformat}
bq. We should disable PerNodeTimelineCollectorsAuxService if we don't enable
timeline service v2. Isn't it? If so, I think this is not a necessary change
and we should remove.
This is used for the test method launchServer(). This method is invoked
directly by a unit test (thus the @VisibleForTesting annotation). The same for
TimelineReaderServer.
bq. In addition, I think we should split the change that duplicated with
YARN-3623 and cherry-pick it from trunk/branch-2 when that patch get commit in.
That's fine. I still put up the patch that includes a version of that because
without it things won't even compile. I will wait until YARN-3623 goes in
before I remove that piece from this patch, then this can get committed.
Let me know if this answers your questions.
> ensure the timeline service v.2 is disabled cleanly and has no impact when
> it's turned off
> --
>
> Key: YARN-4356
> URL: https://issues.apache.org/jira/browse/YARN-4356
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Critical
> Labels: yarn-2928-1st-milestone
> Attachments: YARN-4356-feature-YARN-2928.002.patch,
> YARN-4356-feature-YARN-2928.003.patch, YARN-4356-feature-YARN-2928.004.patch,
> YARN-4356-feature-YARN-2928.005.patch,
> YARN-4356-feature-YARN-2928.poc.001.patch
>
>
> For us to be able to merge the first milestone drop to trunk, we want to
> ensure that once disabled the timeline service v.2 has no impact from the
> server side to the client side. If the timeline service is not enabled, no
> action should be done. If v.1 is enabled but not v.2, v.1 should behave the
> same as it does before the merge.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4356) ensure the timeline service v.2 is disabled cleanly and has no impact when it's turned off
[ https://issues.apache.org/jira/browse/YARN-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049982#comment-15049982 ] Naganarasimha G R commented on YARN-4356: - Hi [~sjlee0], bq. Right now those null checks are still limited to a couple of files. Although it may not be ideal, it is still manageable? Its not a blocking thing and its just will help it in keeping it clean so no issues we can go ahead for this patch will try to take care in one of my other patches related to NMMetricsPublisher. bq. Hmm, I know it's not great having 2 similar-sounding config params, but there is also RM_SYSTEM_METRICS_PUBLISHER_DISPATCHER_POOL_SIZE, so this might also make this patch a little bigger. Ok we can take care in another jira, but YarnConfiguration.SYSTEM_METRICS_PUBLISHER_ENABLED and YarnConfiguration.RM_SYSTEM_METRICS_PUBLISHER_ENABLED was introduced because version was not there . > ensure the timeline service v.2 is disabled cleanly and has no impact when > it's turned off > -- > > Key: YARN-4356 > URL: https://issues.apache.org/jira/browse/YARN-4356 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Labels: yarn-2928-1st-milestone > Attachments: YARN-4356-feature-YARN-2928.002.patch, > YARN-4356-feature-YARN-2928.003.patch, YARN-4356-feature-YARN-2928.004.patch, > YARN-4356-feature-YARN-2928.005.patch, > YARN-4356-feature-YARN-2928.poc.001.patch > > > For us to be able to merge the first milestone drop to trunk, we want to > ensure that once disabled the timeline service v.2 has no impact from the > server side to the client side. If the timeline service is not enabled, no > action should be done. If v.1 is enabled but not v.2, v.1 should behave the > same as it does before the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4356) ensure the timeline service v.2 is disabled cleanly and has no impact when it's turned off
[ https://issues.apache.org/jira/browse/YARN-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15047055#comment-15047055 ] Sangjin Lee commented on YARN-4356: --- The jenkins build didn't fire automatically. Kicking off a manual build. > ensure the timeline service v.2 is disabled cleanly and has no impact when > it's turned off > -- > > Key: YARN-4356 > URL: https://issues.apache.org/jira/browse/YARN-4356 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Labels: yarn-2928-1st-milestone > Attachments: YARN-4356-feature-YARN-2928.002.patch, > YARN-4356-feature-YARN-2928.poc.001.patch > > > For us to be able to merge the first milestone drop to trunk, we want to > ensure that once disabled the timeline service v.2 has no impact from the > server side to the client side. If the timeline service is not enabled, no > action should be done. If v.1 is enabled but not v.2, v.1 should behave the > same as it does before the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4356) ensure the timeline service v.2 is disabled cleanly and has no impact when it's turned off
[
https://issues.apache.org/jira/browse/YARN-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15047232#comment-15047232
]
Hadoop QA commented on YARN-4356:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 12 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
5s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 15s
{color} | {color:green} feature-YARN-2928 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 0s
{color} | {color:green} feature-YARN-2928 passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m 0s
{color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
27s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 17s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in
feature-YARN-2928 has 3 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 28s
{color} | {color:red} hadoop-yarn-common in feature-YARN-2928 failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 21s
{color} | {color:red} hadoop-yarn-server-resourcemanager in feature-YARN-2928
failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 58s
{color} | {color:green} feature-YARN-2928 passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 50s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 22m 2s {color}
| {color:red} root-jdk1.8.0_66 with JDK v1.8.0_66 generated 5 new issues (was
780, now 780). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 46s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 30m 49s
{color} | {color:red} root-jdk1.7.0_91 with JDK v1.7.0_91 generated 5 new
issues (was 772, now 772). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 46s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 1s
{color} | {color:red} Patch generated 26 new checkstyle issues in root (total
was 1938, now 1932). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m 55s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 10m
6s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 4m 11s
{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-jdk1.8.0_66 with JDK v1.8.0_66
generated 6 new issues (was 100, now 100). {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 27s
{color} | {color:red} hadoop-yarn-common in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 21s
{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed
with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} |
[jira] [Commented] (YARN-4356) ensure the timeline service v.2 is disabled cleanly and has no impact when it's turned off
[
https://issues.apache.org/jira/browse/YARN-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15047738#comment-15047738
]
Hadoop QA commented on YARN-4356:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 12 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
45s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
13s {color} | {color:green} feature-YARN-2928 passed with JDK v1.8.0_66 {color}
|
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 40s
{color} | {color:green} feature-YARN-2928 passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m 17s
{color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
33s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 21s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in
feature-YARN-2928 has 3 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 32s
{color} | {color:red} hadoop-yarn-common in feature-YARN-2928 failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 25s
{color} | {color:red} hadoop-yarn-server-resourcemanager in feature-YARN-2928
failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m 39s
{color} | {color:green} feature-YARN-2928 passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 51s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 24m 41s
{color} | {color:red} root-jdk1.8.0_66 with JDK v1.8.0_66 generated 5 new
issues (was 779, now 779). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 30s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 34m 11s
{color} | {color:red} root-jdk1.7.0_85 with JDK v1.7.0_85 generated 5 new
issues (was 772, now 772). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m 30s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 3s
{color} | {color:red} Patch generated 26 new checkstyle issues in root (total
was 1937, now 1931). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 10m
43s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 35s
{color} | {color:red} hadoop-yarn-common in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 27s
{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed
with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m 27s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 26s
{color} | {color:green} hadoop-yarn-api in the patch
[jira] [Commented] (YARN-4356) ensure the timeline service v.2 is disabled cleanly and has no impact when it's turned off
[ https://issues.apache.org/jira/browse/YARN-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048106#comment-15048106 ] Li Lu commented on YARN-4356: - Latest patch LGTM. +1 pending Jenkins. I'll wait for one more day and if there's no objection I'll commit it. > ensure the timeline service v.2 is disabled cleanly and has no impact when > it's turned off > -- > > Key: YARN-4356 > URL: https://issues.apache.org/jira/browse/YARN-4356 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Labels: yarn-2928-1st-milestone > Attachments: YARN-4356-feature-YARN-2928.002.patch, > YARN-4356-feature-YARN-2928.003.patch, YARN-4356-feature-YARN-2928.004.patch, > YARN-4356-feature-YARN-2928.poc.001.patch > > > For us to be able to merge the first milestone drop to trunk, we want to > ensure that once disabled the timeline service v.2 has no impact from the > server side to the client side. If the timeline service is not enabled, no > action should be done. If v.1 is enabled but not v.2, v.1 should behave the > same as it does before the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4356) ensure the timeline service v.2 is disabled cleanly and has no impact when it's turned off
[ https://issues.apache.org/jira/browse/YARN-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048049#comment-15048049 ] Sangjin Lee commented on YARN-4356: --- Thanks for your review [~gtCarrera9]. bq. I noticed in some files we're verifying v2 in a hard-coded fashion (version == 2). Why do we still need this especially when we have timelineServiceV2Enabled()? The only reason for them is because timelineServiceV2Enabled() is timelineServiceEnabled() + (timelineServiceVersion == 2). In those cases, timelineServiceEnabled() was already checked. Thus, as a (small) optimization I just checked the version directly. Having said that, I'm comfortable with changing them to call timelineServiceV2Enabled() even though it may check timelineServiceEnabled() one extra time. I'll make those changes. bq. That is, if the timeline-service.version is set to 2.x in future, are the applications allowed to use other versions of ATS? It should be possible in principle with the assumption that some compatibility mechanism is in place so an old API invocation can succeed. The config is there to discover what's running on the cluster. If there is a compatibility mechanism, applications may invoke a different API (it's entirely up to them at that point). bq. ApplicationMaster, function names "...OnNewTimelineService" can be more specific like "...V2"? Sounds good. I didn't rename methods as part of this work, but let me see if I can rename them to use "v2". bq. ContainerManagerImpl, I just want to double check one behavior: the SMP is enabled for the NM only when timeline version is v2 and SMP is enabled in the config? What about v1.x versions? If this is a v2 only feature, shall we clarify that in the log message? The v.1 behavior should be essentially the same as today. The existing v.1 behavior is to check TIMELINE_SERVICE_ENABLED and then RM_SYSTEM_METRICS_PUBLISHER_ENABLED. You'll see that the current patch checks TIMELINE_SERVICE_ENABLED, TIMELINE_SERVICE_VERSION == 1, and RM_SYSTEM_METRICS_PUBLISHER_ENABLED. I do see that it's checking strictly for version = 1. I'll change it to check for version < 2 so it can match 1.5 as well. > ensure the timeline service v.2 is disabled cleanly and has no impact when > it's turned off > -- > > Key: YARN-4356 > URL: https://issues.apache.org/jira/browse/YARN-4356 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Labels: yarn-2928-1st-milestone > Attachments: YARN-4356-feature-YARN-2928.002.patch, > YARN-4356-feature-YARN-2928.003.patch, > YARN-4356-feature-YARN-2928.poc.001.patch > > > For us to be able to merge the first milestone drop to trunk, we want to > ensure that once disabled the timeline service v.2 has no impact from the > server side to the client side. If the timeline service is not enabled, no > action should be done. If v.1 is enabled but not v.2, v.1 should behave the > same as it does before the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4356) ensure the timeline service v.2 is disabled cleanly and has no impact when it's turned off
[ https://issues.apache.org/jira/browse/YARN-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15047912#comment-15047912 ] Li Lu commented on YARN-4356: - Hi [~sjlee0], thanks for the work! Mostly LGTM, just a few thing to check: 1. I noticed in some files we're verifying v2 in a hard-coded fashion (version == 2). Why do we still need this especially when we have timelineServiceV2Enabled()? 2. MapRed will use the timeline.version config as the current active API version. I'm fine with this design. One thing to check: do we allow other applications to customize the active API version for themselves? That is, if the timeline-service.version is set to 2.x in future, are the applications allowed to use other versions of ATS? (I think in this case the compatibility story should be made by the application itself? ) 3. ApplicationMaster, function names "...OnNewTimelineService" can be more specific like "...V2"? 4. ContainerManagerImpl, I just want to double check one behavior: the SMP is enabled for the NM only when timeline version is v2 and SMP is enabled in the config? What about v1.x versions? If this is a v2 only feature, shall we clarify that in the log message? Thanks! > ensure the timeline service v.2 is disabled cleanly and has no impact when > it's turned off > -- > > Key: YARN-4356 > URL: https://issues.apache.org/jira/browse/YARN-4356 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Labels: yarn-2928-1st-milestone > Attachments: YARN-4356-feature-YARN-2928.002.patch, > YARN-4356-feature-YARN-2928.003.patch, > YARN-4356-feature-YARN-2928.poc.001.patch > > > For us to be able to merge the first milestone drop to trunk, we want to > ensure that once disabled the timeline service v.2 has no impact from the > server side to the client side. If the timeline service is not enabled, no > action should be done. If v.1 is enabled but not v.2, v.1 should behave the > same as it does before the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4356) ensure the timeline service v.2 is disabled cleanly and has no impact when it's turned off
[ https://issues.apache.org/jira/browse/YARN-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048067#comment-15048067 ] Sangjin Lee commented on YARN-4356: --- Oh I see. Yes, there is no NM metrics publisher in ATS v.1.x, so it should be fine. Thanks for that. > ensure the timeline service v.2 is disabled cleanly and has no impact when > it's turned off > -- > > Key: YARN-4356 > URL: https://issues.apache.org/jira/browse/YARN-4356 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Labels: yarn-2928-1st-milestone > Attachments: YARN-4356-feature-YARN-2928.002.patch, > YARN-4356-feature-YARN-2928.003.patch, > YARN-4356-feature-YARN-2928.poc.001.patch > > > For us to be able to merge the first milestone drop to trunk, we want to > ensure that once disabled the timeline service v.2 has no impact from the > server side to the client side. If the timeline service is not enabled, no > action should be done. If v.1 is enabled but not v.2, v.1 should behave the > same as it does before the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4356) ensure the timeline service v.2 is disabled cleanly and has no impact when it's turned off
[
https://issues.apache.org/jira/browse/YARN-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15045528#comment-15045528
]
Sangjin Lee commented on YARN-4356:
---
Thanks for your quick comments [~vrushalic]!
bq. +1 on creating a FlowContext. Could we add user as a class member too?
I thought about that briefly, but ultimately decided against it. One main
reason is to mind the memory size of the ApplicationImpl object. If the
timeline service v.2 is disabled, the current patch adds one object reference
(null FlowContext) to ApplicationImpl. Note user already exists on
ApplicationImpl. If we were to move user into FlowContext, then we would add
one object reference (FlowContext) and two more object references (flowName and
flowVersion) and a long reference. So in the interest of keeping the impact on
ApplicationImpl small, I leaned towards keeping only v.2-specific data in
FlowContext. This FlowContext is purely a node-manager internal data structure
that simplifies creating ApplicationImpl, so this isn't really a public API.
Let me know if you're OK with that.
bq. Do we want to put in a check if timeline v2 is enabled in the hbase writer
and reader too?
Yes, I believe it would only be consistent. I'm going to add that in the next
patch.
{quote}
at line 274 in JobHistoryEventHandler
MRJobConfig.MAPREDUCE_JOB_NEW_TIMELINE_SERVICE_ENABLED
I see that this code is in the map reduce context since most other variables
are MAPREDUCE_something but do you think we should also check the Yarn Config
variable? Or is that not visible in this code.
Else we would need to set two config variables one being
MRJobConfig.MAPREDUCE_JOB_NEW_TIMELINE_SERVICE_ENABLED and the other
YarnConfiguration.TIMELINE_SERVICE_ENABLED ?
{quote}
YarnConfiguration.TIMELINE_SERVICE_ENABLED is already being checked (see line
265).
bq. Also the logging messages on lines 275 and 276 perhaps need to be updated?
Good catch. I'll touch it up. I also found that I should have surrounded
creating the thread pool there with a check for newTimelineService. That will
be fixed too.
bq. In ApplicationMaster.java, it looks like some of the functions are no
longer static. I did see that you mentioned that all the unit tests pass but if
we change this, would any user code break? For folks who write their own app
master? Or perhaps the unit tests cover that case.
All those methods were private. Also, this is the application master code for
the distributed shell. As such, it should not impact anyone even if they were
public.
bq. at line 357 in TimelineClientImpl.java, do we want to throw an exception or
log a warning and return? If we throw an error, would existing apps break if
they are to be run without recompilation for the new client.
This is a new method that was added specifically for v.2. Thus, there shouldn't
be existing code that calls this (as it does not exist in trunk yet).
> ensure the timeline service v.2 is disabled cleanly and has no impact when
> it's turned off
> --
>
> Key: YARN-4356
> URL: https://issues.apache.org/jira/browse/YARN-4356
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Critical
> Labels: yarn-2928-1st-milestone
> Attachments: YARN-4356-feature-YARN-2928.poc.001.patch
>
>
> For us to be able to merge the first milestone drop to trunk, we want to
> ensure that once disabled the timeline service v.2 has no impact from the
> server side to the client side. If the timeline service is not enabled, no
> action should be done. If v.1 is enabled but not v.2, v.1 should behave the
> same as it does before the merge.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4356) ensure the timeline service v.2 is disabled cleanly and has no impact when it's turned off
[ https://issues.apache.org/jira/browse/YARN-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15042180#comment-15042180 ] Vrushali C commented on YARN-4356: -- Thanks Sangjin for the initial patch! I had a few discussion points. - +1 on creating a FlowContext. Could we add user as a class member too? - Do we want to put in a check if timeline v2 is enabled in the hbase writer and reader too? - at line 274 in JobHistoryEventHandler - MRJobConfig.MAPREDUCE_JOB_NEW_TIMELINE_SERVICE_ENABLED I see that this code is in the map reduce context since most other variables are MAPREDUCE_something but do you think we should also check the Yarn Config variable? Or is that not visible in this code. Else we would need to set two config variables one being MRJobConfig.MAPREDUCE_JOB_NEW_TIMELINE_SERVICE_ENABLED and the other YarnConfiguration.TIMELINE_SERVICE_ENABLED ? - Also the logging messages on lines 275 and 276 perhaps need to be updated? - In ApplicationMaster.java, it looks like some of the functions are no longer static. I did see that you mentioned that all the unit tests pass but if we change this, would any user code break? For folks who write their own app master? Or perhaps the unit tests cover that case. - at line 357 in TimelineClientImpl.java, do we want to throw an exception or log a warning and return? If we throw an error, would existing apps break if they are to be run without recompilation for the new client. > ensure the timeline service v.2 is disabled cleanly and has no impact when > it's turned off > -- > > Key: YARN-4356 > URL: https://issues.apache.org/jira/browse/YARN-4356 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Labels: yarn-2928-1st-milestone > Attachments: YARN-4356-feature-YARN-2928.poc.001.patch > > > For us to be able to merge the first milestone drop to trunk, we want to > ensure that once disabled the timeline service v.2 has no impact from the > server side to the client side. If the timeline service is not enabled, no > action should be done. If v.1 is enabled but not v.2, v.1 should behave the > same as it does before the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4356) ensure the timeline service v.2 is disabled cleanly and has no impact when it's turned off
[ https://issues.apache.org/jira/browse/YARN-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15011604#comment-15011604 ] Li Lu commented on YARN-4356: - bq. I haven't looked at YARN-4234 in detail yet. Are you referring to the version config? Yes. bq. While I see value in being able to do so, I recognize that it would be a pretty major rework, as the current code is making a strong assumption of this being a binary choice (mutually exclusive). Agreed. Let's move forward with checking all v2 code can be cleanly disabled here, and see how we can allow two versions running side by side in the other JIRA? > ensure the timeline service v.2 is disabled cleanly and has no impact when > it's turned off > -- > > Key: YARN-4356 > URL: https://issues.apache.org/jira/browse/YARN-4356 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Labels: yarn-2928-1st-milestone > > For us to be able to merge the first milestone drop to trunk, we want to > ensure that once disabled the timeline service v.2 has no impact from the > server side to the client side. If the timeline service is not enabled, no > action should be done. If v.1 is enabled but not v.2, v.1 should behave the > same as it does before the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4356) ensure the timeline service v.2 is disabled cleanly and has no impact when it's turned off
[ https://issues.apache.org/jira/browse/YARN-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15012525#comment-15012525 ] Sangjin Lee commented on YARN-4356: --- Sounds good. That other JIRA is YARN-4368. > ensure the timeline service v.2 is disabled cleanly and has no impact when > it's turned off > -- > > Key: YARN-4356 > URL: https://issues.apache.org/jira/browse/YARN-4356 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Labels: yarn-2928-1st-milestone > > For us to be able to merge the first milestone drop to trunk, we want to > ensure that once disabled the timeline service v.2 has no impact from the > server side to the client side. If the timeline service is not enabled, no > action should be done. If v.1 is enabled but not v.2, v.1 should behave the > same as it does before the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4356) ensure the timeline service v.2 is disabled cleanly and has no impact when it's turned off
[ https://issues.apache.org/jira/browse/YARN-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15012535#comment-15012535 ] Sangjin Lee commented on YARN-4356: --- In order to make progress with this work, I would like to propose the following. Let's use yarn.timeline-service.enabled = true && yarn.timeline-service.version = 2.0 (per YARN-4234) as the test if the timeline service v.2 is enabled both on the server side and the client side. With that check, let's wire off the timeline service v.2 feature as cleanly and early as possible. As we said earlier, we can focus on disabling the feature in this JIRA (assume a binary behavior for now). We can look at whether we can enable both in YARN-4368. Does it sound like a plan? > ensure the timeline service v.2 is disabled cleanly and has no impact when > it's turned off > -- > > Key: YARN-4356 > URL: https://issues.apache.org/jira/browse/YARN-4356 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Labels: yarn-2928-1st-milestone > > For us to be able to merge the first milestone drop to trunk, we want to > ensure that once disabled the timeline service v.2 has no impact from the > server side to the client side. If the timeline service is not enabled, no > action should be done. If v.1 is enabled but not v.2, v.1 should behave the > same as it does before the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4356) ensure the timeline service v.2 is disabled cleanly and has no impact when it's turned off
[ https://issues.apache.org/jira/browse/YARN-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009932#comment-15009932 ] Li Lu commented on YARN-4356: - Thanks for the long list [~sjlee0] and [~jrottinghuis]! I'm gradually shifting my cycles back to v2 work right now, and would like to help verifying this for 2.8 release. Any concrete action items I can take? Meanwhile, if we're fine with the current version setting in YARN-4234, shall we push that JIRA forward? > ensure the timeline service v.2 is disabled cleanly and has no impact when > it's turned off > -- > > Key: YARN-4356 > URL: https://issues.apache.org/jira/browse/YARN-4356 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Labels: yarn-2928-1st-milestone > > For us to be able to merge the first milestone drop to trunk, we want to > ensure that once disabled the timeline service v.2 has no impact from the > server side to the client side. If the timeline service is not enabled, no > action should be done. If v.1 is enabled but not v.2, v.1 should behave the > same as it does before the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4356) ensure the timeline service v.2 is disabled cleanly and has no impact when it's turned off
[ https://issues.apache.org/jira/browse/YARN-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15010094#comment-15010094 ] Sangjin Lee commented on YARN-4356: --- I haven't looked at YARN-4234 in detail yet. Are you referring to the version config? One thing we need to resolve early is whether we should support having both v.1.x and v.2 enabled. While I see value in being able to do so, I recognize that it would be a pretty major rework, as the current code is making a strong assumption of this being a binary choice (mutually exclusive). cc [~jrottinghuis] Also, as I was arguing on YARN-4183, if the v.2 timeline service is disabled (whichever config it may be), it would be desirable to disable the v.2 code cleanly and as early as possible so as not to leave impact. It would be basically "wiring off" the feature. Once we have some consensus around it, we can start work on this. > ensure the timeline service v.2 is disabled cleanly and has no impact when > it's turned off > -- > > Key: YARN-4356 > URL: https://issues.apache.org/jira/browse/YARN-4356 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Labels: yarn-2928-1st-milestone > > For us to be able to merge the first milestone drop to trunk, we want to > ensure that once disabled the timeline service v.2 has no impact from the > server side to the client side. If the timeline service is not enabled, no > action should be done. If v.1 is enabled but not v.2, v.1 should behave the > same as it does before the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4356) ensure the timeline service v.2 is disabled cleanly and has no impact when it's turned off
[ https://issues.apache.org/jira/browse/YARN-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15010172#comment-15010172 ] Naganarasimha G R commented on YARN-4356: - Yes [~varun_saxena], i agree that the approach which is taken for enabling ATSv1 and ATSv2 is bit shaky, but we had discussed about supporting version in YARN-3034 and many of initial patches there were based on version but later [~zjshen] had informed in the [comment|https://issues.apache.org/jira/browse/YARN-3034?focusedCommentId=14376575=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14376575] not to create version based configuration and to handle based on old config and new config to identify the old version and the new version. May be assuming that ATSv1 will be replaced by V2 with complete compatability. But definetely it requires overhaul which i plan to do as part of YARN-3623. > ensure the timeline service v.2 is disabled cleanly and has no impact when > it's turned off > -- > > Key: YARN-4356 > URL: https://issues.apache.org/jira/browse/YARN-4356 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Labels: yarn-2928-1st-milestone > > For us to be able to merge the first milestone drop to trunk, we want to > ensure that once disabled the timeline service v.2 has no impact from the > server side to the client side. If the timeline service is not enabled, no > action should be done. If v.1 is enabled but not v.2, v.1 should behave the > same as it does before the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4356) ensure the timeline service v.2 is disabled cleanly and has no impact when it's turned off
[ https://issues.apache.org/jira/browse/YARN-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15004714#comment-15004714 ] Varun Saxena commented on YARN-4356: That's quite a comprehensive list. I was wondering if we should relook at some of our configurations. For instance, in RM for ATSv1, we were publishing system metrics if timeline service was enabled and system metrics publishing was also *enabled* Now we publish to ATSv2 if timeline service is enabled and system metrics publishing is *disabled* This doesn't sound semantically correct to me and can be quite confusing for the end user as well. Maybe we can use the timeline service version config introduced in YARN-4234. > ensure the timeline service v.2 is disabled cleanly and has no impact when > it's turned off > -- > > Key: YARN-4356 > URL: https://issues.apache.org/jira/browse/YARN-4356 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Labels: yarn-2928-1st-milestone > > For us to be able to merge the first milestone drop to trunk, we want to > ensure that once disabled the timeline service v.2 has no impact from the > server side to the client side. If the timeline service is not enabled, no > action should be done. If v.1 is enabled but not v.2, v.1 should behave the > same as it does before the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4356) ensure the timeline service v.2 is disabled cleanly and has no impact when it's turned off
[ https://issues.apache.org/jira/browse/YARN-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15004661#comment-15004661 ] Sangjin Lee commented on YARN-4356: --- This is the rough note I jotted down while going over the changes made in YARN-2928 so far. (high level assessment) - when to enable the timeline service and which version, and which side is not clear - on the server-side, lots of the v.2 code comes into picture even if the timeline service (v.2) is disabled - interestingly, it appears that the ATS v.1 isn't enabled for MR on trunk - additional dependencies (through hbase-client and phoenix) are something to verify (node manager) main features - timelinecollector-to-NM service (NM collector service) - NM-to-RM heartbeat addition for discovery - NM metrics publisher NodeHeartbeatRequestPBImpl - the collectors map is added to the node heartbeat to the RM unconditionally NodeManager - the NM collector service (from timeline collectors to NM) is started unconditionally - the registered collectors map is added to the node manager unconditionally (possible memory risk) ContainerManagerImpl - NM timeline (metrics) publisher is created unconditionally - the event handling for the localization is added unconditionally - flow run id is parsed and passed around unconditionally in creating an application ApplicationImpl - at least creating the timeline client is predicated on systemMetricsPublisherEnabled NMTimelinePublisher - it appears that the intent is to make it no-op if systemMetricsPublisherEnabled is false - but it's not clear at all whether that is really the case (resource manager) main features - additional dependencies - discovery/registry - major refactoring around RM system metrics publisher pom.xml - resourcemanager now depends on timelineservice and pulls its dependencies ApplicationMasterService - collector address is managed unconditionally (timeline service uses RM as registry) ClientRMService - when an application is submitted the flow run id from the yarn tag is verified unconditionally (this could be an issue if some invalid yarn tag is coming over but timeline service is disabled) RMAppManager - the application is added to the RT timeline collector unconditionally - the timeline collector an app is started unconditionally RMTimelineCollectorManager - it gets added unconditionally in ResourceManager ResourceTrackerService - things like updateAppCollectorsMap, keepAliveApps are done unconditionally AMLauncher - sets the flow tags in the environment unconditionally SystemMetricsPublisher and associated event classes - a fairly major refactoring: risk for missing out upstream changes RMAppImpl - collector transitions are added unconditionally (yarn api) YarnConfiguration/yarn-default.xml - we need to review the configs and streamline them if necessary - some phoenix-related configs (perhaps we should remove all phoenix code, dependencies, and config, and introduce them later?) (distributed shell) ApplicationMaster - the thread pool for posting writes is created unconditionally Client - added options for enabling timeline service v.2 and supporting flows (-> yarn tags) (mapreduce client app) JobHistoryEventHandler - the thread pool for posting writes is created unconditionally - whether to use v.2 turns on MAPREDUCE_JOB_NEW_TIMELINE_SERVICE_ENABLED (boolean) MRAppMaster - timeline client is newly enabled either for v.1 or v.2! > ensure the timeline service v.2 is disabled cleanly and has no impact when > it's turned off > -- > > Key: YARN-4356 > URL: https://issues.apache.org/jira/browse/YARN-4356 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Labels: yarn-2928-1st-milestone > > For us to be able to merge the first milestone drop to trunk, we want to > ensure that once disabled the timeline service v.2 has no impact from the > server side to the client side. If the timeline service is not enabled, no > action should be done. If v.1 is enabled but not v.2, v.1 should behave the > same as it does before the merge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
