[jira] [Commented] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[ https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14186895#comment-14186895 ] Hudson commented on MAPREDUCE-5933: --- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1915 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1915/]) MAPREDUCE-5933. Enabled MR AM to post history events to the timeline server. Contributed by Robert Kanter. (zjshen: rev 6b2f11b54bc679b0715fe66bd129e340e8c61c5c) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEvent.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMRTimelineEventHandling.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java > Enable MR AM to post history events to the timeline server > -- > > Key: MAPREDUCE-5933 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mr-am >Reporter: Zhijie Shen >Assignee: Robert Kanter > Fix For: 2.6.0 > > Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, > MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, > MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, > MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, mr_timelineserver_response.txt > > > Nowadays, MR AM collects the history events and writes it to HDFS for JHS to > source. With the timeline server, MR AM can put these events there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[ https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14186832#comment-14186832 ] Hudson commented on MAPREDUCE-5933: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1940 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1940/]) MAPREDUCE-5933. Enabled MR AM to post history events to the timeline server. Contributed by Robert Kanter. (zjshen: rev 6b2f11b54bc679b0715fe66bd129e340e8c61c5c) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMRTimelineEventHandling.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEvent.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java * hadoop-mapreduce-project/CHANGES.txt > Enable MR AM to post history events to the timeline server > -- > > Key: MAPREDUCE-5933 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mr-am >Reporter: Zhijie Shen >Assignee: Robert Kanter > Fix For: 2.6.0 > > Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, > MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, > MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, > MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, mr_timelineserver_response.txt > > > Nowadays, MR AM collects the history events and writes it to HDFS for JHS to > source. With the timeline server, MR AM can put these events there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[ https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14186733#comment-14186733 ] Hudson commented on MAPREDUCE-5933: --- SUCCESS: Integrated in Hadoop-Yarn-trunk #726 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/726/]) MAPREDUCE-5933. Enabled MR AM to post history events to the timeline server. Contributed by Robert Kanter. (zjshen: rev 6b2f11b54bc679b0715fe66bd129e340e8c61c5c) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMRTimelineEventHandling.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEvent.java > Enable MR AM to post history events to the timeline server > -- > > Key: MAPREDUCE-5933 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mr-am >Reporter: Zhijie Shen >Assignee: Robert Kanter > Fix For: 2.6.0 > > Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, > MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, > MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, > MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, mr_timelineserver_response.txt > > > Nowadays, MR AM collects the history events and writes it to HDFS for JHS to > source. With the timeline server, MR AM can put these events there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[ https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14186306#comment-14186306 ] Hudson commented on MAPREDUCE-5933: --- FAILURE: Integrated in Hadoop-trunk-Commit #6362 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6362/]) MAPREDUCE-5933. Enabled MR AM to post history events to the timeline server. Contributed by Robert Kanter. (zjshen: rev 6b2f11b54bc679b0715fe66bd129e340e8c61c5c) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEvent.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMRTimelineEventHandling.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml > Enable MR AM to post history events to the timeline server > -- > > Key: MAPREDUCE-5933 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mr-am >Reporter: Zhijie Shen >Assignee: Robert Kanter > Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, > MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, > MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, > MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, mr_timelineserver_response.txt > > > Nowadays, MR AM collects the history events and writes it to HDFS for JHS to > source. With the timeline server, MR AM can put these events there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[ https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14186289#comment-14186289 ] Zhijie Shen commented on MAPREDUCE-5933: Jenkins is broken (HADOOP-11240). It's a slight change since last patch. Will go ahead to commit it. > Enable MR AM to post history events to the timeline server > -- > > Key: MAPREDUCE-5933 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mr-am >Reporter: Zhijie Shen >Assignee: Robert Kanter > Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, > MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, > MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, > MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, mr_timelineserver_response.txt > > > Nowadays, MR AM collects the history events and writes it to HDFS for JHS to > source. With the timeline server, MR AM can put these events there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[
https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14186235#comment-14186235
]
Hadoop QA commented on MAPREDUCE-5933:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12677496/MAPREDUCE-5933.patch
against trunk revision b0e19c9.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4981//console
This message is automatically generated.
> Enable MR AM to post history events to the timeline server
> --
>
> Key: MAPREDUCE-5933
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am
>Reporter: Zhijie Shen
>Assignee: Robert Kanter
> Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, mr_timelineserver_response.txt
>
>
> Nowadays, MR AM collects the history events and writes it to HDFS for JHS to
> source. With the timeline server, MR AM can put these events there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[
https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183637#comment-14183637
]
Hadoop QA commented on MAPREDUCE-5933:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12677025/MAPREDUCE-5933.patch
against trunk revision 0ac6988.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4979//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4979//console
This message is automatically generated.
> Enable MR AM to post history events to the timeline server
> --
>
> Key: MAPREDUCE-5933
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am
>Reporter: Zhijie Shen
>Assignee: Robert Kanter
> Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> MAPREDUCE-5933.patch, mr_timelineserver_response.txt
>
>
> Nowadays, MR AM collects the history events and writes it to HDFS for JHS to
> source. With the timeline server, MR AM can put these events there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[
https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183259#comment-14183259
]
Zhijie Shen commented on MAPREDUCE-5933:
It seems that the following ID object is missed.
{code}
tEvent.addEventInfo("FAILED_ATTEMPT_ID", tfe.getFailedAttemptID());
{code}
> Enable MR AM to post history events to the timeline server
> --
>
> Key: MAPREDUCE-5933
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am
>Reporter: Zhijie Shen
>Assignee: Robert Kanter
> Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> mr_timelineserver_response.txt
>
>
> Nowadays, MR AM collects the history events and writes it to HDFS for JHS to
> source. With the timeline server, MR AM can put these events there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[
https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182316#comment-14182316
]
Hadoop QA commented on MAPREDUCE-5933:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12676742/MAPREDUCE-5933.patch
against trunk revision 670879e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4978//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4978//console
This message is automatically generated.
> Enable MR AM to post history events to the timeline server
> --
>
> Key: MAPREDUCE-5933
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am
>Reporter: Zhijie Shen
>Assignee: Robert Kanter
> Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> mr_timelineserver_response.txt
>
>
> Nowadays, MR AM collects the history events and writes it to HDFS for JHS to
> source. With the timeline server, MR AM can put these events there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[ https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182210#comment-14182210 ] Robert Kanter commented on MAPREDUCE-5933: -- Looks like something funny happened with the tests; I've kicked off another run. > Enable MR AM to post history events to the timeline server > -- > > Key: MAPREDUCE-5933 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mr-am >Reporter: Zhijie Shen >Assignee: Robert Kanter > Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, > MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, > MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, > mr_timelineserver_response.txt > > > Nowadays, MR AM collects the history events and writes it to HDFS for JHS to > source. With the timeline server, MR AM can put these events there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[
https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182143#comment-14182143
]
Hadoop QA commented on MAPREDUCE-5933:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12676742/MAPREDUCE-5933.patch
against trunk revision 828429d.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The test build failed in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4977//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4977//console
This message is automatically generated.
> Enable MR AM to post history events to the timeline server
> --
>
> Key: MAPREDUCE-5933
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am
>Reporter: Zhijie Shen
>Assignee: Robert Kanter
> Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> mr_timelineserver_response.txt
>
>
> Nowadays, MR AM collects the history events and writes it to HDFS for JHS to
> source. With the timeline server, MR AM can put these events there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[
https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182000#comment-14182000
]
Hadoop QA commented on MAPREDUCE-5933:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12676727/MAPREDUCE-5933.patch
against trunk revision 828429d.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4976//console
This message is automatically generated.
> Enable MR AM to post history events to the timeline server
> --
>
> Key: MAPREDUCE-5933
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am
>Reporter: Zhijie Shen
>Assignee: Robert Kanter
> Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, mr_timelineserver_response.txt
>
>
> Nowadays, MR AM collects the history events and writes it to HDFS for JHS to
> source. With the timeline server, MR AM can put these events there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[ https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14180540#comment-14180540 ] Zhijie Shen commented on MAPREDUCE-5933: Tried the patch on the cluster with Kerberos http auth and ssl enabled and it seemed to work fine. > Enable MR AM to post history events to the timeline server > -- > > Key: MAPREDUCE-5933 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mr-am >Reporter: Zhijie Shen >Assignee: Robert Kanter > Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, > MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, > MAPREDUCE-5933.patch, mr_timelineserver_response.txt > > > Nowadays, MR AM collects the history events and writes it to HDFS for JHS to > source. With the timeline server, MR AM can put these events there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[
https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14180283#comment-14180283
]
Zhijie Shen commented on MAPREDUCE-5933:
The latest patch looks much better. There're some minor comments about it.
1. SUBMIT_TIME is set twice
{code}
tEvent.addEventInfo("SUBMIT_TIME", jse.getSubmitTime());
tEvent.addEventInfo("QUEUE_NAME", jse.getJobQueueName());
tEvent.addEventInfo("JOB_NAME", jse.getJobName());
tEvent.addEventInfo("USER_NAME", jse.getUserName());
tEvent.addEventInfo("SUBMIT_TIME", jse.getSubmitTime());
{code}
2. Make ""MAPREDUCE_JOB" and "MAPREDUCE_TASK" constants?
3. Would you please make task attempt Id obj toString? Otherwise, it will
result in a nested structure in json content. Same for the other
getXXXAttemptId() that is not followed by toString().
{code}
tEvent.addEventInfo("SUCCESSFUL_TASK_ATTEMPT_ID",
tfe2.getSuccessfulTaskAttemptId());
{code}
4. In addition to set related entity, it's better to add the job ID to the
primary filter of a MR task entity, such that we can support a common query as
follows:
{code}
http://localhost:8188/ws/v1/timeline/MAPREDUCE_TASK?primaryFilter=PARENT_JOB:job_1413998833197_0001
{code}
In fact, there could be some other optimization to speed up the potential
queries. For example, to answer the query of JHS as follows:
{code}
http://10.22.2.115:19888/jobhistory/attempts/job_1413998833197_0001/m/SUCCESSFUL
{code}
It's good also have task type and task final state been put into the other info
field for in-memory filtering or even put into primary filter field for index
in the store (which is much more expensive store space usage).
I think we should do the store schema optimization according the particular
queries in a separate Jira, as it seems not to be the straightforward addition
to this patch. Let's focus on posting events in this one.
The patch is working properly on a insecure cluster. Will try this patch on a
secure cluster too.
> Enable MR AM to post history events to the timeline server
> --
>
> Key: MAPREDUCE-5933
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am
>Reporter: Zhijie Shen
>Assignee: Robert Kanter
> Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> MAPREDUCE-5933.patch, mr_timelineserver_response.txt
>
>
> Nowadays, MR AM collects the history events and writes it to HDFS for JHS to
> source. With the timeline server, MR AM can put these events there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[
https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164532#comment-14164532
]
Hadoop QA commented on MAPREDUCE-5933:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12673741/MAPREDUCE-5933.patch
against trunk revision a169051.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
org.apache.hadoop.mapred.pipes.TestPipeApplication
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4944//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4944//artifact/patchprocess/patchReleaseAuditProblems.txt
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4944//console
This message is automatically generated.
> Enable MR AM to post history events to the timeline server
> --
>
> Key: MAPREDUCE-5933
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am
>Reporter: Zhijie Shen
>Assignee: Robert Kanter
> Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> MAPREDUCE-5933.patch, mr_timelineserver_response.txt
>
>
> Nowadays, MR AM collects the history events and writes it to HDFS for JHS to
> source. With the timeline server, MR AM can put these events there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[
https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161360#comment-14161360
]
Zhijie Shen commented on MAPREDUCE-5933:
[~rkanter], thanks for being patient with my comments. The latest patch looks
much better. I've applied the patch locally, and run through some MR examples.
Here're some additional comments:
bq. Alternatively, the REST API should be case insensitive
URL should be case-sensitive:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.2.3
bq. Are you aware of any reason for that or if it's just an oversight and I
should fix it?
I'm not aware of it, but it seems that MR job is not generating this event
except the rumen tool.
IMHO, we can do the following instead of using TaskType as the entity type: at
the task level, we only have one entity type - "MAPREDUCE_TASK", and put
TaskType into the other info section if the information is available. The
reason to put them under the same entity type is to make these TASK entities
close to each other, and to be friendly to a range query of the tasks of one
particular job. In fact, it seems that JOB_SETUP, JOB_CLEANUP, TASK_CLEANUP
task event are not generated by the MR job.
1. Make keys be more consistent: "\[_\]COUNTERS" ->
"\[_\]COUNTER_GROUPS", "name" -> "NAME", "displayName" -> "DISPLAY_NAME",
"value" -> "VALUE".
{code}
public JsonNode countersToJSON(Counters counters) {
{code}
2. The following event timestamp may be problematic, the timestamp should
reflect the moment when the event happens instead of when the event is sent to
the timeline server. Timeline server uses this timestamp to sort events. For
example, if start event happens at 1 and is recorded at 4, while finish event
happens at 2 and is recorded 3. We still need to consider start event happens
before end event. It seems that most history events have the associated
timestamp.
{code}
tEvent.setTimestamp(System.currentTimeMillis());
{code}
3. All the entities are put into the default domain, since we haven't specify
one (new feature from YARN-2102). In fact, we should allow MR job to config the
domain where it wants to put the entities, but let's do it in another Jira.
4. In testTimelineEventHandling, would you please set meaningful timestamp for
the events to make them be ordered logically, and check getEntities's response
preserve the order.
5. It's good to have a end-to-end test like NotificationTestCase if it is not
really a big addition (e.g. NotificationTestCase).
> Enable MR AM to post history events to the timeline server
> --
>
> Key: MAPREDUCE-5933
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am
>Reporter: Zhijie Shen
>Assignee: Robert Kanter
> Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> mr_timelineserver_response.txt
>
>
> Nowadays, MR AM collects the history events and writes it to HDFS for JHS to
> source. With the timeline server, MR AM can put these events there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[
https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14158776#comment-14158776
]
Hadoop QA commented on MAPREDUCE-5933:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12672879/MAPREDUCE-5933.patch
against trunk revision 7f6ed7f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4930//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4930//console
This message is automatically generated.
> Enable MR AM to post history events to the timeline server
> --
>
> Key: MAPREDUCE-5933
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am
>Reporter: Zhijie Shen
>Assignee: Robert Kanter
> Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> mr_timelineserver_response.txt
>
>
> Nowadays, MR AM collects the history events and writes it to HDFS for JHS to
> source. With the timeline server, MR AM can put these events there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[ https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14157543#comment-14157543 ] Zhijie Shen commented on MAPREDUCE-5933: bq. When I do this, should I add the task entities as related entities to the job entity (and/or the other way around)? Otherwise, they won't be connected because the entities cannot be nested. Also, does being a related entity do anything? I think we can add the relationship: task -> entity. In SystemMetricsPublisher, I didn't do this, but put app entity ID into the attempt entity primary filter because I know I'm going to list the attempts given an appId. For job entity and task entity, we do the similar thing, or wait until we are clear about the queries. > Enable MR AM to post history events to the timeline server > -- > > Key: MAPREDUCE-5933 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mr-am >Reporter: Zhijie Shen >Assignee: Robert Kanter > Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, > MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, mr_timelineserver_response.txt > > > Nowadays, MR AM collects the history events and writes it to HDFS for JHS to > source. With the timeline server, MR AM can put these events there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[
https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14157506#comment-14157506
]
Robert Kanter commented on MAPREDUCE-5933:
--
{quote}1. It seems that we have associated all the events to a single MR job
entity, which doesn't reflect the nested structure: a MR job contains multiple
map/reduce tasks. IMHO, we should have two level of entities: 1) MR job entity
and 2) MR task entity with two type: Map and Reduce. The job events are added
to the job entity while the task attempts are added to the particular task
entity.{quote}
When I do this, should I add the task entities as related entities to the job
entity (and/or the other way around)? Otherwise, they won't be connected
because the entities cannot be nested. Also, does being a related entity _do_
anything?
> Enable MR AM to post history events to the timeline server
> --
>
> Key: MAPREDUCE-5933
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am
>Reporter: Zhijie Shen
>Assignee: Robert Kanter
> Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, mr_timelineserver_response.txt
>
>
> Nowadays, MR AM collects the history events and writes it to HDFS for JHS to
> source. With the timeline server, MR AM can put these events there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[
https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14155490#comment-14155490
]
Zhijie Shen commented on MAPREDUCE-5933:
[~rkanter], thanks for the new patch. Here're my comments about it.
1. It seems that we have associated all the events to a single MR job entity,
which doesn't reflect the nested structure: a MR job contains multiple
map/reduce tasks. IMHO, we should have two level of entities: 1) MR job entity
and 2) MR task entity with two type: Map and Reduce. The job events are added
to the job entity while the task attempts are added to the particular task
entity.
2. For entity and event info key, can we follow the convention that has applied
to SystemMetricsPublisher, distributed shell and tez job: UPPER_CASE?
3. Given neither the scalable timeline server nor the reliable writing is going
to happen in 2.6, shall we get rid of crashing MR AM when TimelineClient throws
the exception, because we expect it's not as reliable as HDFS right now?
{code}
+ tEntity.addEvent(tEvent);
+ try {
+timelineClient.putEntities(tEntity);
+ } catch (IOException ex) {
+throw new YarnRuntimeException(ex);
+ } catch (YarnException ex) {
+throw new YarnRuntimeException(ex);
+ }
{code}
4. It seems not to be necessary to put some events around the summary and the
other in processAdditionalEventsForTimelineServer. We can put all timeline
server event handling in the same method, and invoke it around
{code}
HistoryEvent historyEvent = event.getHistoryEvent();
if (! (historyEvent instanceof NormalizedResourceEvent)) {
mi.writeEvent(historyEvent);
}
{code}
5. IMHO, one option to save the summary information (pulled from some
particular events) as the other info of the job-level entity, while we still
store these events. Later on, we can simply query the job-level entity and only
retrieve the other info fields to quickly get the summary information in JHS.
6. It would have to know how to read them back and then convert them to JSON
for display.
Yeah, my question is whether we'd like to display the counter in JSON or keep
it as unreadable bytes, but it's fine to keep to the concurrent solution.
> Enable MR AM to post history events to the timeline server
> --
>
> Key: MAPREDUCE-5933
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am
>Reporter: Zhijie Shen
>Assignee: Robert Kanter
> Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, mr_timelineserver_response.txt
>
>
> Nowadays, MR AM collects the history events and writes it to HDFS for JHS to
> source. With the timeline server, MR AM can put these events there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[
https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14154233#comment-14154233
]
Hadoop QA commented on MAPREDUCE-5933:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12672181/MAPREDUCE-5933.patch
against trunk revision 14d60da.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1310 javac
compiler warnings (more than the trunk's current 1264 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS
org.apache.hadoop.hdfs.web.TestWebHDFS
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4925//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4925//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Javac warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4925//artifact/patchprocess/diffJavacWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4925//console
This message is automatically generated.
> Enable MR AM to post history events to the timeline server
> --
>
> Key: MAPREDUCE-5933
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am
>Reporter: Zhijie Shen
>Assignee: Robert Kanter
> Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, mr_timelineserver_response.txt
>
>
> Nowadays, MR AM collects the history events and writes it to HDFS for JHS to
> source. With the timeline server, MR AM can put these events there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[
https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14149720#comment-14149720
]
Zhijie Shen commented on MAPREDUCE-5933:
bq. 1. TimelineClient should do nothing in the case where the ATS is not
enabled. Or did you mean that users may want to disable posting to the ATS for
a specific job, even when the ATS is enabled?
We can either do per-frame work or per-job config. Depending on YARN's
system-wide config makes all kinds of applications enable or disable the client
simultaneously. There's another ticket: MAPREDUCE-6018. We can handle it
separately.
bq. 2. I think we should revisit this after YARN-2517 is done. No reason to
duplicate work here.
Make sense.
bq. 3. The current behavior is to throw a YarnRuntimeException when there's a
problem processing the events for the JHS,
That's true. Let's keep to it.
bq. 4. I think we want to publish each event as it comes in. This should make
the Job available in the ATS sooner than it is in the JHS.
Good point. If the job has a lot of events, we may not need to wait until job
finish to flush all of them, but we can periodically flush the outstanding
events. Again, this may be handled by YARN-2517. I did some investigation: it
seems that we only record the final counters, but not the realtime ones, which
should be much traffic. I'm not sure if it exists other high frequent events.
Any idea?
bq. 5. The countersToJSON method already turns the Counters into JSON so the
ATS can display them.
Yeah, my question is whether we really want to make counters' details
visualization-friendly via timeline read APIs, while the JHS will query this
data, reconstruct counter objects and show them there. If we don't, the easiest
way is to write counters into bytes when storing, and recover them from bytes
when retrieving in.
bq. 6. As before, I was trying to mirror what has been done for the JHS. I'm
publishing to the ATS the same events and information that the JHS gets.
It seems that you published all the events related to job summary, which may
not be enough. Before processing the summary, all events except
NormalizedResourceEvent is going to be written into a job history file.
{code}
HistoryEvent historyEvent = event.getHistoryEvent();
if (! (historyEvent instanceof NormalizedResourceEvent)) {
mi.writeEvent(historyEvent);
}
processEventForJobSummary(event.getHistoryEvent(), mi.getJobSummary(),
{code}
You can take a look at HistoryFileManager and JobHistoryParser. It seems that
while the events in a job history file are used to recover a job's history
info, while a summary file is going to be used to log a job summary at JHS.
JobHistoryParser may tell you all the events that JHS wants to use.
> Enable MR AM to post history events to the timeline server
> --
>
> Key: MAPREDUCE-5933
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am
>Reporter: Zhijie Shen
>Assignee: Robert Kanter
> Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> MAPREDUCE-5933.patch, mr_timelineserver_response.txt
>
>
> Nowadays, MR AM collects the history events and writes it to HDFS for JHS to
> source. With the timeline server, MR AM can put these events there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[ https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14143949#comment-14143949 ] Robert Kanter commented on MAPREDUCE-5933: -- Thanks for taking a look [~zjshen]. Here are my responses: 1. TimelineClient should do nothing in the case where the ATS is not enabled. Or did you mean that users may want to disable posting to the ATS for a specific job, even when the ATS is enabled? 2. I think we should revisit this after YARN-2517 is done. No reason to duplicate work here. 3. The current behavior is to throw a YarnRuntimeException when there's a problem processing the events for the JHS, so I was just following that for the TimelineClient. Unless you think we should do it differently for the TimelineClient and log a warning instead? 4. I think we want to publish each event as it comes in. This should make the Job available in the ATS sooner than it is in the JHS. And in the case of a failure where the ATS has partial information, the ATS should be able to handle partial job info (obviously it probably can't display every piece of information about a job and/or some info might be inaccurate, but at least there's a record of the job); the JHS doesn't even know about many of the jobs that fail part way through. 5. The countersToJSON method already turns the Counters into JSON so the ATS can display them. 6. As before, I was trying to mirror what has been done for the JHS. I'm publishing to the ATS the same events and information that the JHS gets. The JHS doesn't get the TaskFailedEvent, so I didn't do it here. Should publish every type of event? > Enable MR AM to post history events to the timeline server > -- > > Key: MAPREDUCE-5933 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mr-am >Reporter: Zhijie Shen >Assignee: Robert Kanter > Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, > MAPREDUCE-5933.patch, mr_timelineserver_response.txt > > > Nowadays, MR AM collects the history events and writes it to HDFS for JHS to > source. With the timeline server, MR AM can put these events there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[ https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14123658#comment-14123658 ] Zhijie Shen commented on MAPREDUCE-5933: Hi [~rkanter], thanks for taking care of this work. I just scanned through the patch. The approach looks fine to me in general. Here're some comments: 1. Should we make publishing MR job information to the timeline server be configurable. I mean the specific config for MR job. 2. Is it better to do the put entity operation on the a separate thread? JobHistoryHandler seems to run on the main dispatcher thread. We'd better not block other event processing. 3. When error happens to MR job information publishing, IMHO, it shouldn't fail the MR job. The arguable point is that MR job relies on job history to determine the final status, and even for recovery. Once we rebase the job history server based on the data in the timeline store, we should somehow know the job history is (partly) missing or corrupted. 4. Another choice is that you can buffer the events locally and push it once when the job is done/committed. This is a more conservative way as we did for the job history file. OTOH, publishing the event immediately may provide realtime/near-realtime job monitoring. We may want to think more about the choice here. For example, if the job crashes in the middle, the timeline server is going to have a partial history for a MR job. 5. If we want to visualize the counter details in JSON output, it's good build nested JSON data structure. Another efficient way is to use Writable interface to ser/deser the counter into/from bytes. 6. Are all HistoryEvent subclasses properly handled? I randomly searched for TaskFailedEvent, which seemed not to be dealt with. It seems that some of them are only used for rumen, such as JobStatusChangedEvent and TaskUpdatedEvent. These update events may be problematic if they're very frequent. > Enable MR AM to post history events to the timeline server > -- > > Key: MAPREDUCE-5933 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mr-am >Reporter: Zhijie Shen >Assignee: Robert Kanter > Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, > MAPREDUCE-5933.patch, mr_timelineserver_response.txt > > > Nowadays, MR AM collects the history events and writes it to HDFS for JHS to > source. With the timeline server, MR AM can put these events there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[ https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14110080#comment-14110080 ] Robert Kanter commented on MAPREDUCE-5933: -- I don't see any actual test failures on Jenkins, and I ran all of the tests under hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app locally without failures. Perhaps this is related to the svn-to-git migration? > Enable MR AM to post history events to the timeline server > -- > > Key: MAPREDUCE-5933 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mr-am >Reporter: Zhijie Shen >Assignee: Robert Kanter > Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, > MAPREDUCE-5933.patch, mr_timelineserver_response.txt > > > Nowadays, MR AM collects the history events and writes it to HDFS for JHS to > source. With the timeline server, MR AM can put these events there. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[
https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14110067#comment-14110067
]
Hadoop QA commented on MAPREDUCE-5933:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12664258/MAPREDUCE-5933.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The test build failed in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4822//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4822//console
This message is automatically generated.
> Enable MR AM to post history events to the timeline server
> --
>
> Key: MAPREDUCE-5933
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am
>Reporter: Zhijie Shen
>Assignee: Robert Kanter
> Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> MAPREDUCE-5933.patch, mr_timelineserver_response.txt
>
>
> Nowadays, MR AM collects the history events and writes it to HDFS for JHS to
> source. With the timeline server, MR AM can put these events there.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[
https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14107736#comment-14107736
]
Robert Kanter commented on MAPREDUCE-5933:
--
I'll fix the javac warnings and look into the test timeout next week (the
release audit is again not from this patch)
{quote}Robert Kanter, Appoligies for not responding earlier to this JIRA. I've
posted my design requests to MAPREDUCE-5858 to make sure existing can be kept
happy.{quote}
My patch doesn't affect the JHS in any way. It simply makes the AM push the
events to the ATS in addition to the file that the JHS gets; and if the ATS is
disabled, there should be no affect. So I think this meets your requests.
> Enable MR AM to post history events to the timeline server
> --
>
> Key: MAPREDUCE-5933
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am
>Reporter: Zhijie Shen
>Assignee: Robert Kanter
> Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> mr_timelineserver_response.txt
>
>
> Nowadays, MR AM collects the history events and writes it to HDFS for JHS to
> source. With the timeline server, MR AM can put these events there.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[
https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14106329#comment-14106329
]
Hadoop QA commented on MAPREDUCE-5933:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12663553/MAPREDUCE-5933.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1443 javac
compiler warnings (more than the trunk's current 1261 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 3
release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app:
org.apache.hadoop.mapreduce.v2.app.job.impl.TestMapReduceChildJVM
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4815//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4815//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Javac warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4815//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4815//console
This message is automatically generated.
> Enable MR AM to post history events to the timeline server
> --
>
> Key: MAPREDUCE-5933
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am
>Reporter: Zhijie Shen
>Assignee: Robert Kanter
> Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> mr_timelineserver_response.txt
>
>
> Nowadays, MR AM collects the history events and writes it to HDFS for JHS to
> source. With the timeline server, MR AM can put these events there.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[ https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14106289#comment-14106289 ] Jonathan Eagles commented on MAPREDUCE-5933: [~rkanter], Appoligies for not responding earlier to this JIRA. I've posted my design requests to MAPREDUCE-5858 to make sure existing can be kept happy. https://issues.apache.org/jira/browse/MAPREDUCE-5858?focusedCommentId=14085166&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14085166 > Enable MR AM to post history events to the timeline server > -- > > Key: MAPREDUCE-5933 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mr-am >Reporter: Zhijie Shen >Assignee: Robert Kanter > Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, > mr_timelineserver_response.txt > > > Nowadays, MR AM collects the history events and writes it to HDFS for JHS to > source. With the timeline server, MR AM can put these events there. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[
https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14101878#comment-14101878
]
Hadoop QA commented on MAPREDUCE-5933:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12662616/MAPREDUCE-5933.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1441 javac
compiler warnings (more than the trunk's current 1259 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app:
org.apache.hadoop.mapreduce.jobhistory.TestJobHistoryEventHandler
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4809//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4809//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4809//console
This message is automatically generated.
> Enable MR AM to post history events to the timeline server
> --
>
> Key: MAPREDUCE-5933
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am
>Reporter: Zhijie Shen
>Assignee: Robert Kanter
> Attachments: MAPREDUCE-5933.patch, mr_timelineserver_response.txt
>
>
> Nowadays, MR AM collects the history events and writes it to HDFS for JHS to
> source. With the timeline server, MR AM can put these events there.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
