[
https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14155490#comment-14155490
]
Zhijie Shen commented on MAPREDUCE-5933:
----------------------------------------
[~rkanter], thanks for the new patch. Here're my comments about it.
1. It seems that we have associated all the events to a single MR job entity,
which doesn't reflect the nested structure: a MR job contains multiple
map/reduce tasks. IMHO, we should have two level of entities: 1) MR job entity
and 2) MR task entity with two type: Map and Reduce. The job events are added
to the job entity while the task attempts are added to the particular task
entity.
2. For entity and event info key, can we follow the convention that has applied
to SystemMetricsPublisher, distributed shell and tez job: UPPER_CASE?
3. Given neither the scalable timeline server nor the reliable writing is going
to happen in 2.6, shall we get rid of crashing MR AM when TimelineClient throws
the exception, because we expect it's not as reliable as HDFS right now?
{code}
+ tEntity.addEvent(tEvent);
+ try {
+ timelineClient.putEntities(tEntity);
+ } catch (IOException ex) {
+ throw new YarnRuntimeException(ex);
+ } catch (YarnException ex) {
+ throw new YarnRuntimeException(ex);
+ }
{code}
4. It seems not to be necessary to put some events around the summary and the
other in processAdditionalEventsForTimelineServer. We can put all timeline
server event handling in the same method, and invoke it around
{code}
HistoryEvent historyEvent = event.getHistoryEvent();
if (! (historyEvent instanceof NormalizedResourceEvent)) {
mi.writeEvent(historyEvent);
}
{code}
5. IMHO, one option to save the summary information (pulled from some
particular events) as the other info of the job-level entity, while we still
store these events. Later on, we can simply query the job-level entity and only
retrieve the other info fields to quickly get the summary information in JHS.
6. It would have to know how to read them back and then convert them to JSON
for display.
Yeah, my question is whether we'd like to display the counter in JSON or keep
it as unreadable bytes, but it's fine to keep to the concurrent solution.
> Enable MR AM to post history events to the timeline server
> ----------------------------------------------------------
>
> Key: MAPREDUCE-5933
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am
> Reporter: Zhijie Shen
> Assignee: Robert Kanter
> Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
> MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, mr_timelineserver_response.txt
>
>
> Nowadays, MR AM collects the history events and writes it to HDFS for JHS to
> source. With the timeline server, MR AM can put these events there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
