[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15265023#comment-15265023 ]
Li Lu commented on MAPREDUCE-6424: ---------------------------------- Hmm I think I found the root cause of the UT failure on testMRNewTimelineServiceEventHandling. Note that in the unit test we're directly reading the entity file from the file system. This is different to the implementation of file system based timeline reader, where we merge all updates to the same entity before we return. With the newly added aggregation logic, there are aggregation updates to the application entity. Therefore, the processing logic in verifyMetricsWhenEvent needs to handle this kind of entities (right now it's an empty update). Maybe we'd like to use the file system timeline reader to do this, since there we have a more comprehensive reader implementation? Right now the logic in verifyMetricsWhenEvent seems to be a little bit fragile to future changes? There are certainly improvements we can do on the aggregation side. Let's address that after the first milestone. > Store MR counters as timeline metrics instead of event > ------------------------------------------------------ > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Reporter: Junping Du > Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch, > MAPREDUCE-6424-YARN-2928.v1.002.patch, MAPREDUCE-6424-YARN-2928.v1.003.patch, > MAPREDUCE-6424-YARN-2928.v1.004.patch > > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org