[ https://issues.apache.org/jira/browse/YARN-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14725845#comment-14725845 ]
Sangjin Lee commented on YARN-3901: ----------------------------------- Yeah, I can basically reproduce this issue too. What you said is largely correct. If you try to read the flow run metrics via {{readResultsWithTimestamps()}}, it will choke on {{GenericObjectMapper.read()}}. This is what I see: {noformat} org.codehaus.jackson.JsonParseException: Illegal character ((CTRL-CHAR, code 0)): only regular white space (\r, \n, \t) is allowed between tokens at [Source: [B@19fcfd96; line: 1, column: 2] at org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1433) at org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:521) at org.codehaus.jackson.impl.JsonParserMinimalBase._throwInvalidSpace(JsonParserMinimalBase.java:467) at org.codehaus.jackson.impl.ReaderBasedParser._skipWSOrEnd(ReaderBasedParser.java:1491) at org.codehaus.jackson.impl.ReaderBasedParser.nextToken(ReaderBasedParser.java:368) at org.codehaus.jackson.map.ObjectReader._initForReading(ObjectReader.java:828) at org.codehaus.jackson.map.ObjectReader._bindAndClose(ObjectReader.java:752) at org.codehaus.jackson.map.ObjectReader.readValue(ObjectReader.java:486) at org.apache.hadoop.yarn.server.timeline.GenericObjectMapper.read(GenericObjectMapper.java:93) at org.apache.hadoop.yarn.server.timeline.GenericObjectMapper.read(GenericObjectMapper.java:77) at org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper.readResultsWithTimestamps(ColumnHelper.java:202) at org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunColumnPrefix.readResultsWithTimestamps(FlowRunColumnPrefix.java:155) at org.apache.hadoop.yarn.server.timelineservice.storage.HBaseTimelineReaderImpl.readMetrics(HBaseTimelineReaderImpl.java:513) at org.apache.hadoop.yarn.server.timelineservice.storage.HBaseTimelineReaderImpl.readFlowRunEntity(HBaseTimelineReaderImpl.java:669) at org.apache.hadoop.yarn.server.timelineservice.storage.HBaseTimelineReaderImpl.getFlowRunEntity(HBaseTimelineReaderImpl.java:625) at org.apache.hadoop.yarn.server.timelineservice.storage.HBaseTimelineReaderImpl.getEntity(HBaseTimelineReaderImpl.java:122) {noformat} Is that what you see, or is it different? At any rate, the data is being written different that the GenericObjectMapper cannot parse it, it seems. > Populate flow run data in the flow_run table > -------------------------------------------- > > Key: YARN-3901 > URL: https://issues.apache.org/jira/browse/YARN-3901 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver > Reporter: Vrushali C > Assignee: Vrushali C > Attachments: YARN-3901-YARN-2928.1.patch, > YARN-3901-YARN-2928.WIP.2.patch, YARN-3901-YARN-2928.WIP.patch > > > As per the schema proposed in YARN-3815 in > https://issues.apache.org/jira/secure/attachment/12743391/hbase-schema-proposal-for-aggregation.pdf > filing jira to track creation and population of data in the flow run table. > Some points that are being considered: > - Stores per flow run information aggregated across applications, flow version > RM’s collector writes to on app creation and app completion > - Per App collector writes to it for metric updates at a slower frequency > than the metric updates to application table > primary key: cluster ! user ! flow ! flow run id > - Only the latest version of flow-level aggregated metrics will be kept, even > if the entity and application level keep a timeseries. > - The running_apps column will be incremented on app creation, and > decremented on app completion. > - For min_start_time the RM writer will simply write a value with the tag for > the applicationId. A coprocessor will return the min value of all written > values. - > - Upon flush and compactions, the min value between all the cells of this > column will be written to the cell without any tag (empty tag) and all the > other cells will be discarded. > - Ditto for the max_end_time, but then the max will be kept. > - Tags are represented as #type:value. The type can be not set (0), or can > indicate running (1) or complete (2). In those cases (for metrics) only > complete app metrics are collapsed on compaction. > - The m! values are aggregated (summed) upon read. Only when applications are > completed (indicated by tag type 2) can the values be collapsed. > - The application ids that have completed and been aggregated into the flow > numbers are retained in a separate column for historical tracking: we don’t > want to re-aggregate for those upon replay > -- This message was sent by Atlassian JIRA (v6.3.4#6332)