[ https://issues.apache.org/jira/browse/HIVE-12205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088795#comment-15088795 ]
Chengxiang Li commented on HIVE-12205: -------------------------------------- [~chinnalalam], thanks working on this. In your patch, the statistic aggregation is still computed separately in different methods(although in same class now) for {{LocalSparkJobStatus}} and {{RemoteSparkJobStatus}}, i suggest you can add a initialize method in {{MetrisCollection}} with parameter {{String jobId, Map<String, List<TaskMetrics>> jobMetrics}}, so that {{LocalSparkJobStatus}} can reuse {{MetricsCollection}} to aggregate statistics as well. What do you think? Besides, could you create a ticket on RB for this? > Spark: unify spark statististics aggregation between local and remote spark > client > ---------------------------------------------------------------------------------- > > Key: HIVE-12205 > URL: https://issues.apache.org/jira/browse/HIVE-12205 > Project: Hive > Issue Type: Task > Components: Spark > Affects Versions: 1.1.0 > Reporter: Xuefu Zhang > Assignee: Chinna Rao Lalam > Attachments: HIVE-12205.1.patch > > > In class {{LocalSparkJobStatus}} and {{RemoteSparkJobStatus}}, spark > statistics aggregation are done similar but in different code paths. Ideally, > we should have a unified approach to simply maintenance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)