[ https://issues.apache.org/jira/browse/HIVE-18690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16504933#comment-16504933 ]
Aihua Xu commented on HIVE-18690: --------------------------------- [~stakiar] The patch looks great. Can you check the style errors above? Also, one more question: would "file not visible" the only cause for the exception in updateSparkBytesWrittenMetrics? If not, maybe we can change the message to {{log.debug("Unable to collect file stats for file:" + path + ". Output metrics may be inaccurate", e);}} > Integrate with Spark OutputMetrics > ---------------------------------- > > Key: HIVE-18690 > URL: https://issues.apache.org/jira/browse/HIVE-18690 > Project: Hive > Issue Type: Sub-task > Components: Spark > Reporter: Sahil Takiar > Assignee: Sahil Takiar > Priority: Major > Attachments: HIVE-18690.1.patch, HIVE-18690.2.patch, > HIVE-18690.3.patch, HIVE-18690.4.patch, HIVE-18690.5.patch > > > Spark has an {{OutputMetrics}} it uses to expose records / bytes written. We > currently don't integrate with it and the Spark UI shows a blank value for > output records / bytes. We have our own customer accumulators instead (like > {{HIVE_RECORDS_OUT}}). > Spark exposes the {{OutputMetrics}} object inside individual tasks via the > {{TaskContext.get()}} method. We can use this method to access the > {{OutputMetrics}} object and update it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)