[ https://issues.apache.org/jira/browse/SPARK-20703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16010021#comment-16010021 ]
Liang-Chi Hsieh commented on SPARK-20703: ----------------------------------------- [~tejasp] * It is a physical plan. Currently it is only used to track metrics. * For these metrics, if the writing logic is implemented in Spark, e.g., FileFormatWriter for file-based relations, it is relatively easy. For the writing logic delegated to datasource APIs, e.g., InsertableRelation.insert, CreatableRelationProvider.createRelation, as we can't expect how the API is implemented, we may not be able to track writing action and get/set such metrics. > Add an operator for writing data out > ------------------------------------ > > Key: SPARK-20703 > URL: https://issues.apache.org/jira/browse/SPARK-20703 > Project: Spark > Issue Type: New Feature > Components: SQL > Affects Versions: 2.2.0 > Reporter: Reynold Xin > > We should add an operator for writing data out. Right now in the explain plan > / UI there is no way to tell whether a query is writing data out, and also > there is no way to associate metrics with data writes. It'd be tremendously > valuable to do this for adding metrics and for visibility. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org