[
https://issues.apache.org/jira/browse/SPARK-20703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16010021#comment-16010021
]
Liang-Chi Hsieh edited comment on SPARK-20703 at 5/15/17 2:21 PM:
------------------------------------------------------------------
[~tejasp]
* It is a physical plan. Currently it is only used to track metrics. I think it
can be used for defining partitioning + sorting requirements of the child nodes
in later work.
* For these metrics, if the writing logic is implemented in Spark, e.g.,
FileFormatWriter for file-based relations, it is relatively easy. For the
writing logic delegated to datasource APIs, e.g., InsertableRelation.insert,
CreatableRelationProvider.createRelation, as we can't expect how the API is
implemented, we may not be able to track writing action and get/set such
metrics.
was (Author: viirya):
[~tejasp]
* It is a physical plan. Currently it is only used to track metrics.
* For these metrics, if the writing logic is implemented in Spark, e.g.,
FileFormatWriter for file-based relations, it is relatively easy. For the
writing logic delegated to datasource APIs, e.g., InsertableRelation.insert,
CreatableRelationProvider.createRelation, as we can't expect how the API is
implemented, we may not be able to track writing action and get/set such
metrics.
> Add an operator for writing data out
> ------------------------------------
>
> Key: SPARK-20703
> URL: https://issues.apache.org/jira/browse/SPARK-20703
> Project: Spark
> Issue Type: New Feature
> Components: SQL
> Affects Versions: 2.2.0
> Reporter: Reynold Xin
>
> We should add an operator for writing data out. Right now in the explain plan
> / UI there is no way to tell whether a query is writing data out, and also
> there is no way to associate metrics with data writes. It'd be tremendously
> valuable to do this for adding metrics and for visibility.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
