[
https://issues.apache.org/jira/browse/SPARK-16264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15494587#comment-15494587
]
Jakob Odersky commented on SPARK-16264:
---------------------------------------
I just came across this issue through a comment in the ForeachSink. I
understand why Sinks would be better off by not knowing about the type of
QueryExecution, however I'm not quite sure what you mean by "having something
similar to foreachwriter". Is the idea to have only a single foreach sink and
expose all custom user sinks as foreach writers?
> Allow the user to use operators on the received DataFrame
> ---------------------------------------------------------
>
> Key: SPARK-16264
> URL: https://issues.apache.org/jira/browse/SPARK-16264
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Reporter: Shixiong Zhu
>
> Currently Sink cannot apply any operators on the given DataFrame because new
> DataFrame created by the operator will use QueryExecution rather than
> IncrementalExecution.
> There are two options to fix this one:
> 1. Merge IncrementalExecution into QueryExecution so that QueryExecution can
> also deal with streaming operators.
> 2. Make Dataset operators inherits the QueryExecution(IncrementalExecution is
> just a subclass of IncrementalExecution) from it's parent.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
