[ https://issues.apache.org/jira/browse/SPARK-16264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353660#comment-15353660 ]
Reynold Xin commented on SPARK-16264: ------------------------------------- I actually think the sink interface just shouldn't expose DataFrame, and have something similar to the foreach writer. It would be easier to maintain compatibility in the long run. > Allow the user to use operators on the received DataFrame > --------------------------------------------------------- > > Key: SPARK-16264 > URL: https://issues.apache.org/jira/browse/SPARK-16264 > Project: Spark > Issue Type: Improvement > Components: SQL > Reporter: Shixiong Zhu > > Currently Sink cannot apply any operators on the given DataFrame because new > DataFrame created by the operator will use QueryExecution rather than > IncrementalExecution. > There are two options to fix this one: > 1. Merge IncrementalExecution into QueryExecution so that QueryExecution can > also deal with streaming operators. > 2. Make Dataset operators inherits the QueryExecution(IncrementalExecution is > just a subclass of IncrementalExecution) from it's parent. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org