[
https://issues.apache.org/jira/browse/SPARK-16545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15378811#comment-15378811
]
Mario Briggs edited comment on SPARK-16545 at 7/15/16 3:57 AM:
---------------------------------------------------------------
thanks. I looked into it too and i was not getting a fix that was satisfying to
myself. The problem seems to be that Dataset assumes it has a QueryExecution
with the Physical Plan (which is true in the batch case), since most of the
Listener/metrics gathering functions want to dump this info , whereas in
streaming we want only the 'inner' IncrementalExecution to produce the
PhysicalPlan. I will submit what i have tried to do as well to ease the
discussion points.
was (Author: mariobriggs):
thanks. I looked into it too and i was not getting a fix that was satisfying to
myself. The problem seems to be that Dataset assumes it has a QueryExecution
with the Physical Plan (which is true in the batch case), since most of the
Listener/metrics gathering functions want to dump this info , whereas in
streaming we want only the 'inner' IncrementalExecution to produce the
PhysicalPlan. I will submit have i tried to do as well to ease the discussion
points.
> Structured Streaming : foreachSink creates the Physical Plan multiple times
> per TriggerInterval
> ------------------------------------------------------------------------------------------------
>
> Key: SPARK-16545
> URL: https://issues.apache.org/jira/browse/SPARK-16545
> Project: Spark
> Issue Type: Bug
> Components: SQL, Streaming
> Affects Versions: 2.0.0
> Reporter: Mario Briggs
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
