[jira] [Commented] (SPARK-23887) update query progress
[
https://issues.apache.org/jira/browse/SPARK-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17758145#comment-17758145
]
Bryan Qiang commented on SPARK-23887:
-
Hello folks, I'm wondering what's the final decision on this. Because
{{ContinuousExecution}} not calling {{finishTrigger}} so streaming metrics are
not updated in continuous structured streaming. Thank you very much!
> update query progress
> -
>
> Key: SPARK-23887
> URL: https://issues.apache.org/jira/browse/SPARK-23887
> Project: Spark
> Issue Type: Sub-task
> Components: Structured Streaming
>Affects Versions: 2.4.0
>Reporter: Jose Torres
>Priority: Major
> Labels: bulk-closed
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23887) update query progress
[ https://issues.apache.org/jira/browse/SPARK-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16491281#comment-16491281 ] Arun Mahadevan commented on SPARK-23887: We could probably invoke "ProgressReporter.finishTrigger" after each epoch. This would update the query execution stats and post StreamingQueryListener events. However I am not clear on how the SQL metrics could be updated since it relies on accumulator and the accumulators might not be updated unless the task completes. We could probably post some special event to the DAGScheduler to update the accumulators or need to figure out some other mechanisms. [~joseph.torres] [~tdas] , what do you think? > update query progress > - > > Key: SPARK-23887 > URL: https://issues.apache.org/jira/browse/SPARK-23887 > Project: Spark > Issue Type: Sub-task > Components: Structured Streaming >Affects Versions: 2.4.0 >Reporter: Jose Torres >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
