[ https://issues.apache.org/jira/browse/SPARK-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16491281#comment-16491281 ]
Arun Mahadevan commented on SPARK-23887: ---------------------------------------- We could probably invoke "ProgressReporter.finishTrigger" after each epoch. This would update the query execution stats and post StreamingQueryListener events. However I am not clear on how the SQL metrics could be updated since it relies on accumulator and the accumulators might not be updated unless the task completes. We could probably post some special event to the DAGScheduler to update the accumulators or need to figure out some other mechanisms. [~joseph.torres] [~tdas] , what do you think? > update query progress > --------------------- > > Key: SPARK-23887 > URL: https://issues.apache.org/jira/browse/SPARK-23887 > Project: Spark > Issue Type: Sub-task > Components: Structured Streaming > Affects Versions: 2.4.0 > Reporter: Jose Torres > Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org