HeartSaVioR commented on a change in pull request #28040: [SPARK-31278][SS] Fix
StreamingQuery output rows metric
URL: https://github.com/apache/spark/pull/28040#discussion_r399723489
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ProgressReporter.scala
##########
@@ -189,14 +195,14 @@ trait ProgressReporter extends Logging {
sink = sinkProgress,
observedMetrics = new java.util.HashMap(observedMetrics.asJava))
- if (hasNewData) {
- // Reset noDataEventTimestamp if we processed any data
- lastNoDataProgressEventTime = Long.MinValue
+ if (hasExecuted) {
+ // Reset lastProgressEventTime if we processed any data
+ lastProgressEventTime = triggerClock.getTimeMillis()
updateProgress(newProgress)
Review comment:
As we changed the semantic of the variable, I think we can update it in
`updateProgress` to ensure the variable is updated whenever progress update is
happening.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
