aokolnychyi commented on code in PR #55576:
URL: https://github.com/apache/spark/pull/55576#discussion_r3157659936
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala:
##########
@@ -350,9 +350,14 @@ case class ReplaceDataExec(
// One of the metrics couldn't be found, also mark numDeletedRows as
not found.
-1L
}
+
+ // SQLMetric.set call is a no-op if value is -1. Override numDeletedRows
value in summary.
Review Comment:
```
class SQLMetric(
val metricType: String,
initValue: Long = 0L) extends AccumulatorV2[Long, Long] {
// initValue defines the initial value of the metric. 0 is the lowest
value considered valid.
// If a SQLMetric is invalid, it is set to 0 upon receiving any updates,
and it also reports
// 0 as its value to avoid exposing it to the user programmatically.
//
// For many SQLMetrics, we use initValue = -1 to indicate that the metric
is by default invalid.
// At the end of a task, we will update the metric making it valid, and
the invalid metrics will
// be filtered out when calculating min, max, etc. as a workaround
// for SPARK-11013.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
