Diveyam-Mishra commented on code in PR #58476:
URL: https://github.com/apache/spark/pull/58476#discussion_r3937590418
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/statefulOperators.scala:
##########
@@ -230,21 +228,15 @@ trait StateStoreWriter
}
/**
- * Map of all instance metrics (including partition ID and store names) to
- * their SQLMetric counterpart.
- *
- * The instance metric objects hold additional information on how to report
these metrics,
- * while the SQLMetric objects store the metric values.
- *
- * This map is similar to the metrics map, but needs to be kept separate to
prevent propagating
- * all initialized instance metrics to SparkUI.
+ * Aggregator used for executors to pass instance metrics (per
partition/store) back to driver.
*/
- lazy val instanceMetrics: Map[StateStoreInstanceMetric, SQLMetric] =
- stateStoreInstanceMetrics
+ val instanceMetricsAccumulator:
CollectionAccumulator[(StateStoreInstanceMetric, Long)] = {
Review Comment:
So I can make `byPartition` protected and create a
`StateStoreInstanceMetricAccumulator` subclass that can access the partition
map and apply `StateStoreInstanceMetric.combine(...)` instead of inheriting the
default last-write-wins behavior of `PartitionKeyedAccumulator`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
