luis4a0 commented on code in PR #12114:
URL: https://github.com/apache/gluten/pull/12114#discussion_r3265097543
##########
backends-velox/src/main/scala/org/apache/spark/shuffle/ColumnarShuffleWriter.scala:
##########
@@ -258,6 +258,18 @@ class ColumnarShuffleWriter[K, V](
dep.metrics("dataSize").add(splitResult.getRawPartitionLengths.sum)
dep.metrics("compressTime").add(splitResult.getTotalCompressTime)
dep.metrics("peakBytes").add(splitResult.getPeakBytes)
+ // Backend-specific custom metrics (see
GlutenSplitResult.getCustomMetrics).
+ // Only entries whose key is pre-registered in `dep.metrics` are recorded;
+ // unknown keys are silently dropped so that the native side can ship new
+ // metrics ahead of the Spark-side `VeloxMetricsApi` registration without
+ // breaking older builds.
+ splitResult.getCustomMetrics.forEach {
+ (key, value) =>
+ val m = dep.metrics.get(key)
+ if (m.isDefined) {
+ m.get.add(value)
Review Comment:
Good catch — fully agreed about the parity gap across the 3 writer entry
points. This particular concern is now moot because, per offline reviewer
feedback, we dropped the second commit entirely (the one that wired
`getCustomMetrics()` into `ColumnarShuffleWriter.scala`). The Scala consumer
side will come back as small follow-up PRs once specific metrics have proven
useful, and your `applyCustomMetrics(splitResult, dep)` helper is exactly the
right shape for the first such follow-up — I'll factor it that way to keep all
3 writers in sync from day one.
Will add a TODO note next to `getCustomMetrics()` in `GlutenSplitResult` so
the helper-shape requirement is captured for whoever picks up the first
per-metric follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
