dwsmith1983 commented on code in PR #5880:
URL: https://github.com/apache/datafusion-comet/pull/5880#discussion_r4081303812
##########
spark/src/main/scala/org/apache/spark/sql/comet/CometMetricNode.scala:
##########
@@ -105,24 +101,34 @@ case class CometMetricNode(metrics: Map[String,
SQLMetric], children: Seq[CometM
})
/**
- * Reports aggregated scan input metrics (bytesRead, recordsRead) to Spark's
task metrics.
- * Aggregates across all scan leaf nodes to handle plans with multiple scans
(e.g., joins). Must
- * be called in a TaskCompletionListener after the iterator is fully
consumed.
+ * Reports the scan leaves' bytes and rows (summed across joins and unions)
to Spark's task
+ * input metrics, which drive the Input column on the UI's Stages and
Executors tabs.
+ *
+ * Must be registered on the task thread before
[[org.apache.comet.CometExecIterator]] so its
+ * completion listener publishes final SQL metrics before this listener
runs. A block with a JVM
+ * input only publishes on the metrics update interval, and a consumer that
stops early, such as
+ * a limit, leaves the final publish to that close.
+ *
+ * Adds to the task's counters instead of replacing them, so bytes that a
fallback Spark scan
Review Comment:
> Could you carry a short version of the caveat up here?
In df1512682. The scaladoc now says the survival holds only when the
fallback scan registers its completion listener after this one, which a
`CometSparkToColumnarExec` input always does and a coalesced Spark-scan
partition computed first does not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
