rdtr opened a new pull request, #13264:
URL: https://github.com/apache/gluten/pull/13264
## What changes are proposed in this pull request?
This is a revised version of #11724, which was closed as stale after
@FelixYBW's review ("Looks good. Can you fix conflict?"). I found the metrics
transport has since moved to JSON (#12126), so this PR is rebuilt on top of it.
`InputIteratorTransformer` is where data from shuffle, broadcast or a
fallback pipeline enters Velox. Its only time metric is the wall time of
Velox's input operator, which includes waiting for that input. Velox also
tracks `cpuWallTiming.cpuNanos` for every operator, but Gluten did not pass it
on, so you could not tell compute time from waiting time.
This PR:
- Adds `cpuNanos` to the per-operator stats in the metrics JSON
(`WholeStageResultIterator`).
- Reads it into `OperatorMetrics.cpuNanos` and sums it in `MetricsUtil` like
`wallNanos`.
- Shows it on `InputIteratorTransformer` as a new SQL metric, "cpu time of
input".
Other operators don't show the new metric. For most of them, CPU time is
close to wall time. If it helps, the same field can be shown on other operators
in follow-ups.
Related issue: #8063
## How was this patch tested?
Added "Metrics for input iterator of shuffle include cpu time" to
`VeloxMetricsSuite`. It runs an aggregation after a shuffle and checks that the
cpu time and wall time of the shuffle input iterator are recorded.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
