ulysses-you opened a new pull request, #58591:
URL: https://github.com/apache/spark/pull/58591
### What changes were proposed in this pull request?
`GroupPartitionsExec` computes the whole partition alignment on the driver
but reported
nothing. This PR adds seven driver-side metrics, set in
`doExecute`/`doExecuteColumnar` and
posted through `SQLMetrics.postDriverMetricsUpdatedByValue`, following
`AQEShuffleReadExec.sendDriverMetrics`:
| metric | UI description | registered |
|---|---|---|
| `numInputPartitions` | number of input partitions | always |
| `numPartitions` | number of partitions (output) | always |
| `numEmptyPartitions` | number of empty partitions | always |
| `numCoalescedPartitions` | number of coalesced partitions | always |
| `maxPartitionsPerGroup` | max partitions per group | always |
| `numPrunedPartitions` | number of pruned input partitions | alignment path
|
| `numReplicatedPartitions` | number of replicated input partition reads |
align-replicate path |
Semantics, all in input partition reads against the baseline of every input
being read once:
- pruned: inputs of a key the alignment never references. The join proved
such keys cannot
produce output, e.g. an inner join keeps only the intersection
(`mergeAndDedupPartitions`
with `spark.sql.sources.v2.bucketing.partitionFilter.enabled`).
- replicated: reads beyond the first pass. In the replicate mode of partial
clustering every
expected partition of a held key re-reads all its splits, so a key with 3
splits over 2
slots counts 3. The two close the accounting: total input reads =
`numInputPartitions` -
pruned + replicated, which is what explains an inflated scan row count.
- coalesced counts output groups merging more than one input, so a
replicated copy of a
multi-split group is counted by both, by design (one says "merges inputs",
the other says
"is a redundant re-read").
The pruned and replicated counts are accumulated in the existing single pass
of
`alignToExpectedKeys` (pruned as total minus matched, relying on the
producers
deduplicating the expected keys). Registration reads constructor parameters
only, so
rendering a plan never forces the grouping. An `assert(numSplits > 0)` pins
the expected-key
split-count contract the derivation relies on.
No plan shape, result, or explain/description change; the metrics are
additive.
### Why are the changes needed?
The alignment decisions this node makes are invisible today:
- partial clustering replicates a side's key groups, re-reading data; the
only symptom is an
inflated scan `numOutputRows` with nothing to explain the multiplier;
- an inner join's intersection prunes whole input partitions at runtime;
nothing reports how
much of a side was skipped;
- distribute-mode padding, coalescing, and per-key group skew (a key holding
many splits
becomes one big task) have no counters either.
These are exactly the numbers needed to tell whether an SPJ alignment saved
work or
amplified it.
### Does this PR introduce _any_ user-facing change?
Yes, additive: the SQL UI shows the metrics above on `GroupPartitions` nodes
of
storage-partitioned join plans (and single-child grouping plans). No
behavior, plan, or
result change compared to released versions.
### How was this patch tested?
New tests in `GroupPartitionsExecSuite`:
- unit: plain-grouping counts; zero coalesced without duplicate keys;
distribute alignment
padding; intersection pruning plus missing-key padding (replicated stays 0
for an empty
group); replicate alignment read counts, including the
coalesced/replicated overlap;
- end to end through the SQL status store (`planGraph` +
`executionMetrics`): an inner join
whose sides hold different keys prunes both sides, run with AQE on and
off; a partial
clustering join where the smaller side replicates (`replicated = (3-1)*2 +
(3-1)*1 = 6`,
distribute side registers no replicated metric).
Suites: `GroupPartitionsExecSuite` 28/28, `KeyGroupedPartitioningSuite`
172/172,
`EnsureRequirementsSuite` 57/57.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
