ulysses-you opened a new pull request, #58775:
URL: https://github.com/apache/spark/pull/58775
### What changes were proposed in this pull request?
`RemoveRedundantWindowGroupLimits` now looks through the
`GroupPartitionsExec` and the local sort that `EnsureRequirements` may put
between the two `WindowGroupLimitExec` nodes, and removes the partial node
below them. The local sort feeding the partial node goes with it where a sort
between the two nodes already orders the rows the final node ranks as a whole;
where there is none, that sort is what carries that ordering, and it stays.
### Why are the changes needed?
A partial `WindowGroupLimitExec` below a `GroupPartitionsExec` cannot change
the result: a partition of the final node is a union of partitions of the
partial node, so every row the final node keeps is one the partial node kept
too, and the grouping is there to satisfy the final node's distribution.
Removing it, together with the local sort that only fed it when a sort above
already orders the final node's rows, drops an operator and a sort pass from
the plan.
### Does this PR introduce _any_ user-facing change?
Yes, the plan changes for a window top-k whose child needs a
`GroupPartitionsExec`, e.g.
Before:
```
WindowGroupLimit (Final)
+- Sort
+- GroupPartitions
+- WindowGroupLimit (Partial)
+- Sort
+- BatchScan
```
After:
```
WindowGroupLimit (Final)
+- Sort
+- GroupPartitions
+- BatchScan
```
Where no sort lands between the final node and the grouping, the partial
node goes on its own and the sort it fed stays where the partial node was.
Results are unchanged.
### How was this patch tested?
New tests in `KeyGroupedPartitioningSuite` for a window top-k over a v2
table whose partition keys are not grouped, so `EnsureRequirements` coalesces
them with a `GroupPartitionsExec`: one where the ranked column is a partition
key, so no sort sits between the two limit nodes and the partial node's own
sort is the one that stays; one where it is not a partition key, so a sort sits
between them and goes away with the partial node; and one where the source
reports the ordering the window needs, leaving the partial node without a sort
of its own. Each asserts the limit modes left, the total number of sorts, and
where the surviving sort reads from.
A new test in `RemoveRedundantWindowGroupLimitsSuite` covers the case where
the child already satisfies the final node's distribution, and asserts the same
for the sort that fed the partial node there.
Ran `RemoveRedundantWindowGroupLimitsSuite` (with AQE both on and off), the
window tests of `KeyGroupedPartitioningSuite`, `SQLMetricsSuite` (SPARK-37099),
and the `window.sql` golden files.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (deepseek-v4.1-flash)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
