ulysses-you opened a new pull request, #58245: URL: https://github.com/apache/spark/pull/58245
### What changes were proposed in this pull request? For a non-join operator, when a grouped `KeyedPartitioning` satisfies a `ClusteredDistribution` whose clustering keys are a strict subset of the partition keys (via `v2BucketingAllowKeysSubsetOfPartitionKeys`), `EnsureRequirements` now inserts a `GroupPartitionsExec` that projects the partition keys to the operation keys, so partitions sharing the same operation key are coalesced. ### Why are the changes needed? When `v2BucketingAllowKeysSubsetOfPartitionKeys` is enabled and a storage-partitioned source is partitioned by more keys than the operation requires (e.g. a table partitioned by `(id, name)` with a window `PARTITION BY id`), the grouped `KeyedPartitioning` satisfies the `ClusteredDistribution` through the subset-key relaxation, but the partitions are still grouped by the full partition keys rather than by the operation keys. Without coalescing them, the downstream operator ranks/aggregates each partition independently, producing incorrect results. For example, a top-k window over `PARTITION BY id` on a `(id, name)`-partitioned table would surface `id=1` twice (once per `(1,'aa')`/`(1,'bb')` partition) instead of once. ### Does this PR introduce _any_ user-facing change? Yes, it fixes a data correctness issue: storage-partitioned windows (and other non-join operators) with `PARTITION BY` a subset of the partition keys now produce correct results when `v2BucketingAllowKeysSubsetOfPartitionKeys` is enabled. ### How was this patch tested? Added regression tests in `KeyGroupedPartitioningSuite` covering: - window top-k over `PARTITION BY` a subset of the partition keys - window top-k over a duplicated `PARTITION BY` key - window top-k over union output partitioning Verified `org.apache.spark.sql.connector.KeyGroupedPartitioningSuite` and `org.apache.spark.sql.execution.exchange.EnsureRequirementsSuite` pass. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
