ulysses-you opened a new pull request, #58568: URL: https://github.com/apache/spark/pull/58568
### What changes were proposed in this pull request? Backport of #58339 to `branch-4.3`. Two conflicts, resolved in the branch's favor: - `UnionExec.outputPartitioning`: this branch normalizes children's partitionings to the first child's attribute space and lifts the merged `KeyedPartitioning` through `toUnionOutput`, so the merged-case arm keeps that shape; the marker change itself (the `compatible` check) applied as is. - Test imports: `InMemoryCatalystRuntimeFilterCatalog` and the extra join-type imports are not on this branch and are unused by the ported tests, so they are dropped; `Inner` and `KeyedPartitioning` are imported for the new tests. Everything else applies unchanged. ### Why are the changes needed? A storage-partitioned join whose preserved side is one-side-shuffled (`spark.sql.sources.v2.bucketing.shuffle.enabled`) can silently lose matches in a following storage-partitioned join, with no error. See the full analysis and reproduction in #58339. ### Does this PR introduce _any_ user-facing change? Yes: it fixes a wrong-results bug in `spark.sql.sources.v2.bucketing.shuffle.enabled`. Storage-partitioned joins whose preserved side is one-side-shuffled against a smaller keyed side, followed by another storage-partitioned join on a larger key set, now fall back to shuffles and return correct results. ### How was this patch tested? The ported suites pass on `branch-4.3`: `KeyGroupedPartitioningSuite` (21 `SPARK-59050` tests), `ShuffleSpecSuite`, `DistributionSuite`, `GroupPartitionsExecSuite`, `EnsureRequirementsSuite`, `ValidateRequirementsSuite` (284 tests total). ### Was this patch authored or co-authored using generative AI tooling? Yes. Generated-by: Claude Code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
