ulysses-you opened a new pull request, #58571: URL: https://github.com/apache/spark/pull/58571
### What changes were proposed in this pull request? Backport of #58339 to `branch-4.1`, reduced to what this branch's SPJ machinery has. The core fix is unchanged: `KeyGroupedPartitioning` gains `mayContainUnknownPartitionKeys`, set by `KeyGroupedShuffleSpec.createPartitioning` (the one-side shuffle), and `areKeysCompatible` only co-partitions such a side with a subset-keyed (or identically-declared and identically-marked) partner; `satisfies0` gives it the full-key rules only (no subset clustering, no global ordering beyond a single partition); `createShuffleSpec` refuses a narrowing projection of a marked partitioning. There is no `GroupPartitionsExec` on this branch, and neither the union nor the generic projection transforms a `KeyGroupedPartitioning`, so the marker needs no propagation beyond the partitioning itself and no clearing at `ShuffledJoin`: consumers of a joined collection read it with `exists`, so a spurious marked member cannot cost a sound storage-partitioned join. This branch also has no alignment for differing partition counts between two keyed sides, so a second join whose sides differ in `numPartitions` one-side-shuffles the join output onto the larger layout, with or without the marker; the three `master` tests that expect a direct storage-partitioned join there pin this branch's two-exchange shape instead. None of them regressed relative to the pre-fix plan. The commit message carries the full change description with the complete backport notes. ### Why are the changes needed? A storage-partitioned join whose preserved side is one-side-shuffled (`spark.sql.sources.v2.bucketing.shuffle.enabled`, shipping since 4.0.0) can silently lose matches in a following storage-partitioned join, with no error. Reproduction in #58339. ### Does this PR introduce _any_ user-facing change? Yes: it fixes a wrong-results bug in `spark.sql.sources.v2.bucketing.shuffle.enabled`. Storage-partitioned joins whose preserved side is one-side-shuffled against a smaller keyed side, followed by another storage-partitioned join on a larger key set, now fall back to shuffles and return correct results. ### How was this patch tested? The ported suites pass on `branch-4.1`: `KeyGroupedPartitioningSuite` (12 `SPARK-59050` tests), `ShuffleSpecSuite` (3 new tests), `DistributionSuite`, `EnsureRequirementsSuite`, `ValidateRequirementsSuite`. ### Was this patch authored or co-authored using generative AI tooling? Yes. Generated-by: Claude Code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
