ulysses-you opened a new pull request, #58572: URL: https://github.com/apache/spark/pull/58572
### What changes were proposed in this pull request? Backport of #58339 to `branch-4.0`, reduced to what this branch's SPJ machinery has; it is the `branch-4.1` backport (#58571) applied as is, which holds because the two branches share every piece of SPJ machinery this fix touches. The core fix is unchanged: `KeyGroupedPartitioning` gains `mayContainUnknownPartitionKeys`, set by `KeyGroupedShuffleSpec.createPartitioning` (the one-side shuffle), and `areKeysCompatible` only co-partitions such a side with a subset-keyed (or identically-declared and identically-marked) partner; `satisfies0` gives it the full-key rules only (no subset clustering, no global ordering beyond a single partition); `createShuffleSpec` refuses a narrowing projection of a marked partitioning. Two e2e tests are adapted around a pre-existing `branch-4.0` limitation, reproduced on the unmodified branch: with `spark.sql.sources.v2.bucketing.pushPartValues.enabled` (default on), `checkKeyGroupCompatible` merges the two sides' partition values and pushes the merged set down without requiring the keyed side to be a plain scan subtree, so a keyed side sitting above a one-side-shuffled join whose partition count differs from the partner's plans a shuffle-free join over unequal partition counts and throws `Can't zip RDDs with unequal numbers of partitions` at execution. Neither side carries the marker in those shapes, so it is independent of this fix; `branch-4.1` already refuses the shape. The two tests give the second keyed table the first side's key set instead of a superset, keeping the equal-count direct storage-partitioned join they are about. The commit message carries the full change description with the complete backport notes. ### Why are the changes needed? A storage-partitioned join whose preserved side is one-side-shuffled (`spark.sql.sources.v2.bucketing.shuffle.enabled`, shipping since 4.0.0) can silently lose matches in a following storage-partitioned join, with no error. Reproduction in #58339. ### Does this PR introduce _any_ user-facing change? Yes: it fixes a wrong-results bug in `spark.sql.sources.v2.bucketing.shuffle.enabled`. Storage-partitioned joins whose preserved side is one-side-shuffled against a smaller keyed side, followed by another storage-partitioned join on a larger key set, now fall back to shuffles and return correct results. ### How was this patch tested? The ported suites pass on `branch-4.0`: `KeyGroupedPartitioningSuite` (12 `SPARK-59050` tests), `ShuffleSpecSuite` (3 new tests), `DistributionSuite`, `EnsureRequirementsSuite`, `ValidateRequirementsSuite`. ### Was this patch authored or co-authored using generative AI tooling? Yes. Generated-by: Claude Code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
