ulysses-you opened a new pull request, #58570: URL: https://github.com/apache/spark/pull/58570
### What changes were proposed in this pull request? Backport of #58339 to `branch-4.2`, adapted to this branch. The core fix is unchanged: `KeyedPartitioning` gains `mayContainUnknownPartitionKeys`, set by `KeyedShuffleSpec.createPartitioning` (the one-side shuffle), and `areKeysCompatible` only co-partitions such a side with a subset-keyed (or identically-declared and identically-marked) partner; `groupedSatisfies` gives it the full-key rules only; `createShuffleSpec` refuses a narrowing projection of a marked partitioning; `GroupPartitionsExec` keeps the claim only for an identity regrouping; `ShuffledJoin`'s `InnerLike` arm clears the marker when marked and unmarked keyed inputs meet. What differs from `master` on this branch: - The config is spelled `V2_BUCKETING_ALLOW_JOIN_KEYS_SUBSET_OF_PARTITION_KEYS`. - `isCollapsed` (SPARK-46367) and the `PartitioningCollection` invariant with `fromPartitionings`/`keyedMarkerOf` are not here; consumers read collections with `exists`, so `ShuffledJoin`'s clearing is implemented inline and `GroupPartitionsExec` reads any marked member. - `UnionExec` and `AliasAwareOutputExpression` are untouched: this branch's union does not merge `KeyedPartitioning`s, and its generic projection drops a `KeyedPartitioning` whose key column leaves the output, so neither wrong-results shape can arise here. - Consequently the two e2e tests whose `master` expectations rely on the aliasing projection keeping a narrowed claim expect the second join's one-side re-shuffle here, and the union-merge and regrouping repros are not carried. The commit message carries the full change description with the complete backport notes. ### Why are the changes needed? A storage-partitioned join whose preserved side is one-side-shuffled (`spark.sql.sources.v2.bucketing.shuffle.enabled`, shipping since 4.0.0) can silently lose matches in a following storage-partitioned join, with no error. Reproduction in #58339. ### Does this PR introduce _any_ user-facing change? Yes: it fixes a wrong-results bug in `spark.sql.sources.v2.bucketing.shuffle.enabled`. Storage-partitioned joins whose preserved side is one-side-shuffled against a smaller keyed side, followed by another storage-partitioned join on a larger key set, now fall back to shuffles and return correct results. ### How was this patch tested? The ported suites pass on `branch-4.2`: `KeyGroupedPartitioningSuite` (19 `SPARK-59050` tests), `ShuffleSpecSuite`, `DistributionSuite`, `GroupPartitionsExecSuite`, `EnsureRequirementsSuite`, `ValidateRequirementsSuite`. ### Was this patch authored or co-authored using generative AI tooling? Yes. Generated-by: Claude Code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
