ammarchalifah opened a new pull request, #59599: URL: https://github.com/apache/spark/pull/59599
### What changes were proposed in this pull request? Fix nested storage-partitioned join (SPJ) pushdown on `branch-4.1`. When a 3-way join triggers SPJ on both joins, `EnsureRequirements` walks each child subtree and **replaces** `BatchScanExec.spjParams` instead of composing with params already pushed by an inner join. Changes: - **Compose join-key positions** (`composeJoinKeyPositions`) so an outer join does not overwrite inner positions (e.g. keep `[1]` instead of replacing with `[0]`). - **Side-aware nested pushdown** (`populateSpjParamsOnSide` / `nestedSpjPushFlags`): sync `reducers` and `commonPartitionValues` to both inner-join sides; compose `joinKeyPositions` only on the side that feeds the outer join; retain inner common-partition metadata when the outer join uses a subset of partition keys. - **Partition-count guard** in `areChildrenCompatible`: fall back to shuffle alignment when SPJ push leaves mismatched `numPartitions`. - **`KeyGroupedPartitionedScan`**: defer output partitioning projection for leading-prefix subset join keys over full-width common partition values; apply join-key regrouping at read time via `getInputPartitionGrouping`. Related: [SPARK-60203](https://issues.apache.org/jira/browse/SPARK-60203) / #59410. Targets `branch-4.1` only. `master` and `branch-4.2+` use `GroupPartitionsExec` (SPARK-55535) and are unaffected. ### Why are the changes needed? [SPARK-60232](https://issues.apache.org/jira/browse/SPARK-60232) reports three failure shapes on 4.0/4.1 nested SPJ: 1. **Join key positions overwritten** — outer join replaces inner `joinKeyPositions`, causing wrong grouping or 0 rows. 2. **Reducers cleared** — outer push drops inner `reducers`, breaking compatible-transform co-partitioning. 3. **Common partition values misaligned** — outer subset-key push misaligns nested siblings in a `PartitioningCollection`. The 4.2+ fix (`GroupPartitionsExec`) is not backported; this is a minimal 4.1 fix scoped to scan-level SPJ params. ### Does this PR introduce _any_ user-facing change? Yes. Nested 3-way SPJ queries that returned wrong results, failed at planning, or threw at runtime on 4.1.x now run correctly for the JIRA shapes covered by tests. No change on `master` or `branch-4.2+`. ### How was this patch tested? Added tests in `KeyGroupedPartitioningSuite` for SPARK-60232: - `nested SPJ preserves inner join key positions` (shape 1) - `nested SPJ with mixed partition key types preserves key positions` (shape 1b) - `nested SPJ preserves inner join reducers` (shape 2) - `nested SPJ preserves join key positions over common partition values` (shape 3, one name per id) - `nested SPJ with multiple names per id over common partition values` (ignored — needs 4.2+) Also verified SPARK-60203 regression: ```bash build/sbt 'sql/testOnly *KeyGroupedPartitioningSuite -- -z "SPARK-60232"' build/sbt 'sql/testOnly *KeyGroupedPartitioningSuite -- -z "SPARK-60203"' ``` ### Was this patch authored or co-authored using generative AI tooling? Yes: Grok 4.7 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
