peter-toth opened a new pull request, #58500:
URL: https://github.com/apache/spark/pull/58500
### What changes were proposed in this pull request?
`EnsureRequirements` leaves a side out of the comparison of the two sides'
reduced key types when that side has no partition key and its expressions no
longer describe the keys it would have had. The types then come from a side
that does answer for them.
`KeyedPartitioning.keyDataTypes` reports the types the partition key rows
were built with. With no key row to read, it falls back to the partition
expressions' own types. That is still the right answer while the expressions
describe the keys, and a join that reduced both sides' keys leaves expressions
that do not (`TransformExpression.reducedWith`, SPARK-59121). Only then is the
fallback a type no key of that partitioning would hold, and only then must a
caller keep it out of a comparison against a real answer.
An empty side that nothing reduced stays in the comparison, which is what
keeps the comparison doing its other job. Where one side has a reducer, it
holds the connector's `Reducer.resultType()` against the paired transform, and
that needs no key row.
The `keyDataTypes` scaladoc states the rule the fix follows, in place of the
paragraph that described the failure and pointed here.
### Why are the changes needed?
A storage-partitioned join whose two legs each reduced both of their sides
onto one key space is co-partitioned, and joins without a shuffle. If a leg
ends up with no partition key at all, the query fails instead.
`v2BucketingPartitionFilterEnabled` produces such a leg whenever its two sides
hold disjoint keys, i.e. whenever that leg is empty.
SELECT coalesce(l.ts, r.ts) FROM
(SELECT d.ts FROM days1 d JOIN years1 y ON y.ts = d.ts) l
JOIN
(SELECT y.ts FROM days2 d JOIN years2 y ON y.ts = d.ts) r
ON l.ts = r.ts
With `days2` and `years2` holding disjoint years, and `days` and `years`
reducing onto a common `LongType` year key:
[STORAGE_PARTITION_JOIN_INCOMPATIBLE_REDUCED_TYPES] Storage-partition
join partition
transforms produced incompatible reduced types, left reducers: []
returned: ["BIGINT"],
right reducers: [] returned: ["INT"]. SQLSTATE: 42K09
Both reducer lists are empty, which is the sign that there was nothing left
to reduce and nothing to compare. `INT` is the `years` transform's own result
type, not a type any key row holds.
### Does this PR introduce _any_ user-facing change?
Yes. The query above returns its result instead of failing. Only unreleased
versions are affected: the failure is reachable through SPARK-59121, and before
that the same shape failed on a `ClassCastException` from applying the reduce a
second time.
### How was this patch tested?
Two new `KeyGroupedPartitioningSuite` tests.
The first covers the shape above in both join orders, since the side to
leave out can be either one, and with both an inner and a full outer join. The
inner join intersects the two key sets to nothing and so has nothing to sort,
while the full outer join keeps the other side's keys and sorts them by the
reported types, which is what makes those types matter. Each part of the fix
fails this test on its own when disabled.
The second covers an empty side that is not marked, to pin that it stays in
the comparison. A one-side `days` -> `years` reduce whose `years` side is
emptied by an upstream inner join under the partition filter, against a reducer
returning `DateType` where the target transform is `IntegerType`, still raises
`STORAGE_PARTITION_JOIN_INCOMPATIBLE_REDUCED_TYPES`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5)
#### Backport to branch-4.2
A cherry-pick of apache#58486's two commits, squashed, with **one line
tailored**: the config is named
`V2_BUCKETING_ALLOW_JOIN_KEYS_SUBSET_OF_PARTITION_KEYS` on this branch, not
`V2_BUCKETING_ALLOW_KEYS_SUBSET_OF_PARTITION_KEYS`. Nothing else differs from
the `branch-4.3` backport.
Measured on the branch tip: `SPARK-59176: a leg reduced onto no key at all
still joins` fails there with
`STORAGE_PARTITION_JOIN_INCOMPATIBLE_REDUCED_TYPES`, the same shape as on
master, so the branch is affected. The precondition is present: SPARK-59121
reached `branch-4.2` as apache#58482.
172 tests green across `KeyGroupedPartitioningSuite`,
`GroupPartitionsExecSuite` and `EnsureRequirementsSuite`, plus 12 in
`ShuffleSpecSuite`. `dev/lint-scala` clean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
