peter-toth opened a new pull request, #58486:
URL: https://github.com/apache/spark/pull/58486
### What changes were proposed in this pull request?
`EnsureRequirements` compares the two sides' reduced key types only when
both sides have partition
keys, and takes the types from the side that has them.
`KeyedPartitioning.keyDataTypes` reports the types the partition key rows
were built with. With no
key row to read, it falls back to the partition expressions' own types.
Those types describe the
keys only while the expressions do, and a join that reduced both sides' keys
leaves expressions
that do not (`TransformExpression.reducedWith`, SPARK-59121). The fallback
is then not an answer
about the keys, and holding it against the other side's real answer fails
the query.
The `keyDataTypes` scaladoc states the rule the fix follows, in place of the
paragraph that
described the failure and pointed here.
One check is given up along with the untruthful comparison. Where a side has
a reducer, the
comparison also holds the connector's `Reducer.resultType()` against the
paired transform, which
needs no key row. That diagnostic is now skipped whenever a filter empties
that side. Only a
connector that already breaks `r(f1(x)) = f2(x)` on types is affected, and
it loses a message it
used to get. In one shape it loses more than the message: with
`v2BucketingShuffleEnabled` on, a
third child shuffled onto the merged keys evaluates the target transform per
row, so the mismatch
surfaces as a `ClassCastException` instead.
The alternative is to skip only an already-reduced pair, which would keep
the check for a side that
is empty but not marked. That was not taken. The fallback is untruthful for a
`KeyedShuffleSpec.createPartitioning` result as well, so a narrower guard
trades a lost message for
a query that fails while it is correct, which is the worse of the two and is
the bug being fixed
here.
### Why are the changes needed?
A storage-partitioned join whose two legs each reduced both of their sides
onto one key space is
co-partitioned, and joins without a shuffle. If a leg ends up with no
partition key at all, the
query fails instead. `v2BucketingPartitionFilterEnabled` produces such a leg
whenever its two
sides hold disjoint keys, i.e. whenever that leg is empty.
SELECT coalesce(l.ts, r.ts) FROM
(SELECT d.ts FROM days1 d JOIN years1 y ON y.ts = d.ts) l
JOIN
(SELECT y.ts FROM days2 d JOIN years2 y ON y.ts = d.ts) r
ON l.ts = r.ts
With `days2` and `years2` holding disjoint years, and `days` and `years`
reducing onto a common
`LongType` year key:
[STORAGE_PARTITION_JOIN_INCOMPATIBLE_REDUCED_TYPES] Storage-partition
join partition
transforms produced incompatible reduced types, left reducers: []
returned: ["BIGINT"],
right reducers: [] returned: ["INT"]. SQLSTATE: 42K09
Both reducer lists are empty, which is the sign that there was nothing left
to reduce and nothing
to compare. `INT` is the `years` transform's own result type, not a type any
key row holds.
### Does this PR introduce _any_ user-facing change?
Yes. The query above returns its result instead of failing. Only unreleased
versions are affected:
the failure is reachable through SPARK-59121, and before that the same shape
failed on a
`ClassCastException` from applying the reduce a second time.
### How was this patch tested?
A new `KeyGroupedPartitioningSuite` test covers the shape in both join
orders, since the side to
leave out can be either one, and with both an inner and a full outer join.
The inner join
intersects the two key sets to nothing and never sorts them, so the full
outer join is what makes
the reported types matter. Each of the three parts of the fix fails the test
on its own when
disabled.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
