ulysses-you commented on code in PR #58339:
URL: https://github.com/apache/spark/pull/58339#discussion_r3891995996
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala:
##########
@@ -648,7 +660,8 @@ case class KeyedPartitioning(
// and the other side is re-shuffled using this spec, the two
`KeyedPartitioning`s carry the
// same keys in a different order and
`PartitioningCollection.fromPartitionings` rejects them.
val projectedPartitioning =
- new KeyedPartitioning(projectedExpressions, projectedKeys, isGrouped =
false).toGrouped
+ new KeyedPartitioning(projectedExpressions, projectedKeys, isGrouped =
false,
+ mayContainUnknownPartitionKeys =
mayContainUnknownPartitionKeys).toGrouped
Review Comment:
Fixed in 9a450ca. The refusal is on the producer rather than in `tryCreate`:
`KeyedPartitioning.createShuffleSpec` returns the unprojected spec when the
partitioning is unknown-keyed and `joinKeyPositions.length <
expressions.length`. The `tryCreate` gate would not catch the repro because the
one-side-shuffle path reaches `createShuffleSpec` directly through
`Partitioning.createShuffleSpec` in `EnsureRequirements`' shuffle loop,
bypassing `checkKeyGroupCompatible`. The `clustering(positionSet.head)` hazard
you flagged: `KeyedShuffleSpec.canCreatePartitioning` now refuses a partition
expression that maps to no clustering key, so the unprojected spec cannot be
chosen to re-shuffle another child, and pairing with a narrower partner fails
on arity in `isCompatibleWith`, sending the child down the ordinary shuffle.
`GroupPartitionsExec.outputPartitioning` got the same rule for its
`joinKeyPositions` projection. Repro test `SPARK-59050: SPJ: join-key
projection of an unknown-keyed layou
t drops the claim`: 0 of 4 rows on 948df55, 4 of 4 with the second join
shuffled now. Per your guidance this deliberately does not route through
`isNarrowed`/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
