peter-toth commented on code in PR #59599:
URL: https://github.com/apache/spark/pull/59599#discussion_r4339248452
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala:
##########
@@ -698,42 +709,131 @@ case class EnsureRequirements(
joinType == LeftAnti || joinType == LeftOuter
}
- // Populate the common partition information down to the scan nodes
- private def populateCommonPartitionInfo(
- plan: SparkPlan,
- values: Seq[(InternalRow, Int)],
+ /**
+ * Compose join-key positions from an outer SPJ with positions already
pushed by an inner SPJ.
+ * Outer positions index the layout the outer join sees; inner positions
index the scan's raw
+ * partition expressions, so they are composed rather than replaced.
+ */
+ private def composeJoinKeyPositions(
+ existing: Option[Seq[Int]],
+ newPositions: Option[Seq[Int]]): Option[Seq[Int]] = {
+ (existing, newPositions) match {
+ case (Some(old), Some(newPos)) => Some(newPos.map(old))
Review Comment:
**Finding 5.** This assumes `newPositions` index the layout that `existing`
projects to. When `joinKeyPositionsForOutput` defers the projection (finding
3), the scan keeps reporting the full-width partitioning, so the next join's
spec positions index the full layout. For `((i JOIN j ON id, name) JOIN k ON
i.id = k.id) JOIN m ON i.name = m.name`, the third push composes `Some(Seq(0))`
with `Some(Seq(1))`, and `Seq(1).map(Seq(0))` throws
`IndexOutOfBoundsException` during planning.
I traced this by reading and did not run it; the four-way query above would
show it. Keeping the reported layout and the positions in step, or at least
checking `newPos.forall(_ < old.length)` and falling back, would make it fail
safe.
---
_Generated by [Claude Code](https://claude.ai/code)_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
