peter-toth commented on code in PR #58335:
URL: https://github.com/apache/spark/pull/58335#discussion_r3880262752
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/TransformExpression.scala:
##########
@@ -98,6 +98,14 @@ case class TransformExpression(
}
}
+ /**
+ * Re-targets this partition transform expression at `attr`. A partition
transform expression
+ * has a single leaf attribute (`KeyedPartitioning.supportsExpressions`), so
this replaces it
+ * and the result reports the same transform over `attr`.
+ */
+ def withReference(attr: Attribute): TransformExpression =
Review Comment:
**Finding 9.** The premise is right —
`KeyedPartitioning.supportsExpressions` allows one leaf attribute per transform
— but it also allows a `GetStructField` above that attribute, and `transform`
replaces only the attribute, so the *path* comes from the source expression
rather than from the target.
`GroupPartitionsExec.outputPartitioning` calls this with
`expr.references.head`, which is the bare attribute. Re-targeting the other
side's `bucket(4, GetStructField(other, 0))` at a struct `s` whose own key
expression is `bucket(4, GetStructField(s, 1))` therefore reports field 0 — the
source's ordinal read on this side's struct.
I could not reach it from a query, so this is a scaladoc point rather than a
live defect: a nested partition reference resolves to an `Alias` over
`GetStructField`, and `supportsExpressions`' `isReference` does not unwrap
`Alias`, so a nested-field partition transform produces no `KeyedPartitioning`
at all today (I measured that for `identity("s.a")`, which reports
`UnknownPartitioning`). Worth saying in the doc that the caller has to pass the
reference of the *whole* key expression, not just an attribute, so that whoever
makes nested keys work does not inherit a silent wrong-field report.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
