sunchao commented on code in PR #57437:
URL: https://github.com/apache/spark/pull/57437#discussion_r3687363883
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/subquery.scala:
##########
@@ -127,23 +127,36 @@ case class InSubqueryExec(
override def nullable: Boolean = child.nullable
override def toString: String = s"$child IN ${plan.name}"
- override def withNewPlan(plan: BaseSubqueryExec): InSubqueryExec = copy(plan
= plan)
+ override def withNewPlan(plan: BaseSubqueryExec): InSubqueryExec =
+ copy(plan = plan, resultBroadcast = null, result = null)
Review Comment:
Fixed in `549b6db`, thanks @peter-toth. `InSubqueryExec.withNewPlan` now
clears only `result` and preserves `resultBroadcast`, restoring the existing
behavior for ordinary `IN` subqueries. The new `IN predicate subquery preserves
its broadcast when replacing its plan` regression clears the transient result
and verifies that evaluation still succeeds through the retained broadcast.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/DynamicPruning.scala:
##########
@@ -80,7 +86,20 @@ case class DynamicPruningSubquery(
// DynamicPruningSubquery should only have a single broadcasting key
since
// there are no usage for multiple broadcasting keys at the moment.
broadcastKeyIndices.size == 1 &&
- child.dataType == buildKeys(broadcastKeyIndices.head).dataType
+ child.dataType == buildKeys(broadcastKeyIndices.head).dataType &&
+ broadcastValueProjection.forall { projection =>
Review Comment:
Fixed in `549b6db`, thanks @peter-toth. Projection validation now lives in
`usableBroadcastValueProjection` instead of `DynamicPruningSubquery.resolved`.
Both nonadaptive planning and the adaptive handoff consume only the filtered
projection, so malformed optional metadata leaves the DPP expression resolved
and simply disables projected reuse. The Catalyst regression covers missing
source/value attributes, empty hash keys, and mismatched value types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
