pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4072333034
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,14 +405,26 @@ object RewritePredicateSubquery extends Rule[LogicalPlan]
with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly
introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite`
returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is
not descended into:
+ * rewriting an existential sub-query nested in its join condition would
graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place
may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
Review Comment:
Accepted in full — this was a regression I introduced when I replaced the
traversal, not a
pre-existing gap, and your reading of the two lines is exactly right.
Each arm now builds its replacement through a small helper,
```scala
def replacing(sq: Expression)(replacement: => Expression): Expression = {
val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
newExpr.copyTagsFrom(sq)
newExpr
}
```
so everything an arm constructs — the `exists` attribute, the join
conditions and the `Join` itself
— is built under the origin of the subquery expression being replaced, and
that expression's tags
are carried over, which is what `transformDownWithPruning` did.
Your note about where the cost lands is what decided the shape of the fix:
`handleUnaryNode` sends
every `UnaryNode` through this method with the default `canRewrite`, so the
repair had to sit in the
arms themselves rather than in the new routing callers. The pruning check
and the unchanged-node
fast path are unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
