cloud-fan commented on code in PR #56575:
URL: https://github.com/apache/spark/pull/56575#discussion_r3503088334
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala:
##########
@@ -817,7 +817,10 @@ object QueryExecution {
plan: LogicalPlan): SparkPlan = {
// TODO: We use next(), i.e. take the first plan returned by the planner,
here for now,
// but we will implement to choose the best plan.
- planner.plan(ReturnAnswer(plan)).next()
+ // Strip DelegateExpression to its definition right before planning, so
the planner and every
+ // physical consumer (pushdown, columnar rules, codegen) sees the real
executed expression. The
+ // wrapper is purely informational and stays in the optimized logical plan
for EXPLAIN.
+ planner.plan(ReturnAnswer(LowerDelegateExpression(plan))).next()
Review Comment:
Correct that these logical consumers (`ExtractEquiJoinKeys`,
`PartitionPruning`, `InjectRuntimeFilter`, `StreamingJoinHelper`) run before
`createSparkPlan` and match a bare `EqualTo`/`And`/..., so they won't see
through the wrapper to a join equality in `definition`.
I've left this as a documented limitation rather than a code change, because
it's latent today: no shipped delegate produces a boolean/predicate
`definition` used as a join condition -- `right` returns a string and
`multi_get_json_object` returns a struct. The V2 test constructs a
predicate-shaped delegate by hand to exercise the pushdown unfold, but nothing
registers one. Teaching every logical consumer to look through the wrapper
would re-introduce the per-consumer-unfold problem this redesign set out to
avoid (keeping the wrapper opaque and lowering once at the planner entry).
Added a note to the `DelegateExpression` class doc in `2f4260d6507`: a
future predicate-shaped delegate meant to participate in these rewrites must
either lower earlier or teach those consumers to look through it. Happy to file
a follow-up JIRA to track it if you'd prefer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
