sunchao commented on code in PR #57437:
URL: https://github.com/apache/spark/pull/57437#discussion_r3641016573
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/dynamicpruning/PartitionPruning.scala:
##########
@@ -110,6 +110,11 @@ object PartitionPruning extends Rule[LogicalPlan] with
PredicateHelper with Join
require(filteringKeys.size == 1, "DPP Filters should only have a single
broadcasting key " +
"since there are no usage for multiple broadcasting keys at the moment.")
val indices = Seq(joinKeys.indexOf(filteringKeys.head))
+ val broadcastValueProjection = if
(conf.dynamicPartitionPruningBroadcastProjectionEnabled) {
+ ReusableBroadcastValueProjection.find(filteringKeys.head, filteringPlan,
partScan)
Review Comment:
Thanks, this is a helpful distinction. You are right that `find` is eager
once the internal flag is enabled, even though projected reuse is only needed
after direct reuse fails.
For this revision I kept the transient tag so that the exact pruned-leaf
exclusion and selected source are carried consistently into both the adaptive
and non-adaptive planners, without changing the existing case-class constructor
or canonicalization. The tag is transient, all four rewrite paths are covered
by `broadcast value metadata survives logical rewrites without changing DPP
identity`, and a missing tag disables the optimization rather than changing the
query result.
I agree that either lazy re-derivation or a transient second parameter list
could simplify the implementation. The second-parameter-list approach in
particular addresses the `productArity` concern. I would be happy to switch if
you or @viirya prefer that direction; I did not want to silently broaden the
metadata and AQE changes while addressing the correctness feedback.
The same update also switches to `ExtractEquiJoinKeys`, so residual
ancestor-join conditions no longer incorrectly prevent reuse.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
