peter-toth commented on code in PR #59950:
URL: https://github.com/apache/spark/pull/59950#discussion_r4443320582
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/dynamicpruning/PlanDynamicPruningFilters.scala:
##########
@@ -125,7 +125,12 @@ case class PlanDynamicPruningFilters(sparkSession:
SparkSession) extends Rule[Sp
case None =>
val aliases = broadcastKeyIndices.map(idx =>
Alias(buildKeys(idx), buildKeys(idx).toString)())
- val aggregate = Aggregate(aliases, aliases, buildPlan)
+ // Alias the keys in a Project instead of making them the grouping
expressions.
Review Comment:
**Finding 6.** This branch runs only when `onlyInBroadcast` is false.
`PartitionPruning` sets it to `conf.dynamicPartitionPruningReuseBroadcastOnly
|| !hasBenefit` (`PartitionPruning.scala:126`).
`spark.sql.optimizer.dynamicPartitionPruning.reuseBroadcastOnly` is internal
and defaults to true. So with the default confs DPP never plans this subquery,
and no plan changes. Both new tests set it to false for that reason. It also
bounds the scan-parallelism cost the user-facing section describes, since only
users who set that conf can see it.
Could the user-facing section say so? For example: "This applies only when
the internal `spark.sql.optimizer.dynamicPartitionPruning.reuseBroadcastOnly`
is set to false. With the default, DPP only reuses a broadcast and plans no
subquery."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
