aokolnychyi commented on a change in pull request #32921:
URL: https://github.com/apache/spark/pull/32921#discussion_r655733488
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala
##########
@@ -96,6 +96,7 @@ case class AdaptiveSparkPlanExec(
@transient private val queryStageOptimizerRules: Seq[Rule[SparkPlan]] = Seq(
PlanAdaptiveDynamicPruningFilters(this),
ReuseAdaptiveSubquery(context.subqueryCache),
+ PrepareScans,
Review comment:
Thanks for looking into this, @cloud-fan!
I made dynamic filtering work in query prep rules but there are issues with
dynamic subquery reuse. In short, if we want to reuse dynamic subqueries, we
cannot execute them in prep rules. If we execute dynamic subqueries in prep
rules, this will break `ReuseAdaptiveSubquery` and that one must be run in the
query stage optimizer rules.
The more I think the more it feels like we cannot execute dynamic filters
before running `EnsureRequirements` for the first time. That's why I am
exploring the option of running prep rules one more time once a stage is
created.
Does anybody have any thoughts on this? AQE currently runs prep rules only
before creating stages. Will it be too bad to call prep rules twice? Once
before dynamic filtering and once after?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
