aokolnychyi commented on a change in pull request #32921: URL: https://github.com/apache/spark/pull/32921#discussion_r655733488
########## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala ########## @@ -96,6 +96,7 @@ case class AdaptiveSparkPlanExec( @transient private val queryStageOptimizerRules: Seq[Rule[SparkPlan]] = Seq( PlanAdaptiveDynamicPruningFilters(this), ReuseAdaptiveSubquery(context.subqueryCache), + PrepareScans, Review comment: Thanks for looking into this, @cloud-fan! I made dynamic filtering work in query prep rules but there are issues with dynamic subquery reuse. In short, if we want to reuse dynamic subqueries, we cannot execute them in prep rules. If we execute dynamic subqueries in prep rules, this will break `ReuseAdaptiveSubquery` and that one must be run in the query stage optimizer rules. The more I think the more it feels like we won't be able to execute dynamic filters before running `EnsureRequirements` for the first time. That's why I am exploring the option of running prep rules one more time once a stage is created. Does anybody have any thoughts on this? AQE currently runs prep rules only before creating stages. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org