[ https://issues.apache.org/jira/browse/SPARK-36032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
XiDuo You updated SPARK-36032: ------------------------------ Description: At {{initialPlan}} we may remove some Spark Plan with {{queryStagePreparationRules}}, if removed Spark Plan is top level node, then we will lose the linked logical node. Since we support AQE side broadcast join config. It's more common that a join is SMJ at normal planner and changed to BHJ after AQE reOptimize. However, {{RemoveRedundantSorts}} is applied before reOptimize at {{initialPlan}}, then a local sort might be removed incorrectly if a join is SMJ at first but changed to BHJ during reOptimize. was: Since we support AQE side broadcast join config. It's more common that a join is SMJ at normal planner and changed to BHJ after AQE reOptimize. However, `RemoveRedundantSorts` is applied before reOptimize, then a local sort might be removed incorrectly if a join is SMJ at first but changed to BHJ during reOptimize. > Use inputPlan instead of currentPhysicalPlan to initialize logical link > ----------------------------------------------------------------------- > > Key: SPARK-36032 > URL: https://issues.apache.org/jira/browse/SPARK-36032 > Project: Spark > Issue Type: Sub-task > Components: SQL > Affects Versions: 3.2.0, 3.3.0 > Reporter: XiDuo You > Priority: Major > > At {{initialPlan}} we may remove some Spark Plan with > {{queryStagePreparationRules}}, if removed Spark Plan is top level node, then > we will lose the linked logical node. > Since we support AQE side broadcast join config. It's more common that a join > is SMJ at normal planner and changed to BHJ after AQE reOptimize. However, > {{RemoveRedundantSorts}} is applied before reOptimize at {{initialPlan}}, > then a local sort might be removed incorrectly if a join is SMJ at first but > changed to BHJ during reOptimize. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org