[
https://issues.apache.org/jira/browse/SPARK-29552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ke Jia updated SPARK-29552:
---------------------------
Description: AQE will optimize the logical plan once there is query stage
finished. So for inner join, when two join side is all small to be the build
side. The planner of converting logical plan to physical plan will select the
build side as BuildRight if right side finished firstly or BuildLeft if left
side finished firstly. In some case, when BuildRight or BuildLeft may introduce
additional exchange to the parent node. The revert approach in
OptimizeLocalShuffleReader rule may be too conservative, which revert all the
local shuffle reader when introduce additional exchange not revert the local
shuffle reader introduced shuffle. It may be expense to only revert the local
shuffle reader introduced shuffle. The workaround is to apply the
OptimizeLocalShuffleReader rule again when creating new query stage to further
optimize the sub tree shuffle reader to local shuffle reader. (was: AQE will
optimize the logical plan once there is query stage finished. So for inner
join, when two join side is all small to be the build side. The planner of
converting logical plan to physical plan will select the build side as
BuildRight if right side finished firstly or BuildLeft if left side finished
firstly. In some case, when BuildRight or BuildLeft may introduce additioanl
exchange to the parent node. The revert approach in OptimizeLocalShuffleReader
rule may be too conservative, which revert all the local shuffle reader when
introduce additional exchange not revert the local shuffle reader introduced
shuffle. It may be expense to only revert the local shuffle reader introduced
shuffle. The workaround is to apply the OptimizeLocalShuffleReader rule again
when creating new query stage to further optimize the sub tree shuffle reader
to local shuffle reader.)
> Fix the flaky test failed in AdaptiveQueryExecSuite # multiple joins
> --------------------------------------------------------------------
>
> Key: SPARK-29552
> URL: https://issues.apache.org/jira/browse/SPARK-29552
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Ke Jia
> Priority: Major
>
> AQE will optimize the logical plan once there is query stage finished. So for
> inner join, when two join side is all small to be the build side. The planner
> of converting logical plan to physical plan will select the build side as
> BuildRight if right side finished firstly or BuildLeft if left side finished
> firstly. In some case, when BuildRight or BuildLeft may introduce additional
> exchange to the parent node. The revert approach in
> OptimizeLocalShuffleReader rule may be too conservative, which revert all the
> local shuffle reader when introduce additional exchange not revert the local
> shuffle reader introduced shuffle. It may be expense to only revert the
> local shuffle reader introduced shuffle. The workaround is to apply the
> OptimizeLocalShuffleReader rule again when creating new query stage to
> further optimize the sub tree shuffle reader to local shuffle reader.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
