Kapil Singh created SPARK-39690:
-----------------------------------
Summary: Reuse exchange across subqueries is broken with AQE if
subquery side exchange materialized first
Key: SPARK-39690
URL: https://issues.apache.org/jira/browse/SPARK-39690
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.3.0
Reporter: Kapil Singh
When trying to reuse Exchange of a subquery in main plan, if the Exchange
inside subquery materialize first then main ASPE node won't have that stage
info (in
[stageToReplace|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala#L243])
to replace in current logical plan. This will cause AQE to produce new
candidate physical plan without reusing the exchange present inside subquery.
And depending on how complex the inner plan is (no. of exchanges) AQE could
choose plan without ReusedExchange.
We have seen in with multiple queries with our private build. This can happen
in DPP also.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
