Hemanth Boyina created SPARK-61447:
--------------------------------------
Summary: Avoid exponential planning cost for nested dynamic
pruning subqueries with AQE
Key: SPARK-61447
URL: https://issues.apache.org/jira/browse/SPARK-61447
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Hemanth Boyina
With adaptive query execution enabled, a query that joins a partitioned table
to many other partitioned tables through a chain of left joins spends
exponential time and memory in planning.
PartitionPruning adds a dynamic pruning filter on each right-side table. The
subquery of the i-th filter contains the left side of the join, which already
contains the filters from joins 1 to i-1. So the filters nest inside each
other, and the optimized logical plan stays linear (about 5 nodes per join).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]