Peter Toth created SPARK-58562:
----------------------------------
Summary: Document subplan merging in the SQL performance tuning
guide
Key: SPARK-58562
URL: https://issues.apache.org/jira/browse/SPARK-58562
Project: Spark
Issue Type: Documentation
Components: SQL
Affects Versions: 5.0.0
Reporter: Peter Toth
The MergeSubplans optimizer rule
(org.apache.spark.sql.execution.planmerging.MergeSubplans) merges
one-row-result subplans - scalar subqueries and non-grouping aggregates - that
read the same input, so the input is scanned once instead of once per subplan.
The rule is on by default and has 4 public configs under
spark.sql.optimizer.mergeSubplans.filterPropagation.*, 3 of which are disabled
by default and need a user decision.
Today none of this is described in the docs. The configs only show up in the
generated SQL config table in configuration.md, and the single mention anywhere
else is the 4.3 migration-guide note about the rule's package move. In
particular users have no way to discover that symmetric filter propagation,
while off by default because OR-widening the merged filter can cost IO pruning,
gave multi-fold speedups on TPC-DS query shapes that compute several
differently-filtered aggregates over the same table.
Add a section to docs/sql-performance-tuning.md that covers what the rule does,
the query shapes it applies to, the configs and their tradeoffs, and how to
disable the rule.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
