XiDuo You created SPARK-60379:
---------------------------------
Summary: Push down a shuffle above a union below it
Key: SPARK-60379
URL: https://issues.apache.org/jira/browse/SPARK-60379
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: XiDuo You
An `EnsureRequirements` shuffle that sits above a `UnionExec` shuffles the
whole union, even
when some of the union's children already produce a layout that answers the
requirement, or when
a layout one of them produces would answer it if the other children were laid
out on it.
Push that shuffle below the union instead: pick one of the layouts the union's
own children
report, keep the children that already report it, and shuffle the rest onto it.
The operator
above then runs with the picked layout's partition count. The plan as a whole
is validated
before the rewrite is taken.
The operators between the shuffle and the union can be looked through (a
project, a local sort,
a filter, a partial aggregate, a partial window group limit) under a config of
their own,
because a child is then shuffled before the reduction, which can send more data
than the one
shuffle the rewrite replaces.
The rule runs during AQE query stage preparation, so no exchange is
materialized before it
gets a chance to remove one.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
