XiDuo You created SPARK-61860:
---------------------------------
Summary: Push a keyed shuffle down through clustered operators in
AQE preparation
Key: SPARK-61860
URL: https://issues.apache.org/jira/browse/SPARK-61860
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: XiDuo You
AQE preparation gains a small rule that removes one shuffle from a
storage-partitioned join whose shuffled side aggregates or windows on the join
keys.
With spark.sql.sources.v2.bucketing.shuffle.enabled, EnsureRequirements plans
two independent shuffles on the shuffled side: a hash shuffle under the
aggregate (or window) and a keyed shuffle above it that lays the aggregate's
output out on the storage-partitioned side's keys. A keyed layout that serves
the aggregate's clustering serves it as well as the hash layout did, so the
rule retargets the hash shuffle to the keyed layout and drops the keyed shuffle
above the operator.
The walk crosses grouped aggregates, partitioned windows and final window group
limits whose keys the layout clusters, plus deterministic projections, filters
and local sorts. It is gated like KeyedShuffleSpec.canCreatePartitioning
(one-side shuffle on, partially clustered off), keeps both shuffles if the plan
reuses either, and validates the whole plan against the root distribution and
recorded user repartitions.
Scope: master. Internal config spark.sql.adaptive.pushDownKeyedShuffle.enabled,
plus an opt-in allowWithSubqueries config because a rewrite inside a subquery
can cost the query an exchange reuse.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]