XiDuo You created SPARK-61860:
---------------------------------

             Summary: Push a keyed shuffle down through clustered operators in 
AQE preparation
                 Key: SPARK-61860
                 URL: https://issues.apache.org/jira/browse/SPARK-61860
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: XiDuo You


AQE preparation gains a small rule that removes one shuffle from a 
storage-partitioned join whose shuffled side aggregates or windows on the join 
keys.

With spark.sql.sources.v2.bucketing.shuffle.enabled, EnsureRequirements plans 
two independent shuffles on the shuffled side: a hash shuffle under the 
aggregate (or window) and a keyed shuffle above it that lays the aggregate's 
output out on the storage-partitioned side's keys. A keyed layout that serves 
the aggregate's clustering serves it as well as the hash layout did, so the 
rule retargets the hash shuffle to the keyed layout and drops the keyed shuffle 
above the operator.

The walk crosses grouped aggregates, partitioned windows and final window group 
limits whose keys the layout clusters, plus deterministic projections, filters 
and local sorts. It is gated like KeyedShuffleSpec.canCreatePartitioning 
(one-side shuffle on, partially clustered off), keeps both shuffles if the plan 
reuses either, and validates the whole plan against the root distribution and 
recorded user repartitions.

Scope: master. Internal config spark.sql.adaptive.pushDownKeyedShuffle.enabled, 
plus an opt-in allowWithSubqueries config because a rewrite inside a subquery 
can cost the query an exchange reuse.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to