[
https://issues.apache.org/jira/browse/SPARK-60277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60277:
-----------------------------------
Labels: pull-request-available (was: )
> Merge a union's keyed partitionings when the unknown-keyed leg declares the
> other legs' keys
> --------------------------------------------------------------------------------------------
>
> Key: SPARK-60277
> URL: https://issues.apache.org/jira/browse/SPARK-60277
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: XiDuo You
> Priority: Major
> Labels: pull-request-available
>
> A `UnionExec` whose children all report a `KeyedPartitioning` merges their
> keys into one
> layout, so a storage-partitioned join or a keyed aggregate above it needs no
> exchange. Today
> that merge is given up whenever one child's layout carries
> `mayContainUnknownPartitionKeys`: such a child routes a row whose key it does
> not declare to
> the partition of some declared key, and another child may declare exactly
> that key, so the
> merged claim could promise co-location the marked child cannot honor.
> The refusal is wider than it has to be. When the marked child declares every
> key the other
> children declare, no child can claim a key the marked one routes by hash, and
> the keys no child
> declares reach the same declared key from every marked child as long as they
> declare the same
> keys in the same order. Each key's rows then share a group, which is what a
> clustering asks of
> the merged layout, and the merge can be kept with the marker carried onto it
> so pairing keeps
> reading it.
> This matters for a plan whose union leg comes out of a one-side-shuffled
> join: the leg is marked
> although its keys are accurate for every key the other legs hold, and the
> operator above the
> union pays an exchange it does not need.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
