XiDuo You created SPARK-60277:
---------------------------------
Summary: Merge a union's keyed partitionings when the
unknown-keyed leg declares the other legs' keys
Key: SPARK-60277
URL: https://issues.apache.org/jira/browse/SPARK-60277
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: XiDuo You
A `UnionExec` whose children all report a `KeyedPartitioning` merges their keys
into one
layout, so a storage-partitioned join or a keyed aggregate above it needs no
exchange. Today
that merge is given up whenever one child's layout carries
`mayContainUnknownPartitionKeys`: such a child routes a row whose key it does
not declare to
the partition of some declared key, and another child may declare exactly that
key, so the
merged claim could promise co-location the marked child cannot honor.
The refusal is wider than it has to be. When the marked child declares every
key the other
children declare, no child can claim a key the marked one routes by hash, and
the keys no child
declares reach the same declared key from every marked child as long as they
declare the same
keys in the same order. Each key's rows then share a group, which is what a
clustering asks of
the merged layout, and the merge can be kept with the marker carried onto it so
pairing keeps
reading it.
This matters for a plan whose union leg comes out of a one-side-shuffled join:
the leg is marked
although its keys are accurate for every key the other legs hold, and the
operator above the
union pays an exchange it does not need.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
