Anil K created SPARK-61654:
------------------------------
Summary: Place a cogroup key stored like the layout's column by
the column's own transform in the one-side shuffle
Key: SPARK-61654
URL: https://issues.apache.org/jira/browse/SPARK-61654
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Anil K
When a storage-partitioned layout has a cogroup side shuffled onto it
({{spark.sql.sources.v2.bucketing.shuffle.enabled=true}}),
{{KeyedShuffleSpec.createPartitioning}} rebuilds the layout's transform over
the other side's key and asks whether Spark can evaluate the function on that
key's type. For a cogroup that does not coerce its keys (e.g. a Pandas or Arrow
cogroup), that is a stricter question than placement needs: the cogroup pairs
groups by comparing key bytes, so a shuffled row only has to land where the
keyed side put rows with the same bytes, which is the column's own transform on
those bytes.
So since SPARK-60813 and SPARK-61650
(https://github.com/apache/spark/pull/60431), a cogroup whose key is stored
like the layout's column but is a datetime type the function would read
differently, e.g. a {{TIME}} or {{TIMESTAMP_NTZ}} key onto a bucket over a
{{TIMESTAMP}} column whose function declares {{TIMESTAMP}}, fails while
planning with {{STORAGE_PARTITION_JOIN_KEY_TYPE_MISMATCH}}. The hash shuffle,
with the flag off, runs the same query and pairs equal bytes.
A fix: when the key passes the storage check
({{TransformExpression.keysStoredAlike}}) and is an attribute, rebuild the
transform over that attribute retyped as the column's type. The shuffle then
reads the key's bytes as the column's type, the call is the layout's own (which
{{canEvaluatePartitioning}} already asserts), and the pairing equals the hash
shuffle's for every key the storage check takes. A join's keys share the
column's type, so joins do not change; other keys keep the current path. The
error then fires only for keys stored otherwise, as for an {{INT}} key onto a
{{BIGINT}} column.
This touches the code SPARK-50593 (https://github.com/apache/spark/pull/55885)
changes: the rebuild has to replace only the transform's column slots
({{rewriteColumnSlots}}), not its literal parameters. So it should follow
SPARK-50593. If SPARK-60782 instead rejects cogroup keys of different types at
analysis, this becomes moot for those types.
Found in the review of https://github.com/apache/spark/pull/60431.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]