Anil K created SPARK-60884:
------------------------------
Summary: Widen a DATE to TIMESTAMP_NTZ and an integral type to a
wide-enough DECIMAL when a storage-partitioned join evaluates a partition
transform
Key: SPARK-60884
URL: https://issues.apache.org/jira/browse/SPARK-60884
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Anil K
Since SPARK-60027, a storage-partitioned join evaluates a connector's partition
transform itself, in the identity reduce and the one-side shuffle, only when
each argument suits the type the function declares. An argument stored the way
the declared type is goes as it is, and a narrower numeric one is widened with
a {{Cast}} ({{UpCastRule.legalNumericPrecedence}}). Anything else is refused,
so the join shuffles both sides.
Two more casts are lossless and need no time zone ({{Cast.needsTimeZone}} is
false for both), but are still refused:
* a {{DATE}} against a declared {{TIMESTAMP_NTZ}}, which is midnight of that
day, on a fixed grid;
* an integral type against a declared {{DECIMAL}} wide enough for it (the
{{to.isWiderThan(from)}} arm of {{UpCastRule.canUpCast}}), e.g. an {{INT}}
against a {{DECIMAL(10, 0)}}.
Master failed on both, since the physical types differ, so a join over such a
layout never kept its plan. Casting them would let it, as the analyzer's
implicit cast does for a SQL call of the same function.
{{canUpCast(argumentType, declaredType) && !Cast.needsTimeZone(argumentType,
declaredType)}} would cover both, but is too broad, since {{canUpCast}} also
admits any atomic type to {{STRING}}. Two explicit arms in
{{TransformExpression.atDeclaredType}} are simpler.
The {{DATE}} against {{TIMESTAMP_NTZ}} arm is a cast, not a pass-through, so it
fits SPARK-60813, which refuses passing a {{DATE}} as is to a function
declaring a type of another unit.
Raised in the review of SPARK-60027:
https://github.com/apache/spark/pull/59760#discussion_r4463685243
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
