Anil K created SPARK-61650:
------------------------------

             Summary: Storage-partitioned join passes a datetime or interval 
value that its cast to the declared type would change by time zone or precision
                 Key: SPARK-61650
                 URL: https://issues.apache.org/jira/browse/SPARK-61650
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Anil K


A storage-partitioned join evaluates a connector's partition transform itself 
in the identity reduce and the one-side shuffle. Since SPARK-60027, it takes an 
argument as it is when it is stored the way the function's declared input type 
is, by comparing their physical types. SPARK-60813 refuses two datetime or 
interval types that count in different units. Two more pairs share a physical 
type and a unit, but the function still gets another value than Spark's cast to 
the declared type gives, which is what the write path 
({{DistributionAndOrderingUtils}}) passes:
* *another time zone reference*: {{TIMESTAMP_NTZ}} holds a local time, which 
the cast to {{TIMESTAMP}} converts with the session time zone. In 
{{America/Los_Angeles}}, the transform got 3600000000 (01:00) where the write 
path passes 32400000000 (09:00 UTC);
* *a finer precision or end field*: the cast from {{TIME(9)}} to {{TIME(6)}} 
truncates to microseconds, and one from a {{DAY TO SECOND}} interval to a {{DAY 
TO HOUR}} one to hours, while the transform got the untruncated value, e.g. 
3600123456789 where the write path passes 3600123456000 (with 
{{spark.sql.ansi.enabled=false}}).

Rows of the shuffled side can then land in the wrong partitions and join 
matches are silently lost.

A fix: let two datetime or interval types go as they are only where Spark's 
cast between them leaves the value unchanged, i.e. the same class at no finer a 
precision or end field, e.g. {{TIME(3)}} against {{TIME(6)}}, or a {{DAY TO 
HOUR}} interval against a {{DAY TO SECOND}} one, and refuse the rest, so the 
join shuffles both sides.

SPARK-60813 kept {{TIMESTAMP_NTZ}} against {{TIMESTAMP}}, on the grounds that 
the connector computes its keys without the session time zone. Refusing it is 
safe either way: if the connector's keys come from the raw value, the join only 
shuffles both sides; if they come from the cast, as Spark's write path computes 
them, it avoids wrong results.

Found in the review of SPARK-60813 (https://github.com/apache/spark/pull/60431).




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to