Anil K created SPARK-61812:
------------------------------

             Summary: Storage-partitioned join passes a decimal argument of 
more precision than its partition transform declares, which the write path's 
cast can change
                 Key: SPARK-61812
                 URL: https://issues.apache.org/jira/browse/SPARK-61812
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Anil K


A storage-partitioned join evaluates a connector's partition transform itself 
in the identity reduce and the one-side shuffle. Since SPARK-60027, it takes an 
argument as it is when it is stored the way the function's declared input type 
is ({{TransformExpression.storedAlike}}). For a top-level decimal, that holds 
for any precision at the declared scale, so a {{DECIMAL(12,2)}} column under a 
function declaring {{DECIMAL(10,2)}} is passed as it is.

The write path ({{DistributionAndOrderingUtils}}) lets type coercion cast the 
argument to the declared type, and both modes insert that cast 
({{TypeCoercion.implicitCast}} takes any numeric to a numeric target; 
{{AnsiTypeCoercion}} allows it by store assignment). For a value of more digits 
than the declared precision, the two then differ. Measured on {{1234567890.12}} 
as a {{DECIMAL(12,2)}} cast to {{DECIMAL(10,2)}}:
* {{spark.sql.ansi.enabled=false}}: the cast gives {{NULL}}, so the write path 
places the row by {{f(NULL)}}, while the identity reduce or a one-side shuffle 
places the other side's {{1234567890.12}} by {{f(1234567890.12)}}, and the join 
can miss the match;
* {{spark.sql.ansi.enabled=true}}: the cast throws 
{{SparkArithmeticException}}, while the storage-partitioned join evaluates the 
function on the raw value.

Values that fit the declared precision, e.g. {{12.34}}, are unchanged by the 
cast, so this only affects values that overflow it.

This is on master, and outside SPARK-60813 and SPARK-61650, which cover 
datetime and interval types, but it breaks the rule they state: a function gets 
the value the write path's cast gives. The top-level exemption exists for a 
cogroup key of more precision against a narrower bucket column (a 
{{DECIMAL(12,2)}} key against a {{DECIMAL(10,2)}} column), which a cogroup that 
does not coerce its keys can produce. Since 
https://github.com/apache/spark/pull/60431, such a key is marked on the 
transform rebuilt over it ({{TransformExpression.storedKey}}). So a fix could 
keep the exemption for a marked transform only, and require {{f.precision <= 
t.precision}} otherwise, so that a join over such a column shuffles both sides.

Raised in the review of https://github.com/apache/spark/pull/60431 
(https://github.com/apache/spark/pull/60431#discussion_r4584464865).




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to