[ 
https://issues.apache.org/jira/browse/SPARK-61812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anil K updated SPARK-61812:
---------------------------
    Affects Version/s: 4.3.0
                           (was: 5.0.0)

> Storage-partitioned join passes a decimal argument of more precision than its 
> partition transform declares, which the write path's cast can change
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61812
>                 URL: https://issues.apache.org/jira/browse/SPARK-61812
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 4.3.0
>            Reporter: Anil K
>            Priority: Major
>
> A storage-partitioned join evaluates a connector's partition transform itself 
> in the identity reduce and the one-side shuffle. Since SPARK-60027, it takes 
> an argument as it is when it is stored the way the function's declared input 
> type is ({{TransformExpression.storedAlike}}). For a top-level decimal, that 
> holds for any precision at the declared scale, so a {{DECIMAL(12,2)}} column 
> under a function declaring {{DECIMAL(10,2)}} is passed as it is.
> The write path ({{DistributionAndOrderingUtils}}) lets type coercion cast the 
> argument to the declared type, and both modes insert that cast 
> ({{TypeCoercion.implicitCast}} takes any numeric to a numeric target; 
> {{AnsiTypeCoercion}} allows it by store assignment). For a value of more 
> digits than the declared precision, the two then differ. Measured on 
> {{1234567890.12}} as a {{DECIMAL(12,2)}} cast to {{DECIMAL(10,2)}}:
> * {{spark.sql.ansi.enabled=false}}: the cast gives {{NULL}}, so the write 
> path places the row by {{f(NULL)}}, while the identity reduce or a one-side 
> shuffle places the other side's {{1234567890.12}} by {{f(1234567890.12)}}, 
> and the join can miss the match;
> * {{spark.sql.ansi.enabled=true}}: the cast throws 
> {{SparkArithmeticException}}, while the storage-partitioned join evaluates 
> the function on the raw value.
> Values that fit the declared precision, e.g. {{12.34}}, are unchanged by the 
> cast, so this only affects values that overflow it.
> This is on master, and outside SPARK-60813 and SPARK-61650, which cover 
> datetime and interval types, but it breaks the rule they state: a function 
> gets the value the write path's cast gives. The top-level exemption exists 
> for a cogroup key of more precision against a narrower bucket column (a 
> {{DECIMAL(12,2)}} key against a {{DECIMAL(10,2)}} column), which a cogroup 
> that does not coerce its keys can produce. Since 
> https://github.com/apache/spark/pull/60431, such a key is marked on the 
> transform rebuilt over it ({{TransformExpression.storedKey}}). So a fix could 
> keep the exemption for a marked transform only, and require {{f.precision <= 
> t.precision}} otherwise, so that a join over such a column shuffles both 
> sides.
> Raised in the review of https://github.com/apache/spark/pull/60431 
> (https://github.com/apache/spark/pull/60431#discussion_r4584464865).



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to