[
https://issues.apache.org/jira/browse/SPARK-29600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16976319#comment-16976319
]
Udbhav Agrawal edited comment on SPARK-29600 at 11/18/19 6:56 AM:
------------------------------------------------------------------
Hi [~hyukjin.kwon], this failure is because we cannot cast the literal to array
type after the above behavior change. For example:
array(0.1,0.2,0.33) is type decimal(2,2) and literal 0.1 and 0.2 is also
changed to decimal(2,2) but if we check 0.2 which actually is of type
decimal(1,1) this query fails its data type doesn't match with array's data
type.
was (Author: udbhav agrawal):
Hi [~hyukjin.kwon], this failure is because we cannot after the above behavior
chnage spark doesn't cast the literal to array type . For example:
> array_contains built in function is not backward compatible in 3.0
> ------------------------------------------------------------------
>
> Key: SPARK-29600
> URL: https://issues.apache.org/jira/browse/SPARK-29600
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: ABHISHEK KUMAR GUPTA
> Priority: Major
>
> SELECT array_contains(array(0,0.1,0.2,0.3,0.5,0.02,0.033), .2); throws
> Exception in 3.0 where as in 2.3.2 is working fine.
> Spark 3.0 output:
> 0: jdbc:hive2://10.18.19.208:23040/default> SELECT
> array_contains(array(0,0.1,0.2,0.3,0.5,0.02,0.033), .2);
> Error: org.apache.spark.sql.AnalysisException: cannot resolve
> 'array_contains(array(CAST(0 AS DECIMAL(13,3)), CAST(0.1BD AS DECIMAL(13,3)),
> CAST(0.2BD AS DECIMAL(13,3)), CAST(0.3BD AS DECIMAL(13,3)), CAST(0.5BD AS
> DECIMAL(13,3)), CAST(0.02BD AS DECIMAL(13,3)), CAST(0.033BD AS
> DECIMAL(13,3))), 0.2BD)' due to data type mismatch: Input to function
> array_contains should have been array followed by a value with same element
> type, but it's [array<decimal(13,3)>, decimal(1,1)].; line 1 pos 7;
> 'Project [unresolvedalias(array_contains(array(cast(0 as decimal(13,3)),
> cast(0.1 as decimal(13,3)), cast(0.2 as decimal(13,3)), cast(0.3 as
> decimal(13,3)), cast(0.5 as decimal(13,3)), cast(0.02 as decimal(13,3)),
> cast(0.033 as decimal(13,3))), 0.2), None)]
> Spark 2.3.2 output
> 0: jdbc:hive2://10.18.18.214:23040/default> SELECT
> array_contains(array(0,0.1,0.2,0.3,0.5,0.02,0.033), .2);
> |array_contains(array(CAST(0 AS DECIMAL(13,3)), CAST(0.1 AS DECIMAL(13,3)),
> CAST(0.2 AS DECIMAL(13,3)), CAST(0.3 AS DECIMAL(13,3)), CAST(0.5 AS
> DECIMAL(13,3)), CAST(0.02 AS DECIMAL(13,3)), CAST(0.033 AS DECIMAL(13,3))),
> CAST(0.2 AS DECIMAL(13,3)))|
> |true|
> 1 row selected (0.18 seconds)
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
