[
https://issues.apache.org/jira/browse/SPARK-27768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16844351#comment-16844351
]
Dilip Biswal commented on SPARK-27768:
--------------------------------------
Hi [~smilegator] [~dongjoon],
Had started to look at this. A few observations :
1) Apart from handling of cast string to double/float, there seem to be another
discrepancy with type promotion.
In the first test case, while handling the inline table, we can't find a
common type between string and double.
Not sure if we want to do anything about it ?
2) Seems like, we need to modify the casting code to handle this i.e handle
this at the runtime.
Made a trial at
[here|https://github.com/dilipbiswal/spark/pull/new/double_infinity]
(Note: I have tweaked the first test to use int literal as opposed to
string literal to bypass (1)
Not sure what the direction we would like to take.. Just wanted to share my
findings.
Regards,
Dilip
> Infinity, -Infinity, NaN should be recognized in a case insensitive manner
> --------------------------------------------------------------------------
>
> Key: SPARK-27768
> URL: https://issues.apache.org/jira/browse/SPARK-27768
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Xiao Li
> Priority: Major
>
> When the inputs contain the constant 'infinity', Spark SQL does not generate
> the expected results.
> {code:java}
> SELECT avg(CAST(x AS DOUBLE)), var_pop(CAST(x AS DOUBLE))
> FROM (VALUES ('1'), (CAST('infinity' AS DOUBLE))) v(x);
> SELECT avg(CAST(x AS DOUBLE)), var_pop(CAST(x AS DOUBLE))
> FROM (VALUES ('infinity'), ('1')) v(x);
> SELECT avg(CAST(x AS DOUBLE)), var_pop(CAST(x AS DOUBLE))
> FROM (VALUES ('infinity'), ('infinity')) v(x);
> SELECT avg(CAST(x AS DOUBLE)), var_pop(CAST(x AS DOUBLE))
> FROM (VALUES ('-infinity'), ('infinity')) v(x);{code}
> The root cause: Spark SQL does not recognize the special constants in a case
> insensitive way. In PostgreSQL, they are recognized in a case insensitive
> way.
> Link: https://www.postgresql.org/docs/9.3/datatype-numeric.html
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
