Ohad Raviv created SPARK-26070:
----------------------------------
Summary: another implicit type coercion bug
Key: SPARK-26070
URL: https://issues.apache.org/jira/browse/SPARK-26070
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.4.0, 2.3.0
Reporter: Ohad Raviv
looks like SPARK-22469 introduced a major bug to our system:
{code}
spark.sql("select '22222222222222222222' = 22222222222222222222BD").show()
spark.sql("select '22222222222222222224' = 22222222222222222223BD").show()
{code}
which results:
{noformat}
+-----------------------------------------------------------------------------+
|(CAST(22222222222222222222 AS DOUBLE) = CAST(22222222222222222222 AS DOUBLE))|
+-----------------------------------------------------------------------------+
| true|
+-----------------------------------------------------------------------------+
+-----------------------------------------------------------------------------+
|(CAST(22222222222222222224 AS DOUBLE) = CAST(22222222222222222223 AS DOUBLE))|
+-----------------------------------------------------------------------------+
| true|
+-----------------------------------------------------------------------------+
{noformat}
this causes downstream transformations to join together unrelated rows just
because their ids are close.
[~cloud_fan],[~liutang123] - could you please explain futher this remark:
{noformat}
// There is no proper decimal type we can pick,
// using double type is the best we can do.
// See SPARK-22469 for details.
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
