[
https://issues.apache.org/jira/browse/SPARK-24834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16680290#comment-16680290
]
Sean Owen commented on SPARK-24834:
-----------------------------------
The goal is to match Hive semantics, if anything. And of course to have
internally-consistent behavior. Matching previous Spark SQL behavior is of
course important too, even across major releases, but can be broken where
appropriate.
Matching another DB's semantics is fine if Hive is 'silent' on behavior. Here I
am not clear what Hive does, but if it is different, yes, it should probably be
fixed in 3.0 unless it would really cause big pain for Spark SQL workloads
today. If it isn't what Hive does, I think we have to leave it.
> Utils#nanSafeCompare{Double,Float} functions do not differ from normal java
> double/float comparison
> ---------------------------------------------------------------------------------------------------
>
> Key: SPARK-24834
> URL: https://issues.apache.org/jira/browse/SPARK-24834
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.3.2
> Reporter: Benjamin Duffield
> Priority: Minor
>
> Utils.scala contains two functions `nanSafeCompareDoubles` and
> `nanSafeCompareFloats` which purport to have special handling of NaN values
> in comparisons.
> The handling in these functions do not appear to differ from
> java.lang.Double.compare and java.lang.Float.compare - they seem to produce
> identical output to the built-in java comparison functions.
> I think it's clearer to not have these special Utils functions, and instead
> just use the standard java comparison functions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
