Thomas Graves created SPARK-32640: ------------------------------------- Summary: Spark 3.1 log(NaN) returns null instead of NaN Key: SPARK-32640 URL: https://issues.apache.org/jira/browse/SPARK-32640 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 3.1.0 Reporter: Thomas Graves
I was testing Spark 3.1.0 and I noticed that if you take the log(NaN) it now returns a null whereas in Spark 3.0 it returned a NaN. I'm not an expert in this but I thought NaN was correct. Spark 3.1.0 Example: >>> df.selectExpr(["value", "log1p(value)"]).show() +-------------+------------------+ | value| LOG1P(value)| +-------------+------------------+ |-3.4028235E38| null| | 3.4028235E38| 88.72283906194683| | 0.0| 0.0| | -0.0| -0.0| | 1.0|0.6931471805599453| | -1.0| null| | NaN| null| +-------------+------------------+ Spark 3.0.0 example: +-------------+-----------------+ | value| LOG(E(), value)| +-------------+-----------------+ |-3.4028235E38| null| | 3.4028235E38|88.72283906194683| | 0.0| null| | -0.0| null| | 1.0| 0.0| | -1.0| null| | NaN| NaN| +-------------+-----------------+ Note it also does the same for log1p, log2, log10 -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org