[
https://issues.apache.org/jira/browse/SPARK-28519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16894524#comment-16894524
]
Sean Owen commented on SPARK-28519:
-----------------------------------
A summary of the dev@ discussion:
This is almost surely because the java.lang.Math by default defers to
java.lang.StrictMath implementations, but in the Oracle JDKs, calls some x86
intrinsic for things like log() and pow(). That is, I think we'd find this
happens in OpenJDK too, but haven't confirmed.
atanh isn't implemented in java.lang.Math, but uses log(): {{0.5 * math.log((1
+ x) / (1 - x))}}. While we can 'fix' the variation by directly using
StrictMath.log(), we can also just use a different formula that should also
happen to be slightly more accurate for small x: {{0.5 * ((math.log1p(x) -
math.log1p(-x))}}. I suggest this.
The other result involving pow() is just a more direct illustration of the
platform variation. There we can't just adjust a formula. We can choose to
implement pow() with StrictMath.pow() directly. This makes the result the same
across platforms. It might cost a little performance, but this probably is
trivial in Spark. The only downside is one would find that Spark's pow() may
then return a 'right' answer but not the same one you'd find, exactly, by just
calling pow() in Java!
Really then the same argument goes for log(). It's odd that log(3) for example
gives a different result on different platforms (on Spark and in the JVM). It
feels like we should require StrictMath for this.
> Tests failed on aarch64 due the value of math.log and power function is
> different
> ---------------------------------------------------------------------------------
>
> Key: SPARK-28519
> URL: https://issues.apache.org/jira/browse/SPARK-28519
> Project: Spark
> Issue Type: Test
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: huangtianhua
> Priority: Major
>
> Sorry to disturb again, we ran unit tests on arm64 instance, and there are
> other sql tests failed:
> {code}
> - pgSQL/float8.sql *** FAILED ***
> Expected "{color:#f691b2}0.549306144334054[9]{color}", but got
> "{color:#f691b2}0.549306144334054[8]{color}" Result did not match for query
> #56
> SELECT atanh(double('0.5')) (SQLQueryTestSuite.scala:362)
> - pgSQL/numeric.sql *** FAILED ***
> Expected "2 {color:#59afe1}2247902679199174[72{color}
> 224790267919917955.1326161858
> 4 7405685069595001 7405685069594999.0773399947
> 5 5068226527.321263 5068226527.3212726541
> 6 281839893606.99365 281839893606.9937234336
> 7 {color:#d04437}1716699575118595840{color} 1716699575118597095.4233081991
> 8 167361463828.0749 167361463828.0749132007
> 9 {color:#14892c}107511333880051856]{color} 107511333880052007....", but got
> "2 {color:#59afe1}2247902679199174[40{color} 224790267919917955.1326161858
> 4 7405685069595001 7405685069594999.0773399947
> 5 5068226527.321263 5068226527.3212726541
> 6 281839893606.99365 281839893606.9937234336
> 7 {color:#d04437}1716699575118595580{color} 1716699575118597095.4233081991
> 8 167361463828.0749 167361463828.0749132007
> 9 {color:#14892c}107511333880051872]{color} 107511333880052007...." Result
> did not match for query #496
> SELECT t1.id1, t1.result, t2.expected
> FROM num_result t1, num_exp_power_10_ln t2
> WHERE t1.id1 = t2.id
> AND t1.result != t2.expected (SQLQueryTestSuite.scala:362)
> {code}
> The first test failed, because the value of math.log(3.0) is different on
> aarch64:
> # on x86_64:
> {code}
> scala> math.log(3.0)
> res50: Double = 1.0986122886681098
> {code}
> # on aarch64:
> {code}
> scala> math.log(3.0)
> res19: Double = 1.0986122886681096
> {code}
> And I tried {{math.log(4.0)}}, {{math.log(5.0)}} and they are same, I don't
> know why {{math.log(3.0)}} is so special? But the result is different indeed
> on aarch64.
> The second test failed, because some values of pow() is different on aarch64,
> according to the test, I took tests on aarch64 and x86_64, take '-83028485'
> as example:
> # on x86_64:
> {code}
> scala> import java.lang.Math._
> import java.lang.Math._
> scala> abs(-83028485)
> res3: Int = 83028485
> scala> var a = -83028485
> a: Int = -83028485
> scala> abs(a)
> res4: Int = 83028485
> scala> math.log(abs(a))
> res5: Double = 18.234694299654787
> scala> pow(10, math.log(abs(a)))
> res6: Double ={color:#d04437} 1.71669957511859584E18{color}
> {code}
> # on aarch64:
> {code}
> scala> var a = -83028485
> a: Int = -83028485
> scala> abs(a)
> res38: Int = 83028485
> scala> math.log(abs(a))
> res39: Double = 18.234694299654787
> scala> pow(10, math.log(abs(a)))
> res40: Double = 1.71669957511859558E18
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
