[ https://issues.apache.org/jira/browse/SPARK-8359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14604041#comment-14604041 ]
Jihong MA commented on SPARK-8359: ---------------------------------- This fix is causing issue with divide over Decimal.Unlimited type when precision and scale are not defined as show below java.lang.ArithmeticException: Non-terminating decimal expansion; no exact representable decimal result. at java.math.BigDecimal.divide(BigDecimal.java:1616) at java.math.BigDecimal.divide(BigDecimal.java:1650) at scala.math.BigDecimal.$div(BigDecimal.scala:256) at org.apache.spark.sql.types.Decimal.$div(Decimal.scala:269) at org.apache.spark.sql.types.Decimal$DecimalIsFractional$.div(Decimal.scala:333) at org.apache.spark.sql.types.Decimal$DecimalIsFractional$.div(Decimal.scala:332) at org.apache.spark.sql.catalyst.expressions.Divide$$anonfun$div$1.apply(arithmetic.scala:193) at org.apache.spark.sql.catalyst.expressions.Divide.eval(arithmetic.scala:206) > Spark SQL Decimal type precision loss on multiplication > ------------------------------------------------------- > > Key: SPARK-8359 > URL: https://issues.apache.org/jira/browse/SPARK-8359 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 1.5.0 > Reporter: Rene Treffer > Assignee: Liang-Chi Hsieh > Fix For: 1.5.0 > > > It looks like the precision of decimal can not be raised beyond ~2^112 > without causing full value truncation. > The following code computes the power of two up to a specific point > {code} > import org.apache.spark.sql.types.Decimal > val one = Decimal(1) > val two = Decimal(2) > def pow(n : Int) : Decimal = if (n <= 0) { one } else { > val a = pow(n - 1) > a.changePrecision(n,0) > two.changePrecision(n,0) > a * two > } > (109 to 120).foreach(n => > println(pow(n).toJavaBigDecimal.unscaledValue.toString)) > 649037107316853453566312041152512 > 1298074214633706907132624082305024 > 2596148429267413814265248164610048 > 5192296858534827628530496329220096 > 1038459371706965525706099265844019 > 2076918743413931051412198531688038 > 4153837486827862102824397063376076 > 8307674973655724205648794126752152 > 1661534994731144841129758825350430 > 3323069989462289682259517650700860 > 6646139978924579364519035301401720 > 1329227995784915872903807060280344 > {code} > Beyond ~2^112 the precision is truncated even if the precision was set to n > and should thus handle 10^n without problems.. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org