[ https://issues.apache.org/jira/browse/SPARK-20341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974698#comment-15974698 ]
Paul Zaczkieiwcz commented on SPARK-20341: ------------------------------------------ Thanks! I saw {{scala.math.BigInt}} and {{java.math.BigInteger}} when looking through {{ScalaReflection.schemaFor}}. I couldn't use a {{Dataset\[java.math.BigDecimal\]}} because that is always precision(38,18). I figured that getting the {{BigInt}} variants to work would be easier than getting {{BigDecimal}} to work with a non-default precision. > Support BigIngeger values > 19 precision > ---------------------------------------- > > Key: SPARK-20341 > URL: https://issues.apache.org/jira/browse/SPARK-20341 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.0.2, 2.1.0 > Reporter: Paul Zaczkieiwcz > > If you create a {{Dataset\[scala.math.BigInt\]}}, then you can't have a > precision > 19. > {code} > scala> case class BigIntWrapper(value:scala.math.BigInt) > defined class BigIntWrapper > scala> val longDf = > spark.createDataset(BigIntWrapper(scala.math.BigInt("10000000000000000002"))::Nil) > 17/04/14 19:45:15 main INFO CodeGenerator: Code generated in 211.949738 ms > java.lang.RuntimeException: Error while encoding: > java.lang.IllegalArgumentException: requirement failed: BigInteger > 10000000000000000002 too large for decimal > staticinvoke(class org.apache.spark.sql.types.Decimal$, DecimalType(38,0), > apply, assertnotnull(input[0, BigIntWrapper, true], top level non-flat input > object).value, true) AS value#16 > +- staticinvoke(class org.apache.spark.sql.types.Decimal$, DecimalType(38,0), > apply, assertnotnull(input[0, BigIntWrapper, true], top level non-flat input > object).value, true) > +- assertnotnull(input[0, BigIntWrapper, true], top level non-flat input > object).value > +- assertnotnull(input[0, BigIntWrapper, true], top level non-flat > input object) > +- input[0, BigIntWrapper, true] > at > org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:280) > at > org.apache.spark.sql.SparkSession$$anonfun$3.apply(SparkSession.scala:421) > at > org.apache.spark.sql.SparkSession$$anonfun$3.apply(SparkSession.scala:421) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at scala.collection.immutable.List.foreach(List.scala:381) > at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) > at scala.collection.immutable.List.map(List.scala:285) > at org.apache.spark.sql.SparkSession.createDataset(SparkSession.scala:421) > ... 54 elided > Caused by: java.lang.IllegalArgumentException: requirement failed: BigInteger > 10000000000000000002 too large for decimal > at scala.Predef$.require(Predef.scala:224) > at org.apache.spark.sql.types.Decimal.set(Decimal.scala:137) > at org.apache.spark.sql.types.Decimal$.apply(Decimal.scala:419) > at org.apache.spark.sql.types.Decimal.apply(Decimal.scala) > at > org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown > Source) > at > org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:277) > ... 62 more > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org