[ https://issues.apache.org/jira/browse/SPARK-41207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17636596#comment-17636596 ]
Apache Spark commented on SPARK-41207: -------------------------------------- User 'ulysses-you' has created a pull request for this issue: https://github.com/apache/spark/pull/38739 > Regression in IntegralDivide > ---------------------------- > > Key: SPARK-41207 > URL: https://issues.apache.org/jira/browse/SPARK-41207 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 3.4.0 > Reporter: Raza Jafri > Priority: Minor > > There has been a regression in Integral Divide after the removal of > PromotePrecision from Spark 3.4.0. > {code:java} > scala> val data = Seq(Row(BigDecimal("-7.70892"), > BigDecimal("4.27138661282262736522411173299611831E+40"))) > scala> val simpleSchema = StructType(Array( > | StructField("a", DecimalType(6,5),true), > | StructField("b", DecimalType(36,-5), true))) > scala> val df = spark.createDataFrame(spark.sparkContext.parallelize(data), > simpleSchema) > {code} > The above statements result in an AnalysisException thrown > > {code:java} > org.apache.spark.sql.AnalysisException: Decimal scale (0) cannot be greater > than precision (-4). > at > org.apache.spark.sql.errors.QueryCompilationErrors$.decimalCannotGreaterThanPrecisionError(QueryCompilationErrors.scala:2237) > at org.apache.spark.sql.types.DecimalType.<init>(DecimalType.scala:49) > at org.apache.spark.sql.types.DecimalType$.bounded(DecimalType.scala:164) > at > org.apache.spark.sql.catalyst.expressions.IntegralDivide.resultDecimalType(arithmetic.scala:868) > at > org.apache.spark.sql.catalyst.expressions.BinaryArithmetic.dataType(arithmetic.scala:238) > at > org.apache.spark.sql.catalyst.expressions.IntegralDivide.org$apache$spark$sql$catalyst$expressions$DivModLike$$super$dataType(arithmetic.scala:842) > {code} > I believe this is happening because we aren't promoting the precision like we > were before this > [PR|https://github.com/apache/spark/commit/301a13963808d1ad44be5cacf0a20f65b853d5a2] > went in. Without promoting precision the resultDecimalType in the example > above tries to return a Decimal with precision of -4 and scale of 0 which is > invalid -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org