[ https://issues.apache.org/jira/browse/SPARK-24468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yifei Wu updated SPARK-24468: ----------------------------- Description: Hi, I am using MySQL JDBC Driver along with Spark to do some sql queries. When multiplying a LongType with a decimal in scientific notation, say {code:java} spark.sql("select some_int * 2.34E10 from t"){code} , decimal 2.34E10 will be treated as decimal(3,-8), and some_int will be casted as decimal(20,0). So according to the rules in comments: {code:java} /* * Operation Result Precision Result Scale * ------------------------------------------------------------------------ * e1 + e2 max(s1, s2) + max(p1-s1, p2-s2) + 1 max(s1, s2) * e1 - e2 max(s1, s2) + max(p1-s1, p2-s2) + 1 max(s1, s2) * e1 * e2 p1 + p2 + 1 s1 + s2 * e1 / e2 p1 - s1 + s2 + max(6, s1 + p2 + 1) max(6, s1 + p2 + 1) * e1 % e2 min(p1-s1, p2-s2) + max(s1, s2) max(s1, s2) * e1 union e2 max(s1, s2) + max(p1-s1, p2-s2) max(s1, s2) */ {code} their multiplication will be decimal(3+20+1,-8+0) and thus fails the assert assumption (scale>=0) on DecimalType.scala:166. My current workaround is to set spark.sql.decimalOperations.allowPrecisionLoss to false. was: Hi, I am using MySQL JDBC Driver along with Spark to do some sql queries. When multiplying a LongType with a decimal in scientific notation, say {code:java} spark.sql("select some_int * 2.34E10 from t").show{code} , decimal 2.34E10 will be treated as decimal(3,-8), and some_int will be casted as decimal(20,0). So according to the rules in comments: {code:java} /* * Operation Result Precision Result Scale * ------------------------------------------------------------------------ * e1 + e2 max(s1, s2) + max(p1-s1, p2-s2) + 1 max(s1, s2) * e1 - e2 max(s1, s2) + max(p1-s1, p2-s2) + 1 max(s1, s2) * e1 * e2 p1 + p2 + 1 s1 + s2 * e1 / e2 p1 - s1 + s2 + max(6, s1 + p2 + 1) max(6, s1 + p2 + 1) * e1 % e2 min(p1-s1, p2-s2) + max(s1, s2) max(s1, s2) * e1 union e2 max(s1, s2) + max(p1-s1, p2-s2) max(s1, s2) */ {code} their multiplication will be decimal(3+20+1,-8+0) and thus fails the assert assumption (scale>=0) on DecimalType.scala:166. My current workaround is to set spark.sql.decimalOperations.allowPrecisionLoss to false. > DecimalType `adjustPrecisionScale` might fail when scale is negative > -------------------------------------------------------------------- > > Key: SPARK-24468 > URL: https://issues.apache.org/jira/browse/SPARK-24468 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.3.0 > Reporter: Yifei Wu > Priority: Minor > > Hi, I am using MySQL JDBC Driver along with Spark to do some sql queries. > When multiplying a LongType with a decimal in scientific notation, say > {code:java} > spark.sql("select some_int * 2.34E10 from t"){code} > , decimal 2.34E10 will be treated as decimal(3,-8), and some_int will be > casted as decimal(20,0). > > So according to the rules in comments: > {code:java} > /* > * Operation Result Precision Result Scale > * ------------------------------------------------------------------------ > * e1 + e2 max(s1, s2) + max(p1-s1, p2-s2) + 1 max(s1, s2) > * e1 - e2 max(s1, s2) + max(p1-s1, p2-s2) + 1 max(s1, s2) > * e1 * e2 p1 + p2 + 1 s1 + s2 > * e1 / e2 p1 - s1 + s2 + max(6, s1 + p2 + 1) max(6, s1 + p2 + 1) > * e1 % e2 min(p1-s1, p2-s2) + max(s1, s2) max(s1, s2) > * e1 union e2 max(s1, s2) + max(p1-s1, p2-s2) max(s1, s2) > */ > {code} > their multiplication will be decimal(3+20+1,-8+0) and thus fails the assert > assumption (scale>=0) on DecimalType.scala:166. > > My current workaround is to set > spark.sql.decimalOperations.allowPrecisionLoss to false. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org