[ https://issues.apache.org/jira/browse/SPARK-25413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16611759#comment-16611759 ]
Ajith S commented on SPARK-25413: --------------------------------- Thank you for reporting the issue sandeep. I think the problem is with org.apache.spark.sql.catalyst.expressions.aggregate.AverageLike#sumDataType as it increases the precision unnecessarily. Adding a PR to fix this. Refer https://github.com/apache/spark/pull/22401 > [2.3.2.rc5 Blocker] Precision Value is going for toss when Avg is done > ---------------------------------------------------------------------- > > Key: SPARK-25413 > URL: https://issues.apache.org/jira/browse/SPARK-25413 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 2.3.1 > Environment: Csv FIle content > > 1,23-07-2015,India,aaa1,phone197,ASD69643,1.23457E+16 > 2,24-07-2015,India,aaa2,phone756,ASD42892,1.23457E+16 > 3,25-07-2015,India,aaa3,phone1904,ASD37014,1.23457E+16 > 4,26-07-2015,India,aaa4,phone2435,ASD66902,1.23457E+16 > 5,27-07-2015,India,aaa5,phone2441,ASD90633,2.23457E+16 > 6,28-07-2015,India,aaa6,phone294,ASD59961,3.23457E+16 > 7,29-07-2015,India,aaa7,phone610,ASD14875,4.23457E+16 > 8,30-07-2015,India,aaa8,phone1848,ASD57308,5.23457E+16 > 9,18-07-2015,India,aaa9,phone706,ASD86717,6.23457E+16 > 10,19-07-2015,usa,aaa10,phone685,ASD30505,7.23457E+16 > 11,18-07-2015,china,aaa11,phone1554,ASD26101,8.23457E+16 > Reporter: sandeep katta > Priority: Blocker > Attachments: decimalBoundaryDataHive.csv > > > sql("create table if not exists hiveBigDecimal(ID Int, date Timestamp, > country String, name String, phonetype String, serialname String, salary > decimal(27, 10))row format delimited fields terminated by ','") > > sql(s"LOAD DATA local inpath '$resourcesPath/decimalBoundaryDataHive.csv' > INTO table hiveBigDecimal") > > sql("select avg(salary)+10 from hiveBigDecimal").show(fals > > Output with 2.3.1 > +----------------------------------------------------------------------------------------+ > |(CAST(avg(salary) AS DECIMAL(32,14)) + CAST(CAST(10 AS DECIMAL(2,0)) AS > DECIMAL(32,14)))| > +----------------------------------------------------------------------------------------+ > |37800224355780013.75982042536364 | > +----------------------------------------------------------------------------------------+ > OutPut with 2.3.2_RC5 > |(CAST(avg(salary) AS DECIMAL(32,14)) + CAST(CAST(10 AS DECIMAL(2,0)) AS > DECIMAL(32,14)))| > +----------------------------------------------------------------------------------------+ > |37800224355780013.75982042536000 > | > +---------------------------------------------------------------------------------------- > *PS:If I revert SPARK-24957 then 2.3.1 and 2.3.2_rc5 output is same* > -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org