[ https://issues.apache.org/jira/browse/SPARK-35265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17337571#comment-17337571 ]
Saurabh Chawla edited comment on SPARK-35265 at 4/30/21, 6:44 PM: ------------------------------------------------------------------ This is not the bug check this scenario {code:java} scala> val k = -2147483648 k: Int = -2147483648 scala> Math.abs(k) res32: Int = -2147483648 {code} The reason the max value of the int is 2147483647 but where as abs of -2147483648 will give the value 2147483648 which surpassed the value 2147483647 and is considered out of range. Try Long instead of int {code:java} scala> val k = -2147483648L k: Long = -2147483648 scala> Math.abs(k) res33: Long = 2147483648 {code} try this {code:java} df = df.withColumn("test", lit(-2147483648L)) df =df.withColumn("test1", abs(col("test"))) {code} was (Author: saurabhc100): This is not the bug check this scenario {code:java} scala> val k = -2147483648 k: Int = -2147483648 scala> Math.abs(k) res32: Int = -2147483648 {code} The reason the max value of the int is 2147483647 but where as abs of -2147483648 will give the value 2147483648 which surpassed the value 2147483647 and is considered out of range. Try Long instead of int {code:java} scala> val k = -2147483648L k: Long = -2147483648 scala> Math.abs(k) res33: Long = 2147483648 {code} try this {code:java} df = df.withColumn("test", lit(-2147483648L)) df =df.withColumn("test1", abs(col("test"))) {code} > abs return negative > ------------------- > > Key: SPARK-35265 > URL: https://issues.apache.org/jira/browse/SPARK-35265 > Project: Spark > Issue Type: Bug > Components: PySpark > Affects Versions: 3.0.1, 3.1.1 > Reporter: liuzhenjie > Priority: Major > > from pyspark.sql.functions import lit, abs, concat, hash,col > df = df.withColumn('partition_id', lit(-2147483648)) > df = df.withColumn('abs_id', abs(col('partition_id'))) > df.select('abs_id','partition_id').show() > > when the number is -2147483648,method abs return negative > +-----------+------------+ > | abs_id |partition_id | > +-----------+------------+ > |-2147483648| -2147483648| > |-2147483648| -2147483648| > |-2147483648| -2147483648| > |-2147483648| -2147483648| > |-2147483648| -2147483648| > +-----------+------------+ > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org