[ https://issues.apache.org/jira/browse/SPARK-21041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wenchen Fan resolved SPARK-21041. --------------------------------- Resolution: Fixed Assignee: Dongjoon Hyun Fix Version/s: 2.3.0 2.2.1 > With whole-stage codegen, SparkSession.range()'s behavior is inconsistent > with SparkContext.range() > --------------------------------------------------------------------------------------------------- > > Key: SPARK-21041 > URL: https://issues.apache.org/jira/browse/SPARK-21041 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.0.2, 2.1.1, 2.2.0 > Reporter: Kris Mok > Assignee: Dongjoon Hyun > Fix For: 2.2.1, 2.3.0 > > > When whole-stage codegen is enabled, in face of integer overflow, > SparkSession.range()'s behavior is inconsistent with when codegen is turned > off, while the latter is consistent with SparkContext.range()'s behavior. > The following Spark Shell session shows the inconsistency: > {code:java} > scala> sc.range > def range(start: Long,end: Long,step: Long,numSlices: Int): > org.apache.spark.rdd.RDD[Long] > scala> spark.range > > > def range(start: Long,end: Long,step: Long,numPartitions: Int): > org.apache.spark.sql.Dataset[Long] > def range(start: Long,end: Long,step: Long): > org.apache.spark.sql.Dataset[Long] > def range(start: Long,end: Long): org.apache.spark.sql.Dataset[Long] > > def range(end: Long): org.apache.spark.sql.Dataset[Long] > scala> sc.range(java.lang.Long.MAX_VALUE - 3, java.lang.Long.MIN_VALUE + 2, > 1).collect > res1: Array[Long] = Array() > scala> spark.range(java.lang.Long.MAX_VALUE - 3, java.lang.Long.MIN_VALUE + > 2, 1).collect > res2: Array[Long] = Array(9223372036854775804, 9223372036854775805, > 9223372036854775806) > scala> spark.conf.set("spark.sql.codegen.wholeStage", false) > scala> spark.range(java.lang.Long.MAX_VALUE - 3, java.lang.Long.MIN_VALUE + > 2, 1).collect > res5: Array[Long] = Array() > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org