*As you know I have been puzzling over this issue :* *How come spark.range(100).reduce(_+_)*
*worked in earlier spark version but not with the most recent versions.* *well,* *When you first create a dataset, by default the column "id" datatype is [BigInt],* *It is a bit like a coin Long on one side and bigint on the other side.* scala> val myrange = spark.range(1,100) myrange: org.apache.spark.sql.Dataset[Long] = [id: bigint] *The Spark framework error message after parsing the reduce(_+_) method confirms this* *and moreover stresses its constraints of expecting data type long as parameter argument(s).* scala> myrange.reduce(_+_) <console>:26: error: overloaded method value reduce with alternatives: (func: org.apache.spark.api.java.function.ReduceFunction[java.lang.Long])java.lang.Long <and> (func: (java.lang.Long, java.lang.Long) => java.lang.Long)java.lang.Long cannot be applied to ((java.lang.Long, java.lang.Long) => scala.Long) myrange.reduce(_+_) ^ *But if you ask the printSchema method it disagrees with both of the above and says the column "id" data is Long.*scala> range100.printSchema() root |-- id: long (nullable = false) *If I ask the collect() method, the collect() method agrees with printSchema() that the datatype of column "id" is Long and not BigInt.* scala> range100.collect() res10: Array[Long] = Array(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99) *To settle the dispute between the methods and get the collect() to "show me the money" I called the collect() to pass its return type to reduce(_+_).* *"Here is the money"* scala> range100.collect().reduce(_+_) res11: Long = 4950 *The collect() and printSchema methods could be implying there is no difference between a Long or a BingInt.* *Questions : These return type differentials, are they by design or an oversight bug ?* *Questions : Why the change from earlier version to later version ?* *Question : Will you be updating the reduce(_+_) method ?* *When it comes to creating a dataset using toDs there is no dispute,* *all the methods agree that it is neither a BigInt or a Long but an int even integer.* scala> val dataset = Seq(1, 2, 3).toDS() dataset: org.apache.spark.sql.Dataset[Int] = [value: int] scala> dataset.collect() res29: Array[Int] = Array(1, 2, 3) scala> dataset.printSchema() root |-- value: integer (nullable = false) scala> dataset.show() +-----+ |value| +-----+ | 1| | 2| | 3| +-----+ scala> dataset.reduce(_+_) res7: Int = 6 <http://www.backbutton.co.uk/>