Jacek Laskowski created SPARK-20299: ---------------------------------------
Summary: NullPointerException when null and string are in a tuple while encoding Dataset Key: SPARK-20299 URL: https://issues.apache.org/jira/browse/SPARK-20299 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 2.2.0 Reporter: Jacek Laskowski Priority: Minor When creating a Dataset from a tuple with {{null}} and a string, NPE is reported. When either is removed, it works fine. {code} scala> Seq((1, null.asInstanceOf[Int]), (2, 1)).toDS res43: org.apache.spark.sql.Dataset[(Int, Int)] = [_1: int, _2: int] scala> Seq(("1", null.asInstanceOf[Int]), ("2", 1)).toDS scala> Seq(("1", null.asInstanceOf[Int]), ("2", 1)).toDS java.lang.RuntimeException: Error while encoding: java.lang.NullPointerException staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, assertnotnull(assertnotnull(input[0, scala.Tuple2, true], top level Product input object), - root class: "scala.Tuple2")._1, true) AS _1#474 assertnotnull(assertnotnull(input[0, scala.Tuple2, true], top level Product input object), - root class: "scala.Tuple2")._2 AS _2#475 at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:290) at org.apache.spark.sql.SparkSession$$anonfun$2.apply(SparkSession.scala:454) at org.apache.spark.sql.SparkSession$$anonfun$2.apply(SparkSession.scala:454) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.immutable.List.foreach(List.scala:381) at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at scala.collection.immutable.List.map(List.scala:285) at org.apache.spark.sql.SparkSession.createDataset(SparkSession.scala:454) at org.apache.spark.sql.SQLContext.createDataset(SQLContext.scala:377) at org.apache.spark.sql.SQLImplicits.localSeqToDatasetHolder(SQLImplicits.scala:246) ... 48 elided Caused by: java.lang.NullPointerException at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply_1$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source) at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:287) ... 58 more {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org