[ https://issues.apache.org/jira/browse/SPARK-22472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16274042#comment-16274042 ]
Felix Cheung commented on SPARK-22472: -------------------------------------- I guess it's too late to add to http://spark.apache.org/docs/latest/sql-programming-guide.html#migration-guide (and we don't seem to document this in patch release there anyway) I guess I'll just add this to the website on the actual release announcement like http://spark.apache.org/releases/spark-release-2-1-2.html sounds good? > Datasets generate random values for null primitive types > -------------------------------------------------------- > > Key: SPARK-22472 > URL: https://issues.apache.org/jira/browse/SPARK-22472 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.1.1, 2.2.0 > Reporter: Vladislav Kuzemchik > Assignee: Wenchen Fan > Labels: release-notes > Fix For: 2.2.1, 2.3.0 > > > Not sure if it ever were reported. > {code} > scala> val s = > sc.parallelize(Seq[Option[Long]](None,Some(1L),Some(5))).toDF("v") > s: org.apache.spark.sql.DataFrame = [v: bigint] > scala> s.show(false) > +----+ > |v | > +----+ > |null| > |1 | > |5 | > +----+ > scala> s.as[Long].map(v => v*2).show(false) > +-----+ > |value| > +-----+ > |-2 | > |2 | > |10 | > +-----+ > scala> s.select($"v"*2).show(false) > +-------+ > |(v * 2)| > +-------+ > |null | > |2 | > |10 | > +-------+ > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org