[ https://issues.apache.org/jira/browse/SPARK-22472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16244438#comment-16244438 ]
Kazuaki Ishizaki commented on SPARK-22472: ------------------------------------------ >From [the initial >version|https://github.com/apache/spark/commit/9e66a53c9955285a85c19f55c3ef62db2e1b868a#diff-94a1f59bcc9b6758c4ca874652437634R227] > of this conversion (about two years ago), this conversion returns {{-1}}... > Datasets generate random values for null primitive types > -------------------------------------------------------- > > Key: SPARK-22472 > URL: https://issues.apache.org/jira/browse/SPARK-22472 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.1.1, 2.2.0 > Reporter: Vladislav Kuzemchik > > Not sure if it ever were reported. > {code} > scala> val s = > sc.parallelize(Seq[Option[Long]](None,Some(1L),Some(5))).toDF("v") > s: org.apache.spark.sql.DataFrame = [v: bigint] > scala> s.show(false) > +----+ > |v | > +----+ > |null| > |1 | > |5 | > +----+ > scala> s.as[Long].map(v => v*2).show(false) > +-----+ > |value| > +-----+ > |-2 | > |2 | > |10 | > +-----+ > scala> s.select($"v"*2).show(false) > +-------+ > |(v * 2)| > +-------+ > |null | > |2 | > |10 | > +-------+ > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org