[ https://issues.apache.org/jira/browse/SPARK-22472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16244310#comment-16244310 ]
Marco Gaido edited comment on SPARK-22472 at 11/8/17 4:58 PM: -------------------------------------------------------------- Two things: 1 - if you use {{as\[Option\[Long\]\]}}, it works fine; 2 - actually if you collect the Dataset, the value for {{null}} is {{0}}, but with the transformation, there is this bad result. Then I am not sure about the right approach here. Because maybe the best thing would be to force using {{Option\[Long\]}} when the value is {{nullable}}, but this might be too restrictive and may break compatibility. was (Author: mgaido): Two things: 1 - if you use `as[Option[Long]]`, it works fine; 2 - actually if you collect the Dataset, the value for `null` is `0`, but with the transformation, there is this bad result. Then I am not sure about the right approach here. Because maybe the best thing would be to force using `Option[Long]` when the value is `nullable`, but this might be too restrictive and may break compatibility. > Datasets generate random values for null primitive types > -------------------------------------------------------- > > Key: SPARK-22472 > URL: https://issues.apache.org/jira/browse/SPARK-22472 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.1.1, 2.2.0 > Reporter: Vladislav Kuzemchik > > Not sure if it ever were reported. > {code} > scala> val s = > sc.parallelize(Seq[Option[Long]](None,Some(1L),Some(5))).toDF("v") > s: org.apache.spark.sql.DataFrame = [v: bigint] > scala> s.show(false) > +----+ > |v | > +----+ > |null| > |1 | > |5 | > +----+ > scala> s.as[Long].map(v => v*2).show(false) > +-----+ > |value| > +-----+ > |-2 | > |2 | > |10 | > +-----+ > scala> s.select($"v"*2).show(false) > +-------+ > |(v * 2)| > +-------+ > |null | > |2 | > |10 | > +-------+ > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org