Nullable is an optimization for Spark SQL. It is telling spark to not even do an if check when accessing that field.
In this case, your data *is* nullable, because timestamp is an object in java and you could put null there. On Thu, Aug 4, 2016 at 2:56 PM, luismattor <luismat...@gmail.com> wrote: > Hi all, > > Consider the following case: > > import java.sql.Timestamp > case class MyProduct(t: Timestamp, a: Float) > val rdd = sc.parallelize(List(MyProduct(new Timestamp(0), 10))).toDF() > rdd.printSchema() > > The output is: > root > |-- t: timestamp (nullable = true) > |-- a: float (nullable = false) > > How can I set the timestamp column to be NOT nullable? > > Regards, > Luis > > > > -- > View this message in context: http://apache-spark-user-list. > 1001560.n3.nabble.com/How-to-set-nullable-field-when- > create-DataFrame-using-case-class-tp27479.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >
