Hi, I have a case class with many columns that are Option[Int] or Option[Array[Byte]] and such. I would like to save it to parquet file and later read it back to my case class too. I found that Option[Int] when null returns 0 when the field is Null. My question: Is there a way to get Option[Int] from a row instead of Int from a dataframe?
...Manas Some more description /*My case class*/ case class Student(name: String, age: Option[Int]) val s = new Student("Manas",Some(35)) val s1 = new Student("Manas1",None) val student =sc.makeRDD(List(s,s1)).toDF /*Now writing the dataframe*/ student.write.parquet("/tmp/t1") /*Lets read it back*/ val st1 = sqlContext.read.parquet("/tmp/t1") st1.show +------+----+ | name| age| +------+----+ | Manas| 35| |Manas1|null| +------+----+ But now I want to cast my dataframe to the dataframe[Student]. What is the easiest way to do it? ..Manas