Re: How to handle Option[Int] in dataframe

2015-11-03 Thread Michael Armbrust
In Spark 1.6 there is an experimental new features called Datasets. You can call df.as[Student] and it should do what you want. Would love any feedback you have if you get a chance to try it out (we'll hopefully publish a preview release next week). On Mon, Nov 2, 2015 at 9:30 PM, manas kar

How to handle Option[Int] in dataframe

2015-11-02 Thread manas kar
Hi, I have a case class with many columns that are Option[Int] or Option[Array[Byte]] and such. I would like to save it to parquet file and later read it back to my case class too. I found that Option[Int] when null returns 0 when the field is Null. My question: Is there a way to get