spark datatypes

2017-12-03 Thread David Hodefi
Looking on the source code, it seems like DateDataType is Int. "class DateType private() extends AtomicType { // The companion object and this class is separated so the companion object also subclasses // this type. Otherwise, the companion object would be of type "DateType$" in byte code.

Re: Spark SQL - Truncate Day / Hour

2017-11-13 Thread David Hodefi
gt; > import org.apache.spark.sql.functions._ > val df = df.select(hour($"myDateColumn"), dayOfMonth($"myDateColumn"), > dayOfYear($"myDateColumn")) > > 2017-11-09 12:05 GMT+01:00 David Hodefi <davidhodeffi.w...@gmail.com>: > >> I would like to

Spark SQL - Truncate Day / Hour

2017-11-09 Thread David Hodefi
I would like to truncate date to his day or hour. currently it is only possible to truncate MONTH or YEAR. 1.How can achieve that? 2.Is there any pull request about this issue? 3.If there is not any open pull request about this issue, what are the implications that I should be aware of when coding