Re: Spark SQL - Truncate Day / Hour

Hi, you can truncate datetimes like this (in pyspark), e.g. to 5 minutes: import pyspark.sql.functions as F df.select((F.floor(F.col('myDateColumn').cast('long') / 300) * 300).cast('timestamp')) Best, Eike David Hodefi schrieb am Mo., 13. Nov. 2017 um 12:27 Uhr:

Re: Spark SQL - Truncate Day / Hour

I am familiar with those functions, none of them is actually truncating a date. We can use those methods to help implement truncate method. I think truncating a day/ hour should be as simple as "truncate(...,"DD") or truncate(...,"HH") ". On Thu, Nov 9, 2017 at 8:23 PM, Gaspar Muñoz

Re: Spark SQL - Truncate Day / Hour

There are functions for day (called dayOfMonth and dayOfYear) and hour (called hour). You can view them here: https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.functions Example: import org.apache.spark.sql.functions._ val df = df.select(hour($"myDateColumn"),

Spark SQL - Truncate Day / Hour

I would like to truncate date to his day or hour. currently it is only possible to truncate MONTH or YEAR. 1.How can achieve that? 2.Is there any pull request about this issue? 3.If there is not any open pull request about this issue, what are the implications that I should be aware of when coding