hi, desperately trying to extract hour from unix seconds year, month, dayofmonth functions work as expected. hour function always returns 0.
val ds = dataset .withColumn("year", year(to_date(from_unixtime(dataset.col("ts") / 1000)))) .withColumn("month", month(to_date(from_unixtime(dataset.col("ts") / 1000)))) .withColumn("day", dayofmonth(to_date(from_unixtime(dataset.col("ts") / 1000)))) .withColumn("hour", hour(from_utc_timestamp(dataset.col("ts") / 1000, "UTC"))) //.withColumn("hour", hour(dataset.col("ts") / 1000)) //.withColumn("hour1", hour(dataset.col("ts"))) //.withColumn("hour", hour(dataset.col("ts"))) //.withColumn("hour", hour("2009-07-30 12:58:59")) I took a look at source code year, month, dayofmonth expect to get override def inputTypes: Seq[AbstractDataType] = Seq(DateType) hour function expects something different override def inputTypes: Seq[AbstractDataType] = Seq(TimestampType) from_utc_timestamp returns Timestamp override def dataType: DataType = TimestampType but It didn't help What do I do wrong? how can I get hour from unix seconds? Thanks!