Hi Paras,Check out the link Spark Scala: DateDiff of two columns by hour or
minute
|
|
|
| | |
|
|
|
| |
Spark Scala: DateDiff of two columns by hour or minute
I have two timestamp columns in a dataframe that I'd like to get the minute
difference of, or alternatively, the hour difference of. Currently I'm able to
get the day difference, with rounding, by ...
| |
|
|
Looks like you can get the difference in seconds as well.Hopefully this helps!
Are you looking for a specific usecase? Can you please elaborate with an
example?
Thanks Srabasti Banerjee
Sent from Yahoo Mail on Android
On Sun, Oct 14, 2018 at 23:41, Paras Agarwal<paras.agar...@datametica.com>
wrote: #yiv8627769989 #yiv8627769989 -- P
{margin-top:0;margin-bottom:0;}#yiv8627769989
Thanks John,
Actually need full date and time difference not just date difference,
which I guess not supported.
Let me know if its possible, or any UDF available for the same.
Thanks And Regards,
Paras
From: John Zhuge <john.zh...@gmail.com>
Sent: Friday, October 12, 2018 9:48:47 PM
To: Paras Agarwal
Cc: user; dev
Subject: Re: Timestamp Difference/operations Yeah, operator "-" does not seem
to be supported, however, you can use "datediff" function:
In [9]: select datediff(CAST('2000-02-01 12:34:34' AS TIMESTAMP),
CAST('2000-01-01 00:00:00' AS
TIMESTAMP))Out[9]:+----------------------------------------------------------------------------------------------------------------------+|
datediff(CAST(CAST(2000-02-01 12:34:34 AS TIMESTAMP) AS DATE),
CAST(CAST(2000-01-01 00:00:00 AS TIMESTAMP) AS DATE))
|+----------------------------------------------------------------------------------------------------------------------+|
31
|+----------------------------------------------------------------------------------------------------------------------+
In [10]: select datediff('2000-02-01 12:34:34', '2000-01-01
00:00:00')Out[10]:+--------------------------------------------------------------------------------+|
datediff(CAST(2000-02-01 12:34:34 AS DATE), CAST(2000-01-01 00:00:00 AS DATE))
|+--------------------------------------------------------------------------------+|
31
|+--------------------------------------------------------------------------------+
In [11]: select datediff(timestamp '2000-02-01 12:34:34', timestamp '2000-01-01
00:00:00')Out[11]:+--------------------------------------------------------------------------------------------------------------+|
datediff(CAST(TIMESTAMP('2000-02-01 12:34:34.0') AS DATE),
CAST(TIMESTAMP('2000-01-01 00:00:00.0') AS DATE))
|+--------------------------------------------------------------------------------------------------------------+|
31
|+--------------------------------------------------------------------------------------------------------------+
On Fri, Oct 12, 2018 at 7:01 AM Paras Agarwal <paras.agar...@datametica.com>
wrote:
Hello Spark Community,
Currently in hive we can do operations on Timestamp Like :
CAST('2000-01-01 12:34:34' AS TIMESTAMP) - CAST('2000-01-01 00:00:00' AS
TIMESTAMP)
Seems its not supporting in spark.
Is there any way available.
Kindly provide some insight on this.
Paras
9130006036
--
John
