Re: 1.6.0 spark.sql datetime conversion problem

2016-03-05 Thread Jan Štěrba
I dont know whats wrong but I can suggest looking up the source of the UDF
and debugging from there. I would think this is some JDK API cleveat and
not a Spark bug

--
Jan Sterba
https://twitter.com/honzasterba | http://flickr.com/honzasterba |
http://500px.com/honzasterba

On Fri, Mar 4, 2016 at 6:08 PM, Michal Vince  wrote:

> Hi guys
> I`m using spark 1.6.0 and I`m not sure if I found a bug or I`m doing
> something wrong
>
> I`m playing with dataframes and I`m converting iso 8601 with millis to my
> timezone - which is Europe/Bratislava with fromt_utc_timestamp function
> from spark.sql.functions
>
> the problem is that Europe/Bratislava is UTC+1 hour in february, and from
> 27th March it`s going to be UTC+2h but from_utc_timestamp ignores this and
> always converts to UTC+2h
>
> am doing something wrong when converting?
>
>
> e.g.
> myDF.withColumn("time",
> from_utc_timestamp(l.col("timestamp"),"Europe/Bratislava"))
>
> and the output:
> time timestamp
>
>
> 2016-02-22 02:59:11.0 2016-02-22T00:59:11.000Z
> 2016-02-20 20:16:35.0 2016-02-20T18:16:35.000Z
> 2016-02-20 05:17:29.0 2016-02-20T03:17:29.000Z
> 2016-02-18 18:29:06.0 2016-02-18T16:29:06.000Z
> 2016-02-17 01:47:20.0 2016-02-16T23:47:20.000Z
> 2016-02-15 07:05:04.0 2016-02-15T05:05:04.000Z
> 2016-02-13 23:25:14.0 2016-02-13T21:25:14.000Z
>


1.6.0 spark.sql datetime conversion problem

2016-03-04 Thread Michal Vince

Hi guys
I`m using spark 1.6.0 and I`m not sure if I found a bug or I`m doing 
something wrong


I`m playing with dataframes and I`m converting iso 8601 with millis to 
my timezone - which is Europe/Bratislava with fromt_utc_timestamp 
function from spark.sql.functions


the problem is that Europe/Bratislava is UTC+1 hour in february, and 
from 27th March it`s going to be UTC+2h but from_utc_timestamp ignores 
this and always converts to UTC+2h


am doing something wrong when converting?


e.g.
myDF.withColumn("time", 
from_utc_timestamp(l.col("timestamp"),"Europe/Bratislava"))


and the output:
timetimestamp



2016-02-22 02:59:11.0   2016-02-22T00:59:11.000Z
2016-02-20 20:16:35.0   2016-02-20T18:16:35.000Z
2016-02-20 05:17:29.0   2016-02-20T03:17:29.000Z
2016-02-18 18:29:06.0   2016-02-18T16:29:06.000Z
2016-02-17 01:47:20.0   2016-02-16T23:47:20.000Z
2016-02-15 07:05:04.0   2016-02-15T05:05:04.000Z
2016-02-13 23:25:14.0   2016-02-13T21:25:14.000Z