Ilya Ganelin created SPARK-13268:
------------------------------------
Summary: SQL Timestamp stored as GMT but toString returns GMT-08:00
Key: SPARK-13268
URL: https://issues.apache.org/jira/browse/SPARK-13268
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 1.6.0
Reporter: Ilya Ganelin
There is an issue with how timestamps are displayed/converted to Strings in
Spark SQL. The documentation states that the timestamp should be created in the
GMT time zone, however, if we do so, we see that the output actually contains a
-8 hour offset:
{{
new
Timestamp(ZonedDateTime.parse("2015-01-01T00:00:00Z[GMT]").toInstant.toEpochMilli)
res144: java.sql.Timestamp = 2014-12-31 16:00:00.0
new
Timestamp(ZonedDateTime.parse("2015-01-01T00:00:00Z[GMT-08:00]").toInstant.toEpochMilli)
res145: java.sql.Timestamp = 2015-01-01 00:00:00.0
}}
This result is confusing, unintuitive, and introduces issues when converting
from DataFrames containing timestamps to RDDs which are then saved as text.
This has the effect of essentially shifting all dates in a dataset by 1 day.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
