Enrico Minack created SPARK-30815:
-------------------------------------
Summary: Function to format timestamp with time zone
Key: SPARK-30815
URL: https://issues.apache.org/jira/browse/SPARK-30815
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.1.0
Reporter: Enrico Minack
Whenever timestamps are turned into strings (`Column.cast(StringType)`,
`date_format(timestamp, format)`, `Dataset.show()`) the default time zone is
used to format the string. This default time zone is either the java default
zone `user.timezone`, the Spark default zone `spark.sql.session.timeZone` or
the default DataFrameWriter zone `timeZone`. Currently there is no way to
format a single column in a different timezone.
{code:java}
scala> spark.conf.set("spark.sql.session.timeZone", "Europe/London")
scala> spark.range(10). \
select(concat(lit("2020-02-01 0"), $"id",
lit(":00:00")).cast(TimestampType).as("time")). \
select(
$"time",
date_format($"time", "uuuu-MM-dd HH:mm:ss ZZZZZ").as("local"),
date_format_tz($"time", "uuuu-MM-dd HH:mm:ss ZZZZZ",
"Europe/Berlin").as("Berlin")
). \
show(false)
+-------------------+---------------------+--------------------------+
|time |local |Berlin |
+-------------------+---------------------+--------------------------+
|2020-02-01 00:00:00|2020-02-01 00:00:00 Z|2020-02-01 01:00:00 +01:00|
|2020-02-01 01:00:00|2020-02-01 01:00:00 Z|2020-02-01 02:00:00 +01:00|
|2020-02-01 02:00:00|2020-02-01 02:00:00 Z|2020-02-01 03:00:00 +01:00|
|2020-02-01 03:00:00|2020-02-01 03:00:00 Z|2020-02-01 04:00:00 +01:00|
|2020-02-01 04:00:00|2020-02-01 04:00:00 Z|2020-02-01 05:00:00 +01:00|
|2020-02-01 05:00:00|2020-02-01 05:00:00 Z|2020-02-01 06:00:00 +01:00|
|2020-02-01 06:00:00|2020-02-01 06:00:00 Z|2020-02-01 07:00:00 +01:00|
|2020-02-01 07:00:00|2020-02-01 07:00:00 Z|2020-02-01 08:00:00 +01:00|
|2020-02-01 08:00:00|2020-02-01 08:00:00 Z|2020-02-01 09:00:00 +01:00|
|2020-02-01 09:00:00|2020-02-01 09:00:00 Z|2020-02-01 10:00:00 +01:00|
+-------------------+---------------------+--------------------------+
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
