Enrico Minack created SPARK-30815: ------------------------------------- Summary: Function to format timestamp with time zone Key: SPARK-30815 URL: https://issues.apache.org/jira/browse/SPARK-30815 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 3.1.0 Reporter: Enrico Minack
Whenever timestamps are turned into strings (`Column.cast(StringType)`, `date_format(timestamp, format)`, `Dataset.show()`) the default time zone is used to format the string. This default time zone is either the java default zone `user.timezone`, the Spark default zone `spark.sql.session.timeZone` or the default DataFrameWriter zone `timeZone`. Currently there is no way to format a single column in a different timezone. {code:java} scala> spark.conf.set("spark.sql.session.timeZone", "Europe/London") scala> spark.range(10). \ select(concat(lit("2020-02-01 0"), $"id", lit(":00:00")).cast(TimestampType).as("time")). \ select( $"time", date_format($"time", "uuuu-MM-dd HH:mm:ss ZZZZZ").as("local"), date_format_tz($"time", "uuuu-MM-dd HH:mm:ss ZZZZZ", "Europe/Berlin").as("Berlin") ). \ show(false) +-------------------+---------------------+--------------------------+ |time |local |Berlin | +-------------------+---------------------+--------------------------+ |2020-02-01 00:00:00|2020-02-01 00:00:00 Z|2020-02-01 01:00:00 +01:00| |2020-02-01 01:00:00|2020-02-01 01:00:00 Z|2020-02-01 02:00:00 +01:00| |2020-02-01 02:00:00|2020-02-01 02:00:00 Z|2020-02-01 03:00:00 +01:00| |2020-02-01 03:00:00|2020-02-01 03:00:00 Z|2020-02-01 04:00:00 +01:00| |2020-02-01 04:00:00|2020-02-01 04:00:00 Z|2020-02-01 05:00:00 +01:00| |2020-02-01 05:00:00|2020-02-01 05:00:00 Z|2020-02-01 06:00:00 +01:00| |2020-02-01 06:00:00|2020-02-01 06:00:00 Z|2020-02-01 07:00:00 +01:00| |2020-02-01 07:00:00|2020-02-01 07:00:00 Z|2020-02-01 08:00:00 +01:00| |2020-02-01 08:00:00|2020-02-01 08:00:00 Z|2020-02-01 09:00:00 +01:00| |2020-02-01 09:00:00|2020-02-01 09:00:00 Z|2020-02-01 10:00:00 +01:00| +-------------------+---------------------+--------------------------+ {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org