[ https://issues.apache.org/jira/browse/SPARK-48776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wenchen Fan resolved SPARK-48776. --------------------------------- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 47177 [https://github.com/apache/spark/pull/47177] > XML, JSON and CSV default timestamp formatting fails for seconds offset > ----------------------------------------------------------------------- > > Key: SPARK-48776 > URL: https://issues.apache.org/jira/browse/SPARK-48776 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 3.5.0, 4.0.0 > Reporter: Milan Stefanovic > Assignee: Milan Stefanovic > Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > Older dates for certain timezones (e.g. America/Los_Angeles) have wierd > offsets - such as: -07:52:58. > JSON, XML and CSV default ISO timestamp formatter is using `XXX` for format > which omits seconds information, so we end up with wrong result. > example: > sql("SET spark.sql.session.timeZone=America/Los_Angeles") > sql("SELECT to_json(struct(CAST('1800-01-01T00:00:00+00:00' AS TIMESTAMP) AS > ts))").show(false) > will display: > {"ts":"1799-12-31T16:07:02.000-07:52"} and hence we will lose 58 seconds -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org