[ https://issues.apache.org/jira/browse/SPARK-31892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wenchen Fan resolved SPARK-31892. --------------------------------- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 28706 [https://github.com/apache/spark/pull/28706] > Disable week-based date filed for parsing > ----------------------------------------- > > Key: SPARK-31892 > URL: https://issues.apache.org/jira/browse/SPARK-31892 > Project: Spark > Issue Type: Sub-task > Components: SQL > Affects Versions: 3.0.0, 3.1.0 > Reporter: Kent Yao > Assignee: Kent Yao > Priority: Major > Fix For: 3.0.0 > > > It's an un-fixable behavior change to fill the gap between SimpleDateFormat > and DateTimeFormater and backward-compatibility for different JDKs. > A lot of effort has been made to prove this at > https://github.com/apache/spark/pull/28674 > The existing behavior itself in 2.4 is confusing, e.g. > {code:sql} > spark-sql> select to_timestamp('1', 'w'); > 1969-12-28 00:00:00 > spark-sql> select to_timestamp('1', 'u'); > 1970-01-05 00:00:00 > {code} > the 'u' here seems not to go to the Monday of the first week in week-based > form or the first day of the year in non-week-based form but go to the Monday > of the second week in week-based form. > {code:sql} > spark-sql> select to_timestamp('2020 2020', 'YYYY yyyy'); > 2020-01-01 00:00:00 > spark-sql> select to_timestamp('2020 2020', 'yyyy YYYY'); > 2019-12-29 00:00:00 > spark-sql> select to_timestamp('2020 2020 1', 'YYYY yyyy w'); > NULL > spark-sql> select to_timestamp('2020 2020 1', 'yyyy YYYY w'); > 2019-12-29 00:00:00 > {code} > I think we don't need to introduce all the weird behavior from Java. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org