[ https://issues.apache.org/jira/browse/SPARK-46769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wenchen Fan reassigned SPARK-46769: ----------------------------------- Assignee: Wenchen Fan (was: Max Gekk) > Refine timestamp related schema inference > ----------------------------------------- > > Key: SPARK-46769 > URL: https://issues.apache.org/jira/browse/SPARK-46769 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 4.0.0 > Reporter: Max Gekk > Assignee: Wenchen Fan > Priority: Major > Labels: pull-request-available > Fix For: 4.0.0, 3.5.1 > > > After the PR https://github.com/apache/spark/pull/43243, the TIMESTAMP_NTZ > type inference in CSV/JSON datasource got 2 new guards which means > TIMESTAMP_NTZ should be inferred either if: > 1. the SQL config `spark.sql.legacy.timeParserPolicy` is set to `LEGACY` or > 2. `spark.sql.timestampType` is set to `TIMESTAMP_NTZ`. > otherwise CSV/JSON should try to infer `TIMESTAMP_LTZ`. > Both guards are unnecessary because: > 1. when `spark.sql.legacy.timeParserPolicy` is `LEGACY` that only means Spark > should use a legacy Java 7- parser: `FastDateFormat` or `SimpleDateFormat`. > Both parser are applicable for parsing `TIMESTAMP_NTZ`. > 2. when `spark.sql.timestampType` is set to `TIMESTAMP_LTZ`, it doesn't mean > that we should skip inferring of `TIMESTAMP_NTZ` types in CSV/JSON, and try > to parse the timestamp string value w/o time zone like > `2024-01-19T09:10:11.123` using a LTZ format **with timezone** like > `yyyy-MM-dd'T'HH:mm:ss.SSSXXX`. _The last one cannot match any NTZ values for > sure._ -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org