Max Gekk created SPARK-46769:
--------------------------------
Summary: Fix inferring of TIMESTAMP_NTZ in CSV/JSON
Key: SPARK-46769
URL: https://issues.apache.org/jira/browse/SPARK-46769
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.0.0
Reporter: Max Gekk
Assignee: Max Gekk
After the PR https://github.com/apache/spark/pull/43243, the TIMESTAMP_NTZ type
inference in CSV/JSON datasource got 2 new guards which means TIMESTAMP_NTZ
should be inferred either if:
1. the SQL config `spark.sql.legacy.timeParserPolicy` is set to `LEGACY` or
2. `spark.sql.timestampType` is set to `TIMESTAMP_NTZ`.
otherwise CSV/JSON should try to infer `TIMESTAMP_LTZ`.
Both guards are unnecessary because:
1. when `spark.sql.legacy.timeParserPolicy` is `LEGACY` that only means Spark
should use a legacy Java 7- parser: `FastDateFormat` or `SimpleDateFormat`.
Both parser are applicable for parsing `TIMESTAMP_NTZ`.
2. when `spark.sql.timestampType` is set to `TIMESTAMP_LTZ`, it doesn't mean
that we should skip inferring of `TIMESTAMP_NTZ` types in CSV/JSON, and try to
parse the timestamp string value w/o time zone like `2024-01-19T09:10:11.123`
using a LTZ format **with timezone** like `yyyy-MM-dd'T'HH:mm:ss.SSSXXX`. _The
last one cannot match any NTZ values for sure._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
