[ 
https://issues.apache.org/jira/browse/SPARK-46769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wenchen Fan updated SPARK-46769:
--------------------------------
    Description:     (was: After the PR 
https://github.com/apache/spark/pull/43243, the TIMESTAMP_NTZ type inference in 
CSV/JSON datasource got 2 new guards which means TIMESTAMP_NTZ should be 
inferred either if:

1. the SQL config `spark.sql.legacy.timeParserPolicy` is set to `LEGACY` or
2. `spark.sql.timestampType` is set to `TIMESTAMP_NTZ`.

otherwise CSV/JSON should try to infer `TIMESTAMP_LTZ`.

Both guards are unnecessary because:

1. when `spark.sql.legacy.timeParserPolicy` is `LEGACY` that only means Spark 
should use a legacy Java 7- parser: `FastDateFormat` or `SimpleDateFormat`. 
Both parser are applicable for parsing `TIMESTAMP_NTZ`.
2. when `spark.sql.timestampType` is set to `TIMESTAMP_LTZ`, it doesn't mean 
that we should skip inferring of `TIMESTAMP_NTZ` types in CSV/JSON, and try to 
parse the timestamp string value w/o time zone like `2024-01-19T09:10:11.123` 
using a LTZ format **with timezone** like `yyyy-MM-dd'T'HH:mm:ss.SSSXXX`. _The 
last one cannot match any NTZ values for sure._)

> Refine timestamp related schema inference
> -----------------------------------------
>
>                 Key: SPARK-46769
>                 URL: https://issues.apache.org/jira/browse/SPARK-46769
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 4.0.0
>            Reporter: Max Gekk
>            Assignee: Wenchen Fan
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.0.0, 3.5.1
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to