[
https://issues.apache.org/jira/browse/SPARK-54908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-54908:
-----------------------------------
Labels: easyfix pull-request-available (was: easyfix)
> dateFormat option is ignored during schema inference for JSON files
> -------------------------------------------------------------------
>
> Key: SPARK-54908
> URL: https://issues.apache.org/jira/browse/SPARK-54908
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.3.2, 3.4.1, 3.5.0, 3.5.2, 4.0.0
> Environment: Tested in Databricks on DBR 16.4 LTS. Similar behavior
> in other DBR versions as well.
> Appears to be a core spark issue in
> [JsonInferSchema.scala|https://github.com/apache/spark/blob/8fe006b20877671c75e4650a27d268b496294299/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JsonInferSchema.scala#L42]
> Reporter: Amruth Ashok
> Priority: Major
> Labels: easyfix, pull-request-available
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> When using COPY INTO ... FILEFORMAT = JSON with schema inference, the
> dateFormat option is ignored during schema inference, timestampFormat works.
> This caused date-only strings to be inferred as StringType instead of
> DateType.
> Example:
> test.json
> {
> "created_at": "02JUL14",
> "updated_at": "02JUL14 12:17:43.39 UTC"
> }
> code:
> Using COPY INTO in JSON
> COPY INTO my_table
> FROM '/path/to/test.json'
> FILEFORMAT = JSON
> OPTIONS (
> inferSchema = true,
> inferTimestamp = true,
> timestampFormat = "ddMMMyy HH:mm:ss.SSS 'UTC'",
> dateFormat = "ddMMMyy"
> )
>
> {*}Observed behavior:{*}{*}{*}
> * created_at: string (should be date)
> * updated_at: timestamp (correct)
> {*}Expected behavior:{*}{*}{*}
> * created_at: date (correct)
> * updated_at: timestamp (correct)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
