[
https://issues.apache.org/jira/browse/SPARK-60792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated SPARK-60792:
-----------------------------
Affects Version/s: 3.5.9, 4.0.5, 4.1.3, 4.1.4, 4.2.1, 4.3.0, 5.0.0
(was: 4.2.0)
> ORC V1 validates a pruned TIMESTAMP_NTZ field against the wrong physical field
> ------------------------------------------------------------------------------
>
> Key: SPARK-60792
> URL: https://issues.apache.org/jira/browse/SPARK-60792
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.5.9, 4.0.5, 4.1.3, 4.1.4, 4.2.1, 4.3.0, 5.0.0
> Reporter: Chao Sun
> Priority: Major
>
> h3. Description
> An ORC array element contains a TIMESTAMP field followed by a TIMESTAMP_NTZ
> field. When a query selects only the second field, the ORC V1 reader can
> validate the requested type against the first physical field. It then rejects
> a read of a valid TIMESTAMP_NTZ value with
> {{UNSUPPORTED_FEATURE.ORC_TYPE_CAST}}.
> h3. Reproduction
> Run in spark-shell:
> {code:scala}
> spark.conf.set("spark.sql.sources.useV1SourceList", "orc")
> spark.conf.set("spark.sql.optimizer.nestedSchemaPruning.enabled", "true")
> val path =
> java.nio.file.Files.createTempDirectory("orc-timestamp").resolve("data").toString
> spark.sql("""
> SELECT array(named_struct(
> 'old', TIMESTAMP '2020-01-01',
> 'wanted', TIMESTAMP_NTZ '2021-01-01'
> )) AS arr,
> 0 AS idx
> """).write.orc(path)
> val df = spark.read.orc(path)
> df.selectExpr("arr[0].wanted").show(false)
> df.selectExpr("arr[idx].wanted").show(false)
> {code}
> *Expected:* Both queries return {{2021-01-01 00:00:00}} from the
> TIMESTAMP_NTZ field {{wanted}}.
> *Actual:* The constant-index query fails with
> {{UNSUPPORTED_FEATURE.ORC_TYPE_CAST}}. The index-column query returns the
> expected timestamp. This reproduces with both row and vectorized readers.
> The compatibility check should validate {{wanted}} against its corresponding
> physical field and continue to reject genuinely incompatible timestamp
> conversions.
> h3. References
> * [Review discussion and
> reproduction|https://github.com/apache/spark/pull/59972#discussion_r4433555096]
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
