Chao Sun created SPARK-60792:
--------------------------------
Summary: ORC V1 validates a pruned TIMESTAMP_NTZ field against the
wrong physical field
Key: SPARK-60792
URL: https://issues.apache.org/jira/browse/SPARK-60792
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.2.0
Reporter: Chao Sun
h3. Description
An ORC array element contains a TIMESTAMP field followed by a TIMESTAMP_NTZ
field. When a query selects only the second field, the ORC V1 reader can
validate the requested type against the first physical field. It then rejects a
read of a valid TIMESTAMP_NTZ value with {{UNSUPPORTED_FEATURE.ORC_TYPE_CAST}}.
h3. Reproduction
Run in spark-shell:
{code:scala}
spark.conf.set("spark.sql.sources.useV1SourceList", "orc")
spark.conf.set("spark.sql.optimizer.nestedSchemaPruning.enabled", "true")
val path =
java.nio.file.Files.createTempDirectory("orc-timestamp").resolve("data").toString
spark.sql("""
SELECT array(named_struct(
'old', TIMESTAMP '2020-01-01',
'wanted', TIMESTAMP_NTZ '2021-01-01'
)) AS arr,
0 AS idx
""").write.orc(path)
val df = spark.read.orc(path)
df.selectExpr("arr[0].wanted").show(false)
df.selectExpr("arr[idx].wanted").show(false)
{code}
*Expected:* Both queries return {{2021-01-01 00:00:00}} from the TIMESTAMP_NTZ
field {{wanted}}.
*Actual:* The constant-index query fails with
{{UNSUPPORTED_FEATURE.ORC_TYPE_CAST}}. The index-column query returns the
expected timestamp. This reproduces with both row and vectorized readers.
The compatibility check should validate {{wanted}} against its corresponding
physical field and continue to reject genuinely incompatible timestamp
conversions.
h3. References
* [Review discussion and
reproduction|https://github.com/apache/spark/pull/59972#discussion_r4433555096]
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
