Chao Sun created SPARK-60791:
--------------------------------
Summary: ORC V2 returns the wrong nested field when pruning under
positional schema evolution
Key: SPARK-60791
URL: https://issues.apache.org/jira/browse/SPARK-60791
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.2.0
Reporter: Chao Sun
h3. Description
With positional schema evolution enabled, pruning an array element to its
second struct field can cause the ORC V2 reader to read the first physical
field instead. The query succeeds and silently returns the wrong value.
h3. Reproduction
Run in spark-shell:
{code:scala}
spark.conf.set("spark.sql.sources.useV1SourceList", "")
spark.conf.set("spark.sql.optimizer.nestedSchemaPruning.enabled", "true")
val path =
java.nio.file.Files.createTempDirectory("orc-positional").resolve("data").toString
spark.sql("""
SELECT array(named_struct('sibling', 11, 'wanted', 22)) AS arr,
0 AS idx,
array(named_struct('keep', 33, 'skip', 44)) AS other
""").write.orc(path)
val df = spark.read.option("orc.force.positional.evolution", "true").orc(path)
df.selectExpr("arr[0].wanted", "other[0].keep").show()
df.selectExpr("arr[idx].wanted", "other[0].keep").show()
{code}
||Projection||Expected||Actual on Spark master||
|Constant index, {{arr[0].wanted}}|{{(22, 33)}}|{{(11, 33)}}|
|Index column containing zero, {{arr[idx].wanted}}|{{(22, 33)}}|{{(22, 33)}}|
Both row and vectorized readers reproduce the wrong result. The value {{11}}
belongs to {{sibling}}, the first physical child. Pruning should preserve the
physical position of {{wanted}}.
h3. References
* [Review discussion and
reproduction|https://github.com/apache/spark/pull/59972#discussion_r4433555096]
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
