Raviston Thanasekar created SPARK-61499:
-------------------------------------------

             Summary: Fix ORC aggregate push-down reading statistics of the 
wrong column
                 Key: SPARK-61499
                 URL: https://issues.apache.org/jira/browse/SPARK-61499
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Raviston Thanasekar


ORC aggregate push-down (spark.sql.orc.aggregatePushdown=true, DSv2 scan) 
answers MIN/MAX/COUNT from the column statistics in the file footer. 
OrcUtils.createAggInternalRowFromFooter looks up a column's statistics with 
dataSchema.getFieldIndex(columnName), the column's position in the data schema, 
and uses it as the index into the statistics that 
OrcFooterReader.readStatistics builds from the ORC file's own schema. The ORC 
reader instead matches columns by name (OrcUtils.requestedColumnIds), unless 
the file has Hive-style _colN names or orc.force.positional.evolution is set. 
So push-down returns another column's statistics when the data schema orders or 
names its columns differently from the file:

{code:python}
spark.conf.set("spark.sql.sources.useV1SourceList", "")
spark.sql("SELECT * FROM VALUES (1, 10, 100), (2, 20, 200) AS t(x, a, 
b)").coalesce(1).write.orc("/tmp/t")

spark.conf.set("spark.sql.orc.aggregatePushdown", "true")
df = spark.read.schema("b INT, a INT, z INT").orc("/tmp/t")
df.selectExpr("min(b)").collect()    # 1, expected 100 (from x)
df.selectExpr("count(z)").collect()  # 2, expected 0 (from b; z is not in the 
file)
df.selectExpr("max(z)").collect()    # 200, expected NULL
{code}

With spark.sql.orc.aggregatePushdown=false, the queries return 100, 0 and NULL. 
A data-schema position past the file's last column can also fail with 
IndexOutOfBoundsException.

The same positional lookup was fixed for Parquet in SPARK-60598. For ORC, the 
footer lookup should resolve the column with the reader's rules: by position 
for _colN files and with forced positional evolution, otherwise by name 
(exactly with spark.sql.caseSensitive=true, through toLowerCase(Locale.ROOT) 
otherwise, failing on an ambiguous match), and treat a column the file does not 
have as all null.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to