[
https://issues.apache.org/jira/browse/SPARK-61499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61499:
-----------------------------------
Labels: pull-request-available (was: )
> Fix ORC aggregate push-down reading statistics of the wrong column
> ------------------------------------------------------------------
>
> Key: SPARK-61499
> URL: https://issues.apache.org/jira/browse/SPARK-61499
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Raviston Thanasekar
> Priority: Major
> Labels: pull-request-available
>
> ORC aggregate push-down (spark.sql.orc.aggregatePushdown=true, DSv2 scan)
> answers MIN/MAX/COUNT from the column statistics in the file footer.
> OrcUtils.createAggInternalRowFromFooter looks up a column's statistics with
> dataSchema.getFieldIndex(columnName), the column's position in the data
> schema, and uses it as the index into the statistics that
> OrcFooterReader.readStatistics builds from the ORC file's own schema. The ORC
> reader instead matches columns by name (OrcUtils.requestedColumnIds), unless
> the file has Hive-style _colN names or orc.force.positional.evolution is set.
> So push-down returns another column's statistics when the data schema orders
> or names its columns differently from the file:
> {code:python}
> spark.conf.set("spark.sql.sources.useV1SourceList", "")
> spark.sql("SELECT * FROM VALUES (1, 10, 100), (2, 20, 200) AS t(x, a,
> b)").coalesce(1).write.orc("/tmp/t")
> spark.conf.set("spark.sql.orc.aggregatePushdown", "true")
> df = spark.read.schema("b INT, a INT, z INT").orc("/tmp/t")
> df.selectExpr("min(b)").collect() # 1, expected 100 (from x)
> df.selectExpr("count(z)").collect() # 2, expected 0 (from b; z is not in the
> file)
> df.selectExpr("max(z)").collect() # 200, expected NULL
> {code}
> With spark.sql.orc.aggregatePushdown=false, the queries return 100, 0 and
> NULL. A data-schema position past the file's last column can also fail with
> IndexOutOfBoundsException.
> The same positional lookup was fixed for Parquet in SPARK-60598. For ORC, the
> footer lookup should resolve the column with the reader's rules: by position
> for _colN files and with forced positional evolution, otherwise by name
> (exactly with spark.sql.caseSensitive=true, through toLowerCase(Locale.ROOT)
> otherwise, failing on an ambiguous match), and treat a column the file does
> not have as all null.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]