[
https://issues.apache.org/jira/browse/SPARK-60598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60598:
-----------------------------------
Labels: correctness pull-request-available (was: correctness)
> Parquet aggregate pushdown returns MIN/MAX/COUNT of the wrong column when a
> nested column precedes it
> -----------------------------------------------------------------------------------------------------
>
> Key: SPARK-60598
> URL: https://issues.apache.org/jira/browse/SPARK-60598
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.0.0, 3.5.8, 4.2.0, 4.1.1
> Reporter: Yaoxuan Wu
> Priority: Major
> Labels: correctness, pull-request-available
>
> {code:python}
> spark.conf.set("spark.sql.sources.useV1SourceList", "") # Parquet
> through the DSv2 scan
> spark.createDataFrame([((100, 200), 1, 2), ((300, 400), None, 6), ((500,
> 600), 5, 4)],
> "s struct<x:int,y:int>, a int, b int") \
> .coalesce(1).write.mode("overwrite").parquet("/tmp/agg_nested")
> spark.read.parquet("/tmp/agg_nested").createOrReplaceTempView("t")
> spark.conf.set("spark.sql.parquet.aggregatePushdown", "false")
> spark.sql("SELECT max(a), min(b), count(a) FROM t").collect() #
> [Row(max(a)=5, min(b)=2, count(a)=2)]
> spark.conf.set("spark.sql.parquet.aggregatePushdown", "true")
> spark.sql("SELECT max(a), min(b), count(a) FROM t").collect() #
> [Row(max(a)=600, min(b)=1, count(a)=3)]
> {code}
> With pushdown, {{max(a)}} is the max of {{s.y}}, {{min(b)}} is the min of
> {{a}}, and {{count(a)}} uses the null count of {{s.y}}. When the shifted
> column has a different physical type the query fails instead, e.g. {{max}} of
> a BOOLEAN column: {{ClassCastException: class java.lang.Integer cannot be
> cast to class java.lang.Boolean}}.
> Cause: in {{ParquetUtils.getPushedDownAggResult}}, {{index =
> dataSchema.fieldNames.toList.indexOf(colName)}} is the column's position
> among the top-level fields, but {{getCurrentBlockMaxOrMin}} / {{getNumNulls}}
> use it as {{columnChunkMetaData.get(index)}}, i.e. as the index of a leaf
> column chunk of the row group. A struct, array or map column has one chunk
> per leaf, so every later column is shifted. Master uses
> {{dataSchema.getFieldIndex(colName)}} (SPARK-58711), the same top-level
> index. Looking the chunk up by its path ({{ColumnChunkMetaData.getPath}})
> would fix it.
> Reproduces on 3.5.8, 4.0.0, 4.1.1 and 4.2.0 with
> {{spark.sql.parquet.aggregatePushdown=true}} (default false).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
