[
https://issues.apache.org/jira/browse/SPARK-39833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17575032#comment-17575032
]
Ivan Sadikov edited comment on SPARK-39833 at 8/5/22 5:07 AM:
--------------------------------------------------------------
Your example should work with
spark.conf.set("spark.sql.caseSensitive", "true")
Or when disabling column index.
spark.conf.set("parquet.filter.columnindex.enabled", "false")
In fact, you should be getting a log4j warning for duplicate columns:
05:44:55.016 WARN org.apache.spark.sql.execution.datasources.DataSource: Found
duplicate column(s) in the data schema and the partition schema: `col`
That said, it is still a bug to return a different row count due to
case-insensitive analysis in Spark and a bug in filtering in Parquet-Mr. I will
open a PR to fix it.
was (Author: ivan.sadikov):
This is related to case insensitive analysis in Spark. Your example should work
with
spark.conf.set("spark.sql.caseSensitive", "true")
Or when disabling column index.
spark.conf.set("parquet.filter.columnindex.enabled", "false")
In fact, you should be getting a log4j warning for duplicate columns:
05:44:55.016 WARN org.apache.spark.sql.execution.datasources.DataSource: Found
duplicate column(s) in the data schema and the partition schema: `col`
That said, it is still a bug to return a different row count. I will open a PR
to fix it.
> Filtered parquet data frame count() and show() produce inconsistent results
> when spark.sql.parquet.filterPushdown is true
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-39833
> URL: https://issues.apache.org/jira/browse/SPARK-39833
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.2.1
> Reporter: Michael Allman
> Priority: Major
> Labels: correctness
>
> One of our data scientists discovered a problem wherein a data frame
> `.show()` call printed non-empty results, but `.count()` printed 0. I've
> narrowed the issue to a small, reproducible test case which exhibits this
> aberrant behavior. In pyspark, run the following code:
> {code:python}
> from pyspark.sql.types import *
> parquet_pushdown_bug_df = spark.createDataFrame([{"COL0": int(0)}],
> schema=StructType(fields=[StructField("COL0",IntegerType(),True)]))
> parquet_pushdown_bug_df.repartition(1).write.mode("overwrite").parquet("parquet_pushdown_bug/col0=0/parquet_pushdown_bug.parquet")
> reread_parquet_pushdown_bug_df = spark.read.parquet("parquet_pushdown_bug")
> reread_parquet_pushdown_bug_df.filter("col0 = 0").show()
> print(reread_parquet_pushdown_bug_df.filter("col0 = 0").count())
> {code}
> In my usage, this prints a data frame with 1 row and a count of 0. However,
> disabling `spark.sql.parquet.filterPushdown` produces consistent results:
> {code:python}
> spark.conf.set("spark.sql.parquet.filterPushdown", False)
> reread_parquet_pushdown_bug_df.filter("col0 = 0").show()
> reread_parquet_pushdown_bug_df.filter("col0 = 0").count()
> {code}
> This will print the same data frame, however it will print a count of 1. The
> key to triggering this bug is not just enabling
> `spark.sql.parquet.filterPushdown` (which is enabled by default). The case of
> the column in the data frame (before writing) must differ from the case of
> the partition column in the file path, i.e. COL0 versus col0 or col0 versus
> COL0.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
