[
https://issues.apache.org/jira/browse/SPARK-60916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60916:
-----------------------------------
Labels: correctness pull-request-available (was: correctness)
> Parquet filter pushdown of IN (..., NULL) on a BOOLEAN column drops matching
> rows when the list is longer than spark.sql.parquet.pushdown.inFilterThreshold
> -----------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60916
> URL: https://issues.apache.org/jira/browse/SPARK-60916
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.0.0, 3.5.8, 4.2.0, 4.1.1
> Reporter: Yaoxuan Wu
> Priority: Minor
> Labels: correctness, pull-request-available
>
> {code:python}
> from pyspark.sql import SparkSession
> spark = SparkSession.builder.getOrCreate()
> path = "/tmp/bool_in_null"
> spark.sql("SELECT * FROM VALUES (true), (false) AS
> t(b)").coalesce(1).write.mode("overwrite").parquet(path)
> spark.sql("SELECT CAST(NULL AS BOOLEAN) AS
> b").write.mode("append").parquet(path)
> spark.read.parquet(path).createOrReplaceTempView("t")
> q = "SELECT b FROM t WHERE b IN (true, NULL)"
> spark.sql(q).show() # one row: true
> spark.conf.set("spark.sql.parquet.pushdown.inFilterThreshold", "1")
> spark.sql(q).show() # expected one row: true; actual: no row
> {code}
> Same on 3.5.8, 4.0.0, 4.1.1 and 4.2.0. {{b IN (true, false, NULL)}} with
> threshold 2 also returns no row. With
> {{spark.sql.parquet.filterPushdown=false}}, or for an INT column ({{x IN (1,
> NULL)}}), the result is right.
> Cause: {{ParquetFilters.createFilterHelper}}, {{case sources.In}} (v4.2.0
> lines 798-818, unchanged on master). When the IN list has more values than
> {{inFilterThreshold}} and contains NULL, the pushed filter is an OR of an
> equality with NULL and a Parquet IN predicate on the other values:
> {code:scala}
> if (values.contains(null)) {
> Seq(makeEq.lift(fieldType).map(_(fieldNames, null)),
> makeInPredicate.lift(fieldType).map(_(fieldNames, values.filter(_ !=
> null)))
> ).flatten.reduceLeftOption(FilterApi.or)
> {code}
> {{makeEq}} has a {{ParquetBooleanType}} case, but {{makeInPredicate}} has
> none (it is the only type {{makeEq}} supports and {{makeInPredicate}} does
> not). For a BOOLEAN column the second element is {{None}}, {{flatten}} drops
> it, and the pushed filter becomes {{b = NULL}} alone, which is stricter than
> the original predicate; Parquet then skips every row group without NULLs.
> Pushing nothing when either part cannot be built, or adding a BOOLEAN case to
> {{makeInPredicate}}, would fix it. The code was added in SPARK-39312 (3.4.0).
> The default threshold (10) is not affected: {{OptimizeIn}} removes duplicate
> values from the IN list ({{ExpressionSet(list)}}), and a BOOLEAN list has at
> most three distinct values, so this needs {{inFilterThreshold}} <= 2.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
