[
https://issues.apache.org/jira/browse/SPARK-61870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Jie updated SPARK-61870:
-----------------------------
Affects Version/s: 3.5.9
4.0.5
4.1.4
4.2.1
4.3.0
Labels: correctness (was: )
> Fix wrong results when a runtime bloom filter reads a DSv2 scan with a pushed
> LIMIT again
> -----------------------------------------------------------------------------------------
>
> Key: SPARK-61870
> URL: https://issues.apache.org/jira/browse/SPARK-61870
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.5.9, 4.3.0, 5.0.0, 4.2.1, 4.0.5, 4.1.4
> Reporter: Yang Jie
> Priority: Major
> Labels: correctness
>
> {{InjectRuntimeFilter}} can take a DSv2 scan with a fully pushed {{LIMIT}},
> {{OFFSET}}, top-N or {{TABLESAMPLE}} as the creation side of a runtime bloom
> filter when a selective {{Filter}} sits above it ({{case _: LeafNode if
> hasHitSelectiveFilter}} in {{extractSelectiveFilterOverScan}}). The bloom
> filter's subquery reads the scan again, apart from the join. A {{LIMIT}}
> without {{ORDER BY}}, or a top-N with ties, may return other rows on that
> read, so the bloom filter can miss keys that the join reads, and the join
> silently drops their rows.
> For example, with the test source {{RotatingLimitDataSourceV2}} (each read of
> a pushed {{LIMIT}} returns other rows),
> {{spark.sql.autoBroadcastJoinThreshold=-1}}, DPP off and a small
> {{spark.sql.optimizer.runtime.bloomFilter.applicationSideScanSizeThreshold}},
> master returns {{[1, 1]}} for the following query instead of {{[0, 91]}},
> with AQE on and off. {{fact}} has 1000 rows with {{k = id % 11}}.
> {code:sql}
> SELECT count_if(f.id IS NULL), count(*)
> FROM (SELECT * FROM dim LIMIT 1) d LEFT JOIN fact f ON d.i = f.k
> WHERE d.i > 0
> {code}
> The same leaf case is on branch-3.5 and branch-4.0 through branch-4.4.
> apache/spark#60387 (SPARK-61173) fixes it on master by taking a DSv2 scan as
> the creation side only if reading it again gives the same rows. Up to
> branch-4.3 the scan has no flag for its pushed operators, so those branches
> need their own change.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]