Yang Jie created SPARK-61870:
--------------------------------
Summary: Fix wrong results when a runtime bloom filter reads a
DSv2 scan with a pushed LIMIT again
Key: SPARK-61870
URL: https://issues.apache.org/jira/browse/SPARK-61870
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Yang Jie
{{InjectRuntimeFilter}} can take a DSv2 scan with a fully pushed {{LIMIT}},
{{OFFSET}}, top-N or {{TABLESAMPLE}} as the creation side of a runtime bloom
filter when a selective {{Filter}} sits above it ({{case _: LeafNode if
hasHitSelectiveFilter}} in {{extractSelectiveFilterOverScan}}). The bloom
filter's subquery reads the scan again, apart from the join. A {{LIMIT}}
without {{ORDER BY}}, or a top-N with ties, may return other rows on that read,
so the bloom filter can miss keys that the join reads, and the join silently
drops their rows.
For example, with the test source {{RotatingLimitDataSourceV2}} (each read of a
pushed {{LIMIT}} returns other rows),
{{spark.sql.autoBroadcastJoinThreshold=-1}}, DPP off and a small
{{spark.sql.optimizer.runtime.bloomFilter.applicationSideScanSizeThreshold}},
master returns {{[1, 1]}} for the following query instead of {{[0, 91]}}, with
AQE on and off. {{fact}} has 1000 rows with {{k = id % 11}}.
{code:sql}
SELECT count_if(f.id IS NULL), count(*)
FROM (SELECT * FROM dim LIMIT 1) d LEFT JOIN fact f ON d.i = f.k
WHERE d.i > 0
{code}
The same leaf case is on branch-3.5 and branch-4.0 through branch-4.4.
apache/spark#60387 (SPARK-61173) fixes it on master by taking a DSv2 scan as
the creation side only if reading it again gives the same rows. Up to
branch-4.3 the scan has no flag for its pushed operators, so those branches
need their own change.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]