peter-toth opened a new pull request, #57357:
URL: https://github.com/apache/spark/pull/57357
### What changes were proposed in this pull request?
This PR guards the `SupportsPushDownV2Filters` branch of
`PushDownUtils.pushFilters` so that only deterministic filters are translated
and pushed to the data source; nondeterministic filters are kept as post-scan
filters and evaluated by Spark after the scan. This mirrors the fix made for
`SupportsPushDownCatalystFilters` in #57235 (SPARK-58112).
### Why are the changes needed?
Before this change the `SupportsPushDownV2Filters` branch translated and
pushed every conjunct, and `V2ExpressionBuilder` translates `Rand`, so a
predicate such as `rand() > 0.5` was pushed to a V2 source.
Pushing a nondeterministic predicate is unsound: the data source may
evaluate it a different number of times or at a different point than Spark, and
a *partial* push — a predicate used for pruning yet also returned for post-scan
re-evaluation (e.g. a parquet row group filter, which is an explicitly
documented mode of `SupportsPushDownV2Filters#pushedPredicates`) — evaluates
the predicate twice with different results, which can change query results.
This is the same class of problem that #57235 (SPARK-58112) fixed for
`SupportsPushDownCatalystFilters`. The other pushdown paths already avoid it:
- `SupportsPushDownFilters` (V1 `sources.Filter`): nondeterministic
predicates are untranslatable to `sources.Filter`, so they fall through to
post-scan.
- the iterative `PartitionPredicate` second pass: guarded by
`PushDownUtils.isPushablePartitionFilter` (`f.deterministic`).
Only the `SupportsPushDownV2Filters` first-pass push was left unguarded;
this closes that gap so all pushdown paths handle nondeterministic filters
consistently.
### Does this PR introduce _any_ user-facing change?
Yes. Data sources implementing `SupportsPushDownV2Filters` will no longer
receive nondeterministic filters through `pushPredicates`; those filters remain
in Spark as post-scan filters. For a source that fully enforced such a
predicate this is an internal change only; for a source that treated a pushed
nondeterministic predicate as a (partial) row group filter, this fixes
potentially incorrect results.
### How was this patch tested?
Added a regression test in `DataSourceV2Suite` ("SPARK-58207: V2 filter
pushdown skips non-deterministic filters") using
`AdvancedDataSourceV2WithV2Filter`: it asserts that `rand() > 0.5` is not
pushed to the source (only the deterministic `i > 3` is) and that the
nondeterministic filter is retained as a post-scan filter. The test fails on
master (the source receives `RAND(...) > 0.5`) and passes with this change.
Also ran `DataSourceV2Suite` and `DataSourceV2EnhancedPartitionFilterSuite`
(the latter exercises the iterative `PartitionPredicate` second pass) locally,
both green.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
