peter-toth commented on PR #57357: URL: https://github.com/apache/spark/pull/57357#issuecomment-5015189806
cc @szehon-ho @cloud-fan @dongjoon-hyun — as the author/reviewers of #57235 (SPARK-58112). Non-deterministic predicate handling looks like a gap in the `SupportsPushDownV2Filters` path as well. #57235 guarded `SupportsPushDownCatalystFilters`, but the V2 `Predicate` branch of `PushDownUtils.pushFilters` still translates and pushes non-deterministic predicates — `V2ExpressionBuilder` translates `Rand`, so `rand() > 0.5` reaches the source. The V1 `sources.Filter` path avoids this (nondeterministic predicates are untranslatable), and the iterative `PartitionPredicate` second pass already guards via `isPushablePartitionFilter`, so this branch is the only one left unguarded. The gap can be closed either way: - by this PR — guard the branch and keep nondeterministic filters post-scan, the same approach as #57235; or - by documenting a "fully accept or fully decline" rule on `SupportsPushDownV2Filters`: a source must never push a nondeterministic predicate partially (prune with it and also return it for post-scan re-check, e.g. as a row group filter), since that evaluates it twice with different results. WDYT — which direction do you prefer for closing it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
