adriangb opened a new pull request, #25673: URL: https://github.com/apache/datafusion/pull/25673
## Which issue does this PR close? - Part of #22883. First PR of an experimental stack that implements "optional filters" (design notes: https://claude.ai/artifact/SSz7t6hPyhFWp1MDPecVqt). ## Rationale for this change Some filters are only performance hints: another operator checks the same condition again. Examples are the dynamic filters that `HashJoinExec` and TopK push into scans, and prune-only filter copies (#25550). When `pushdown_filters = true`, the Parquet scan evaluates these filters row by row. If a filter removes few rows, we pay for it twice and get nothing (#19858, #20324, #3463). Consumers cannot fix this today because nothing tells them that a filter is optional. This PR adds that marker. Later PRs make producers use it and let consumers pause optional filters that do not remove enough rows. ## What changes are included in this PR? - `OptionalFilterPhysicalExpr`: a transparent wrapper, displayed as `Optional(<inner>)`. The contract: - A consumer may skip an optional filter **only** when it is a direct conjunct of the root AND chain of its predicate. - Everywhere else the wrapper is transparent: `evaluate()` always evaluates the inner expression. A misplaced `Optional` (under `NOT`, `IS NULL`, `CASE`, `OR`) can cost performance but can never give wrong results. - Rewriters must not move expressions into or out of the wrapper. - `snapshot()` returns the inner expression, so statistics pruning sees through the wrapper. - Helpers in `physical_expr::utils`: `split_optional` (walks the root AND chain only), `is_optional_filter`, `as_dynamic_filter` (looks through the wrapper) and `debug_assert_optional_on_root_chain`. - Proto: `PhysicalOptionalFilterNode` (field 29 of `PhysicalExprNode`), using the same self-encoding pattern as `DynamicFilterPhysicalExpr`. No producer uses the wrapper yet, so there is no behavior change. Prior art: #22234 (closed by the stale bot). This PR is a smaller, rebased version with the "skip only on the root AND chain" contract. ## What is the testing strategy for this PR? - Unit tests for the wrapper (evaluate, display, bounds) and for `split_optional` with nested ANDs, `NOT(Optional(x))` and `Optional(x) OR y`. - `debug_assert_optional_on_root_chain` accepts root-chain positions and rejects other positions (debug builds). - The physical `NOT` simplifier leaves `NOT(Optional(x))` unchanged. - Pruning: `Optional(i > 0)` and `Optional(DynamicFilter(i > 0))` produce the same pruning predicate as `i > 0`. - Proto round trips, including `Optional(DynamicFilter) AND DynamicFilter` sharing one dynamic filter. ## Are there any user-facing changes? New public API (`OptionalFilterPhysicalExpr` and helpers) and a new proto message. No behavior change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
