adriangb opened a new pull request, #25715: URL: https://github.com/apache/datafusion/pull/25715
## Which issue does this PR close? - Part of #22883 (experimental "optional filters" stack, design notes: https://claude.ai/artifact/SSz7t6hPyhFWp1MDPecVqt). - **Depends on #25673, #25681, #25674 and #25682.** Review only the top commit (`feat: make optional_filter_mode = adaptive the default`). ## Rationale for this change With `datafusion.execution.parquet.pushdown_filters = true`, the Parquet scan evaluates dynamic filters (hash join, TopK, aggregate) row by row, also when they remove few rows or cost more than they save. This is a main cause of the regressions that keep `pushdown_filters` off by default. #25682 adds `optional_filter_mode = adaptive`, which pauses such filters. This PR makes `adaptive` the default, so that users who enable `pushdown_filters` get it without a second setting. | `pushdown_filters` | Before this PR | After this PR | |---|---|---| | `false` (default) | no change | no change | | `true` | dynamic filters always evaluated row by row | dynamic filters paused when they remove too few rows or cost more than they save; tried again at intervals | Query results do not change. Statistics pruning (files, row groups, pages) still uses all optional filters. ## What changes are included in this PR? - `datafusion.execution.optional_filter_mode` default: `always` → `adaptive` (`OptionalFilterMode::default()` and `ParquetSource::new` follow). - `configs.md`, `information_schema.slt`, `optional_filters.slt` comments and a `ParquetSource` unit test updated for the new default. - `push_down_filter_parquet.slt`: the TopK `EXPLAIN ANALYZE` test sets `always`. With `adaptive`, the cost check can pause the filter (it depends on timing), so the row metrics are not deterministic. - Upgrade guide note (56.0.0). To get the previous behavior: ```sql SET datafusion.execution.optional_filter_mode = 'always'; ``` ## What is the testing strategy for this PR? - Existing tests. The mode tests in `core/tests/parquet/optional_filters.rs` and `optional_filters.slt` already set each mode explicitly. - `try_pushdown_filters_reads_optional_filter_config` now checks the new default and that the config can set `always`. - Benchmarks: see the bot results on this PR (pushdown on vs main pushdown on, and pushdown on vs main pushdown off). ## Are there any user-facing changes? Yes, only when `pushdown_filters = true`: optional filters can be paused, and `EXPLAIN ANALYZE` of the Parquet scan can show `optional_filter_pauses` and `optional_filter_eval_time`. No API change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
