kumarUjjawal opened a new pull request, #25862: URL: https://github.com/apache/datafusion/pull/25862
## Which issue does this PR close? Related to #25620. Split from #25648 following [the review request](https://github.com/apache/datafusion/pull/25648#discussion_r4122769097). This PR can be reviewed and merged independently. ## Rationale for this change A pushed-down `fetch` stops each FilterExec partition after a bounded number of rows, but filter statistics currently ignore it. A filter with 100 matching rows and `fetch=3` still estimates 100 rows in a single partition. This can distort downstream planning decisions. ## What changes are included in this PR? Apply the per-partition fetch in both the built-in filter statistics and FilterStatisticsProvider. Adjust row and byte estimates while keeping empty, all-null, and proven singleton column statistics consistent. Overall statistics remain estimates when rows may be distributed unevenly across partitions. Add independent execution tests with one and two partitions, plus tests for zero fetch, all-null columns, singleton preservation, and the provider path. ## What is the testing strategy for this PR? Passed on this independent branch: - `cargo test --profile ci -p datafusion-physical-plan --lib` (2,367 tests) - `cargo clippy --profile ci -p datafusion-physical-plan --all-targets --all-features -- -D warnings` - `cargo fmt --all` and `git diff --check` Benchmarks were not rerun for the split. This extracts the fetch implementation already present in #25648; its earlier combined measurements are not isolated measurements of this PR. ## Are there any user-facing changes? EXPLAIN statistics account for a fetch pushed into FilterExec. Plan choices can change as a result. No public API changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
