mohitgurav20 commented on PR #25811: URL: https://github.com/apache/datafusion/pull/25811#issuecomment-6371928666
@kosiew done, here is what is landed on the branch now: **Commit 1 (benchmark baseline):** 4c3dc5f — ench: add parquet bloom filter pruning benchmarks This is placed directly on top of main so you can checkout this commit on main-equivalent code and run the benchmark to get a clean baseline, then compare against commit 2. **Commit 2 (implementation):** 6b927e5 — perf(datasource-parquet): evaluate Parquet Bloom filters incrementally during row group pruning - Resolves the merge conflict from upstream #26476 (c0ec5127a) — preserved is_fully_pruned() early-exit in both PruneWithStatistics and the LoadBloomFilters handler in ParquetMorselPlanner - Evaluation is now O(N) per row group — directly checks Guarantee::In literals against the loaded Sbbf via is_known_not_present, no full predicate tree re-evaluation - Added count_morsels helper and est_no_morsel_when_every_row_group_is_pruned from upstream - Added unit tests covering: !should_scan branch, is_fully_matched branch, and Bloom filter I/O error fallback (predicate_evaluation_errors metric) **Commit 3 (security):** 7c176f — chore: bump ctutils to 0.4.5 to fix RUSTSEC-2026-0385 The security_audit CI failed because of a new advisory on ctutils v0.4.2 that landed after our rebase. Bumped it in Cargo.lock to match what PR #26666 (damg/fix-sec-audit) does upstream — can drop this commit once that merges into main. All 300 unit tests pass, cargo clippy -D warnings clean, cargo fmt applied. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
