kosiew commented on code in PR #25811:
URL: https://github.com/apache/datafusion/pull/25811#discussion_r4334730512
##########
datafusion/datasource-parquet/src/opener/mod.rs:
##########
@@ -4811,6 +4804,84 @@ mod test {
assert_eq!(matched, 0);
}
+ #[tokio::test]
+ async fn
incremental_bloom_filter_pruning_short_circuits_remaining_columns() {
Review Comment:
Could we extend this test with an early-pruned row group followed by one
containing a matching row? With statistics pruning disabled and decoder
filtering enabled, verify the matching row is returned and Bloom metrics report
`(pruned=1, matched=1)`. Here `matched` means Bloom pruning could not exclude
the group.
##########
datafusion/datasource-parquet/src/opener/mod.rs:
##########
@@ -1557,47 +1549,48 @@ impl RowGroupsPrunedParquetOpen {
*physical_type,
*type_length,
);
+
+ let _timer_guard = bloom_filter_eval_time.timer();
+ match predicate.prune(&row_group_filters) {
Review Comment:
Calling the full `predicate.prune()` after every column reloads previously
checked guarantees. For N Bloom-filtered columns whose queried values are all
present, membership checks grow from N to N(N+1)/2 (528 versus 32 for N=32),
with additional statistics evaluation. Please evaluate only guarantees for the
newly loaded column while preserving conservative missing-filter handling and
early rejection. Add an operation-count regression for surviving groups and
benchmark both rejecting and surviving workloads.
##########
datafusion/datasource-parquet/src/opener/mod.rs:
##########
@@ -4811,6 +4804,84 @@ mod test {
assert_eq!(matched, 0);
}
+ #[tokio::test]
+ async fn
incremental_bloom_filter_pruning_short_circuits_remaining_columns() {
Review Comment:
Could we also test partial Bloom-filter availability through the opener?
Enable filters only for the second predicate column and verify it can still
prune despite the first being unavailable, while surviving rows remain
readable. Check that a surviving group with an available filter increments
`matched` once, and an all-filters-absent group increments neither outcome.
This can share the multi-row-group test setup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
