alamb commented on code in PR #25811:
URL: https://github.com/apache/datafusion/pull/25811#discussion_r4590829652
##########
datafusion/datasource-parquet/src/opener/mod.rs:
##########
@@ -1595,53 +1551,28 @@ impl RowGroupsPrunedParquetOpen {
continue;
}
};
- row_group_filters.insert(
- column_name,
- bf,
- *physical_type,
- *type_length,
- );
+ any_filter_loaded = true;
+
+ if self.row_groups.prune_row_group_with_bloom_filter(
+ idx,
+ col_guarantee,
+ &bf,
+ &prepared.file_metrics,
+ ) {
+ break;
+ }
}
- row_group_bloom_filters[idx] = row_group_filters;
- }
- }
- Ok(BloomFiltersLoadedParquetOpen {
- prepared: self,
- row_group_bloom_filters,
- })
- }
-}
-
-impl BloomFiltersLoadedParquetOpen {
- /// Apply bloom filter pruning using already loaded bloom filters.
- fn prune_bloom_filters(mut self) -> RowGroupsPrunedParquetOpen {
- let bloom_filter_eval_time = self
- .prepared
- .prepared
- .loaded
- .prepared
- .file_metrics
- .bloom_filter_eval_time
- .clone();
- let _timer_guard = bloom_filter_eval_time.timer();
- if let Some(predicate) = self
- .prepared
- .prepared
- .pruning_predicate
- .as_ref()
- .map(|p| p.as_ref())
- && self.prepared.prepared.loaded.prepared.enable_bloom_filter
- && !self.prepared.row_groups.is_empty()
- {
- self.prepared.row_groups.prune_by_bloom_filters(
Review Comment:
I think after this deletion, there are no more callers of
`prune_by_bloom_filters` (other than test code) so we could remove it / mark it
deprecated
##########
datafusion/datasource-parquet/src/row_group_filter.rs:
##########
@@ -1766,3 +1835,19 @@ mod tests {
expected.assert(&row_groups);
}
}
+
+/// Pre-extracted Bloom filter pruning guarantees for a single Parquet column.
+///
+/// Pre-extracting guarantees avoids repeatedly re-traversing the
[`PruningPredicate`]
+/// expression tree for every row group and column during incremental Bloom
filter loading.
+#[derive(Debug, Clone)]
+pub(crate) struct BloomColumnGuarantee {
Review Comment:
This seems like a valuable improvement (keep the same pruning predicate) but
it could be split out from this PR. Is that somethign you are willing to do?
##########
datafusion/datasource-parquet/src/row_group_filter.rs:
##########
@@ -493,12 +496,78 @@ impl RowGroupAccessPlanFilter {
if prune_group {
metrics.row_groups_pruned_bloom_filter.add_pruned(1);
- self.access_plan.skip(idx)
+ self.access_plan.skip(idx);
} else {
metrics.row_groups_pruned_bloom_filter.add_matched(1);
}
}
}
+
+ /// Extracts column guarantees from a [`PruningPredicate`] that can be
evaluated against
+ /// Parquet Bloom filters.
+ pub(crate) fn prepare_bloom_column_guarantees(
Review Comment:
this replicates a bunch of code from pruning predicates extraction -- why
can't we just use the exsiting APIs?
For example, something like
```rust
let mut stats = BloomFilterStatistics::...
for (column_name, column_idx, physical_type, type_length) in
&parquet_columns {
let Some(bf) = load_filter(idx, *column_idx).await else { continue }; //
load bloom filter if present
stats.insert(column_name, bf, *physical_type, *type_length); // save
bloom filter
if self.row_groups.prune_row_group_by_bloom_filters(idx, predicate,
&stats, metrics) {
break; // skip fetching remaining columns
}
}```
##########
datafusion/datasource-parquet/src/opener/mod.rs:
##########
@@ -5229,6 +5160,402 @@ mod test {
assert_eq!(matched, 0);
}
+ #[tokio::test]
+ async fn
incremental_bloom_filter_pruning_short_circuits_remaining_columns() {
+ let store = Arc::new(InMemory::new()) as Arc<dyn ObjectStore>;
+ // Two row groups, each with columns a and b:
+ // RG-0: a = [99, 98, 97], b = [900, 980, 970] — no predicate
values present
Review Comment:
Now that we have the IO transscript tests, what extra cases do thse tests
cover?
I think we can remove them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]