mohitgurav20 commented on code in PR #25811:
URL: https://github.com/apache/datafusion/pull/25811#discussion_r4540256332
##########
datafusion/core/tests/parquet/io/bloom_filter_io.rs:
##########
@@ -66,10 +66,8 @@ async fn test_parquet_bloom_filter_io_matching() {
#[tokio::test]
async fn test_parquet_bloom_filter_io_pruned() {
let test = bloom_filter_test().await;
- // Currently the Bloom filters for all candidate columns (`a` and `b`) are
- // loaded upfront. With incremental pruning
- // (https://github.com/apache/datafusion/issues/25811), `a` would be
- // evaluated first and the Bloom filter for `b` would not be loaded.
+ // Column `a` does not match, so the row group is pruned immediately
Review Comment:
Makes total sense.
@haohuaijin could you take a quick look when you have time? You opened
#25797 for multi-column observability workloads, so your input on this flow
would be really helpful here.
On the test snapshot: `matching` still reads both columns because both
match. The interleaving shows up in `pruned`, where request 5 (col `b`) is
completely dropped because `a` is fetched and evaluated first, pruning the row
group before `b` is ever fetched. Happy to add a multi-row-group test as well
if you'd like.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
