alamb commented on code in PR #11271:
URL: https://github.com/apache/arrow-rs/pull/11271#discussion_r4134161738
##########
arrow-select/benches/filter_bits.rs:
##########
@@ -86,5 +111,44 @@ fn add_benchmark(c: &mut Criterion) {
}
}
-criterion_group!(benches, add_benchmark);
+/// `filter_bits` on masks too large for the branch predictor to learn: the
+/// cases above repeat one 1024-word mask, whose per-word branches a recent
+/// core learns, which makes random masks look faster than a real filter.
+/// Lazy strategies only, which compress word by word
+fn add_large_benchmark(c: &mut Criterion) {
+ const SIZE: usize = 1 << 22;
Review Comment:
what is a real world usecase of a 4MB bitmap? Doesn't that represent a 4
million element Array? I think Arrow workloads are typically much smaller 8K
rows or maybe 100K rows (so like `1<<13`, maybe up to `1<<18`)
##########
arrow-select/benches/filter_bits.rs:
##########
@@ -86,5 +111,44 @@ fn add_benchmark(c: &mut Criterion) {
}
}
-criterion_group!(benches, add_benchmark);
+/// `filter_bits` on masks too large for the branch predictor to learn: the
+/// cases above repeat one 1024-word mask, whose per-word branches a recent
+/// core learns, which makes random masks look faster than a real filter.
+/// Lazy strategies only, which compress word by word
+fn add_large_benchmark(c: &mut Criterion) {
+ const SIZE: usize = 1 << 22;
Review Comment:
If we want to model real filtering, I think a more realistic approach would
be to make a bunch (100?) of 1K bitmaps (representing 8 k rows) and run filter
on them in a series -- that would add a realistic function call / per batch
overhead as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
