haohuaijin commented on code in PR #10968:
URL: https://github.com/apache/arrow-rs/pull/10968#discussion_r4105558520
##########
parquet/src/arrow/arrow_reader/selection/algebra.rs:
##########
@@ -27,6 +27,14 @@ use arrow_buffer::{BooleanBuffer, BooleanBufferBuilder,
MutableBuffer, bit_util}
use std::cmp::Ordering;
use std::iter::Peekable;
+// Use word-at-a-time expansion for masks with at least 8192 rows, roughly
+// 75% outer selectivity and 5% inner selectivity. Smaller or sparser inputs
+// use set indices to avoid scanning every output word.
+// The selectivity constants are divisors: at most 1/4 dropped, at least 1/20
kept.
+const AND_THEN_DENSE_MASK_MIN_LEN: usize = 8192;
Review Comment:
yes, i do a lot of tests locally(Apple M2 Max) to make sure i do not
introduce any obvious regression, but the current values might be a bit
conservative. updated comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
