alamb opened a new issue, #10776: URL: https://github.com/apache/arrow-rs/issues/10776
## Describe the problem Part of the effort to enable parquet filter pushdown by default in DataFusion (see analysis in https://github.com/apache/datafusion/pull/24426, e.g. https://github.com/apache/datafusion/pull/24426#issuecomment-5342167688). Related to #10774 (a different component of the same overhead). Profiling ClickBench Q25 against `hits_partitioned` (no page index) with a `--profile=profiling` build of `datafusion-cli`: ```sql SELECT "SearchPhrase" FROM hits WHERE "SearchPhrase" <> '' ORDER BY "SearchPhrase" LIMIT 10; ``` The predicate selects ~15% of rows, scattered (short runs). Enabling filter pushdown costs ~300ms of CPU per query vs evaluating the same filter in a `FilterExec` above the scan (~1460 vs ~1160 ms CPU/query; ~21ms wall clock at ~14 cores). Per-function self-CPU attribution of that delta (samply, 4kHz, 32 query executions): | self CPU ms/query | % of delta | function | |---|---|---| | 82.3 | 27% | `ReadPlanBuilder::with_predicate_options` | | 21.9 | 7% | `ReadPlan::build` | (The remaining delta is largely `concat_byte_view` under `CachedArrayReader::consume_batch`, tracked separately in #10774.) ## Observations For a first predicate with no prior `RowSelection` (the common no-page-index case), the boolean filter results are always materialized as **selectors**, one `RowSelector` pair per contiguous run: https://github.com/apache/arrow-rs/blob/59.2.0/parquet/src/arrow/arrow_reader/read_plan.rs#L277-L285 The `from_boolean_buffer` arm only fires when a *prior* selection is already mask-form. `RowSelection::from_filters` (https://github.com/apache/arrow-rs/blob/59.2.0/parquet/src/arrow/arrow_reader/selection/mod.rs#L312-L324) walks every boolean array with `SlicesIterator`, producing millions of tiny selectors for a scattered selection — the 82ms/query above. Later, `RowSelectionPolicy::Auto` resolves the strategy to `Mask` for this selection shape, and `ReadPlan::build` converts the selectors back into a mask: https://github.com/apache/arrow-rs/blob/59.2.0/parquet/src/arrow/arrow_reader/read_plan.rs#L331-L337 — the 21.9ms/query above. So the pipeline for this shape is booleans → selectors → mask, where a boolean-buffer concatenation (`filters_to_boolean_buffer`, ~1ms for the same data) would have sufficed. Timing is unchanged by `force_filter_selections=true` or by disabling the predicate cache (`max_predicate_cache_size=0`), consistent with the cost being in the construction/conversion path rather than the chosen cursor representation. ## Expected behavior Selection construction cost should be proportional to the representation the plan will actually use; a scattered selection that will be consumed as a mask should not pay O(#runs) selector materialization plus a selectors→mask conversion. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
