adriangb opened a new issue, #11429: URL: https://github.com/apache/arrow-rs/issues/11429
Follow-up to https://github.com/apache/arrow-rs/pull/10555. **1. The predicate stage fetches with one projection and decodes with another.** ([review comment](https://github.com/apache/arrow-rs/pull/10555#discussion_r4163563934)) `RowGroupReaderBuilder` fetches predicate columns with the `StageSchedule` snapshot (`self.stages.fetch(Stage::Predicate(..))`). But `try_into_in_memory_row_group` and `build_array_reader` still read the live `predicate.projection()`. `fill_column_chunks` requires the two masks to be the same. They are the same today. But `ArrowPredicate::evaluate` takes `&mut self`, and the trait does not require `projection()` to stay constant. The cache has the same split: `Start` recomputes it with `compute_cache_projection`, and the fetch uses `stages.cache_projection`. Proposed fix: the decode side reads the projection and the cache projection from the `StageSchedule`, so there is one source. **2. `ColumnFetch` turns page ranges back into page indexes.** ([review comment](https://github.com/apache/arrow-rs/pull/10555#discussion_r4163563949)) `ColumnFetch::new` calls `RowSelection::scan_ranges`, which builds byte ranges from page locations. It then maps them back to page indexes by matching offsets, and `ranges()` builds the same byte ranges again. On the decoder path, this adds a `Vec<usize>` and a second pass for each column in each stage. It also depends on page offsets being distinct. Proposed fix: add a variant of `scan_ranges` that returns the selected page indexes directly. **3. Test and benchmark gaps.** ([review](https://github.com/apache/arrow-rs/pull/10555#pullrequestreview-5389104140)) - `ColumnFetch` has only indirect coverage. Add unit tests for a dictionary prefix, an offset index that lists no pages, and the page-index mapping. - Add the filter + offset case to `plan_covers_later_requests_at_every_step`. - After item 2, run the full `push_decoder` bench (`bench_1buf` / `bench_nbuf`) against `main`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
