adriangb opened a new pull request, #11330: URL: https://github.com/apache/arrow-rs/pull/11330
# Which issue does this PR close? - Part of https://github.com/apache/arrow-rs/issues/6946. - Prepares https://github.com/apache/arrow-rs/pull/10555, as requested in [this review](https://github.com/apache/arrow-rs/pull/10555#pullrequestreview-5385246575). # Rationale for this change https://github.com/apache/arrow-rs/pull/10555 adds `ParquetPushDecoder::scan_plan()`, which tells an application which byte ranges the decoder will request. In review, the planner copied two parts of the decoder's logic: | Logic | Where the decoder has it | |---|---| | The byte ranges to fetch for one column chunk: the dictionary page, the data pages that the row selection needs, or the full chunk if there is no offset index | `InMemoryRowGroup::fetch_ranges` | | The decoding stages of a row group: each `RowFilter` predicate, then the output projection, with the columns that each stage fetches | `RowGroupReaderBuilder::try_transition` | This PR puts each part in one place that both the decoder and the planner can use. Then https://github.com/apache/arrow-rs/pull/10555 adds only the planner. It does not have to keep a copy of the decoder logic in sync. Terms: - *Offset index*: optional Parquet metadata that gives the byte location of each data page. - *Predicate cache*: decoded predicate columns that the decoder keeps and uses again for the output. Its columns are fetched with the selection expanded to batch boundaries. # What changes are included in this PR? Three commits. Each commit compiles and passes the tests. There is no behavior change. | Commit | Change | |---|---| | 1. Move | `RowBudget` and `BudgetedReadPlan` move to `push_decoder/scan_plan/budget.rs`. `RowGroupFrontier`, `QueuedRowGroups` and `NextRowGroup` move to `push_decoder/scan_plan/frontier.rs`. Their tests move with them. Only visibility changes, plus two accessors that replace direct field access. Review it with `git show --color-moved`. | | 2. `ColumnFetch` | `fetch_ranges` is rewritten as a loop over `ColumnFetch`, which gives the ranges for one column chunk. The output is the same. | | 3. `StageSchedule` | One `StageSchedule` gives the projection and the cache projection of each stage. `try_transition` builds its data requests from it. | # Are these changes tested? Yes, by the existing tests: `push_decoder`, `arrow::` and `async_reader` (539 tests). The new code has no new behavior to test. # Are there any user-facing changes? No. All changes are `pub(crate)` or private. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
