adriangb opened a new issue, #11430: URL: https://github.com/apache/arrow-rs/issues/11430
Follow-up to https://github.com/apache/arrow-rs/pull/10555, from [this review comment](https://github.com/apache/arrow-rs/pull/10555#discussion_r4163563964). With a `RowFilter`, `ScanPlan` cannot apply offset and limit, because the number of rows that pass the filter is not known. So a scan with a filter and `LIMIT 10` plans every remaining row group. A caller with a large read-ahead budget then fetches much more data than the decoder will request. Proposed fix: add a public field to `PlannedRange` (for example `conditional: bool`). It marks a range that the decoder requests only if the earlier predicates and the limit still need it. A caller can then fetch predicate columns eagerly and conditional output columns lazily. `PlannedRange` is `#[non_exhaustive]`, so this change is additive. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
