hhhizzz commented on issue #11478: URL: https://github.com/apache/arrow-rs/issues/11478#issuecomment-6333545397
Hi @alamb @haohuaijin I've tried a few approaches and put together an initial prototype on an experimental branch. I'd like to address both this issue and [[#7973](https://github.com/apache/arrow-rs/issues/7973)](https://github.com/apache/arrow-rs/issues/7973) through a unified approach. The main idea is to decouple internal decoding from the capacity limits of output batches. For regular `Utf8`/`Binary` arrays, the reader temporarily uses 64-bit offsets internally and restores the requested types before returning batches, splitting at complete top-level row boundaries when necessary. For the Mask strategy, instead of accumulating decoded rows until `batch_size` selected rows are reached, the reader decodes bounded windows of raw rows, filters each window immediately, and coalesces selected results where appropriate. This should address both the excessive intermediate decoding in this issue and the case in #7973 where the selected data itself exceeds the 2 GiB limit. The experimental branch is available here: https://github.com/hhhizzz/arrow-rs/tree/pr/byte-array-mask-windows Since the prototype touches several layers of the reader, I'm planning to split it into roughly **3–4 incremental PRs**: 1. **Error handling and overflow safety**: First, harden the byte-array decoding paths against potential panics and overflow edge cases, returning explicit errors instead, with regression tests. This would improve robustness without attempting to solve the entire problem yet. 2. **64-bit offset staging and capacity-safe output**: Use 64-bit offsets internally for regular `Utf8`/`Binary` arrays to avoid cumulative offset overflow during decoding. Restore the requested logical types at the output boundary, splitting oversized results into valid batches when necessary. This should address #7973 and provide the foundation for the Mask changes. 3. **Bounded Mask decoding windows (main change)**: Limit the number of raw rows decoded per Mask window and apply filtering immediately after each window, rather than accumulating potentially huge amounts of unselected data. Adapt the window size based on the observed decoded data size. 4. **Memory and performance improvements**: Coalesce small filtered windows, improve the release of processed data in the predicate cache, avoid retaining unnecessary backing buffers for nested View arrays, and reduce allocation overhead from repeated decoding. The exact PR boundaries may change based on review feedback. I'd prefer to establish correctness and capacity handling first, then introduce windowed Mask decoding and the related performance improvements incrementally, so each part can be reviewed and evaluated independently. The prototype already includes some regression tests, but I still need to evaluate the performance impact across different workloads. Feedback on the overall approach and the proposed PR breakdown would be very welcome. 😄 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
