vedjaw opened a new pull request, #10520: URL: https://github.com/apache/arrow-rs/pull/10520
# Which issue does this PR close? Closes #10448. # Rationale for this change The `add_data_page` function builds the DataPageV2 buffer by incrementally extending a Vec: first repetition levels, then definition levels, then values (compressed or uncompressed). This incremental growth via `extend_from_slice` can leave significant excess capacity. The PARQUET_1_0 (DataPage v1) path already calls `shrink_to_fit` on its compressed buffer, but the PARQUET_2_0 path never reclaimed the excess capacity before converting the buffer to `Bytes` and handing it to the page writer. For dictionary-encoded columns and deferred/buffered page writers, these pages can be retained in memory for extended periods. Each page carrying slack capacity contributes to inflated peak RSS. # What changes are included in this PR? Adds a `buffer.shrink_to_fit()` call immediately before the `Page::DataPageV2` construction in `add_data_page`, matching the v1 path's existing behavior. # Are there any user-facing changes? No API changes. Pages produced by the writer now carry no slack capacity, which may reduce peak memory usage for workloads that use DataPage v2 (the default for `parquet::file::properties::WriterProperties` when `data_page_version` is set to `V2`). # Are these changes tested? The existing DataPageV2 test suite (100 tests in `column::writer::tests`) all pass, including roundtrip and compression tests. The change is a memory-management detail and does not alter the serialized page format. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
