pecsorabs opened a new issue, #12181: URL: https://github.com/apache/arrow-rs/issues/12181
**Describe the bug** `GZipCodec::decompress` and `BrotliCodec::decompress` in `parquet/src/compression.rs` read the decompressed stream to the end (`read_to_end(output_buf)`), regardless of the page's declared uncompressed size, which they receive as `uncompress_size`. GZip ignores the argument entirely (`_uncompress_size`). Brotli uses it only as a buffer-size hint. So a page with a small compressed body grows `output_buf` to whatever the stream inflates to, not to the size the page header declares. The ZSTD codec already uses `uncompress_size`. Seen in parquet 58.4.1 and 59.3.2. **Expected behavior** When `uncompress_size` is `Some(n)`, decompress at most `n` bytes, e.g. `decoder.take(n as u64).read_to_end(..)`. Return an error if the stream holds more than `n`, as the page is then inconsistent with its header. Readers can then rely on the page header to bound memory per page. **Additional context** Happy to send a PR if this approach is acceptable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
