Subodh-17 opened a new pull request, #12315:
URL: https://github.com/apache/arrow-rs/pull/12315
# Which issue does this PR close?
Closes #12181.
# Rationale for this change
In `parquet::compression`, `GZipCodec::decompress` and
`BrotliCodec::decompress` read decompressed data to the end of the input stream
using `read_to_end(output_buf)`, without enforcing the page header's declared
`uncompressed_size`.
- `GZipCodec` explicitly suppressed and ignored `_uncompress_size`.
- `BrotliCodec` used `uncompress_size` solely as an internal buffer size
hint.
While `parquet::file::serialized_reader::decode_page` checks that
`decompressed.len() == uncompressed_page_size` after decompression completes,
that post-check occurs too late to prevent memory exhaustion during
decompression. An untrusted or corrupted Parquet page declaring a small
uncompressed size (e.g. 1 KB) with a compressed payload that expands into
gigabytes forces `read_to_end` to repeatedly reallocate the heap, causing an
uncatchable Out-Of-Memory (OOM) process abort.
# What changes are included in this PR?
1. **Bounded Stream Reading:**
- When `uncompress_size` is `Some(max_size)`, the decompressor stream is
wrapped using `.take(probe_limit)`.
- `probe_limit` is computed as `(max_size as u64).saturating_add(1)`.
Reading up to `max_size + 1` allows the decoder to disambiguate an exact-match
stream from an over-inflated stream, and enables GZip (`MultiGzDecoder`) to
parse and validate the 8-byte trailer (CRC-32 and ISIZE).
2. **Strict Overflow Detection & Rollback:**
- If `bytes_read > max_size`, decompression immediately halts,
`output_buf` is truncated back to its pre-call length (`initial_len`), and a
descriptive `ParquetError::General` error is returned.
- On decoder I/O error (`Err(e)`), `output_buf` is similarly truncated to
`initial_len`, preserving preexisting caller buffer data (such as uncompressed
definition and repetition levels in DataPageV2).
3. **Brotli Buffer Clamping:**
- Clamps the internal Brotli decompressor buffer size `max_size.clamp(1,
BROTLI_DEFAULT_BUFFER_SIZE)` (capped at 4,096 bytes) to prevent large
intermediate buffer pre-allocations from untrusted headers.
4. **Untrusted Metadata Defense:**
- Intentionally avoids eager `output_buf.reserve(max_size)` inside
`Codec::decompress`. Buffers grow dynamically via `read_to_end` up to
`probe_limit`, preventing immediate OOM panics when a corrupted header declares
an enormous size (e.g. 100 GB).
5. **Backward Compatibility:**
- When `uncompress_size` is `None`, falls back to unbounded `read_to_end`
with buffer rollback on error, preserving existing behavior for unversioned
callers.
6. **Documentation Correction:**
- Updates `Codec::decompress` docstring to clarify that `uncompress_size`
acts as a safety bound rather than guaranteeing exact eager memory allocation.
### Scope Exclusions
- This PR specifically hardens GZip and Brotli decompression as reported in
#12181. It does not alter legacy `LZ4_HADOOP` (`LZ4Codec`), `LZ4_RAW`,
`Snappy`, or `ZSTD`. It does not claim to resolve every theoretical
decompression DoS vector across all formats.
# Are these changes tested?
Yes. Unit and integration tests were added in `parquet/src/compression.rs`:
- `test_bounded_decompression_suite`: Evaluates 10 scenarios across all 10
valid GZip levels (`0..=9`) and all 12 valid Brotli levels (`0..=11`):
1. Exact declared size
2. Output smaller than declared size
3. Output larger than declared size (overflow detected, buffer truncated,
descriptive error)
4. Zero-length output (and zero-length limit with non-empty payload)
5. Empty compressed input
6. Pre-populated caller buffer preservation on success
7. Pre-populated caller buffer preservation on overflow and corrupt input
8. Truncated stream handling across all compression levels
9. Missing declared size (`None`)
10. Boundary values (`usize::MAX` probe limit without overflow)
11. Zip-bomb scenario (50 KB payload with 100-byte limit halted at 101
bytes)
- `test_gzip_truncated_compressed_stream`: Dedicated regression test
covering 4 truncation scenarios (header only, half body, truncated 8-byte
trailer, unbounded mode).
- `test_brotli_truncated_compressed_stream`: Dedicated regression test
covering 4 truncation scenarios (1-byte header, half body, truncated near end,
unbounded mode).
### Verified Test Results
- `cargo test -p parquet --lib compression::tests`: 10 passed, 0 failed.
- `cargo test -p parquet --lib file::serialized_reader`: 48 passed, 0 failed
(including `test_multi_gz`).
- `cargo test -p parquet --lib file::reader`: 2 passed, 0 failed.
- `cargo clippy -p parquet --features flate2,brotli -- -D warnings`: 0
warnings.
- `cargo fmt --check -- parquet/src/compression.rs`: clean (0 diffs).
# Are there any significant user-facing changes?
No breaking public API changes. Legitimate Parquet pages are unaffected
because their decompressed size already matches the page header's uncompressed
size. Pages exceeding their declared size fail early with a clear
`ParquetError::General` error rather than exhausting system memory.
# AI usage disclosure
AI assistance (Claude / DeepMind Antigravity) was used to assist in
identifying edge cases, structuring the parameterized test matrix, and drafting
documentation. All source changes, arithmetic safety bounds, and test cases
were independently reviewed, traced, and verified against the Rust standard
library and Arrow-rs codebase by the author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]