Rich-T-kid commented on code in PR #12077:
URL: https://github.com/apache/arrow-rs/pull/12077#discussion_r4481274846
##########
arrow-ipc/src/compression.rs:
##########
@@ -463,6 +482,27 @@ mod tests {
assert_eq!(input_bytes, result.as_slice());
}
+ /// A buffer larger than `MAX_PREALLOC_BYTES` round-trips, and a size its
data does not back fails.
+ #[test]
+ #[cfg(any(feature = "lz4", feature = "zstd"))]
+ fn test_decompression_beyond_prealloc() {
+ let input = vec![7u8; crate::reader::MAX_PREALLOC_BYTES + 1];
+ for codec in [
+ #[cfg(feature = "lz4")]
+ super::CompressionCodec::Lz4Frame,
+ #[cfg(feature = "zstd")]
+
super::CompressionCodec::Zstd(super::DEFAULT_ZSTD_COMPRESSION_LEVEL),
+ ] {
+ let mut compressed = Vec::new();
+ codec
+ .compress(&input, &mut compressed, &mut Default::default())
+ .unwrap();
+ let decompress = |len| codec.decompress(&compressed, len, &mut
Default::default());
+ assert_eq!(decompress(input.len()).unwrap(), input, "{codec:?}");
+ assert!(decompress(usize::MAX).is_err(), "{codec:?}");
+ }
+ }
Review Comment:
This test exercises the new `ZSTD` streaming happy path, but only the LZ4
case fails against pre-PR main; the `ZSTD` assertions pass both before and
after this change. In particular, the `usize::MAX` case reads all available
output and then fails the final length check, so it does not verify the new
oversized-output probe. Could we make the input `MAX_PREALLOC_BYTES` + 2 and
assert that `decompress(input.len() - 1)` returns an error? The advertised size
would remain above the preallocation cap while the actual output exceeds it by
one byte, directly exercising that probe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
