This is an automated email from the ASF dual-hosted git repository.
jbonofre pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-java.git
The following commit(s) were added to refs/heads/main by this push:
new d5450742b GH-1196: Write NO_COMPRESSION_LENGTH sentinel for empty
buffers in compress() (#1248)
d5450742b is described below
commit d5450742b7ceef737ab4b1d92007f92fc549b939
Author: Prateek Ganigi <[email protected]>
AuthorDate: Tue Oct 6 04:26:29 2026 -0700
GH-1196: Write NO_COMPRESSION_LENGTH sentinel for empty buffers in
compress() (#1248)
## What's Changed
`AbstractCompressionCodec.compress()` wrote a length prefix of `0` for
empty buffers. The Arrow IPC compression format only defines a positive
length (compressed) or `-1` / `NO_COMPRESSION_LENGTH` (stored
uncompressed); a `0` prefix is undefined. As a result, C++/PyArrow
reject Java-produced streams that contain empty buffers, even though
Java itself accepts them on read.
This changes the empty-buffer shortcut in `compress()` to write
`CompressionUtil.NO_COMPRESSION_LENGTH` (`-1`) instead of `0`.
`decompress()` is intentionally left unchanged:
- its `NO_COMPRESSION_LENGTH` branch handles the new output, and
- its `decompressedLength == 0` branch remains for backward-compatible
reads of older Java-produced streams.
Added a parameterized test (`testEmptyBufferWritesUncompressedSentinel`,
covering LZ4 and ZSTD) that compresses an empty buffer, asserts the
prefix equals `NO_COMPRESSION_LENGTH`, and asserts it round-trips back
to a zero-length buffer. The pre-existing `testEmptyBuffer` continues to
pass with empty buffers now routed through the `-1` path.
Closes #1196.
---
.../arrow/compression/TestCompressionCodec.java | 29 ++++++++++++++++++++++
.../compression/AbstractCompressionCodec.java | 2 +-
2 files changed, 30 insertions(+), 1 deletion(-)
diff --git
a/compression/src/test/java/org/apache/arrow/compression/TestCompressionCodec.java
b/compression/src/test/java/org/apache/arrow/compression/TestCompressionCodec.java
index 4378be96a..a9940c1dd 100644
---
a/compression/src/test/java/org/apache/arrow/compression/TestCompressionCodec.java
+++
b/compression/src/test/java/org/apache/arrow/compression/TestCompressionCodec.java
@@ -256,6 +256,35 @@ class TestCompressionCodec {
AutoCloseables.close(decompressedBuffers);
}
+ static Stream<CompressionCodec> compressingCodecs() {
+ return Stream.of(new Lz4CompressionCodec(), new ZstdCompressionCodec());
+ }
+
+ @ParameterizedTest
+ @MethodSource("compressingCodecs")
+ void testEmptyBufferWritesUncompressedSentinel(CompressionCodec codec) {
+ // GH-1196: an empty buffer must be encoded with the uncompressed-length
+ // sentinel (-1), not 0. A 0 prefix is undefined in the Arrow IPC format
and
+ // is rejected by the C++ and Python implementations, even though Java
itself
+ // accepts it on read.
+ ArrowBuf empty = allocator.buffer(1);
+ empty.writerIndex(0);
+
+ // compress() takes ownership of and closes the input buffer
+ ArrowBuf compressed = codec.compress(allocator, empty);
+
+ assertEquals(
+ CompressionUtil.NO_COMPRESSION_LENGTH,
+ compressed.getLong(0),
+ "empty buffer must use the -1 uncompressed sentinel, not 0");
+
+ // and it must still round-trip back to an empty buffer
+ ArrowBuf decompressed = codec.decompress(allocator, compressed);
+ assertEquals(0L, decompressed.writerIndex());
+
+ decompressed.close();
+ }
+
@Test
void testLz4DecompressRejectsWrongLength() {
byte[] data = new byte[512]; // all zeros, highly compressible
diff --git
a/vector/src/main/java/org/apache/arrow/vector/compression/AbstractCompressionCodec.java
b/vector/src/main/java/org/apache/arrow/vector/compression/AbstractCompressionCodec.java
index 39b5ceae3..2aeae2748 100644
---
a/vector/src/main/java/org/apache/arrow/vector/compression/AbstractCompressionCodec.java
+++
b/vector/src/main/java/org/apache/arrow/vector/compression/AbstractCompressionCodec.java
@@ -37,7 +37,7 @@ public abstract class AbstractCompressionCodec implements
CompressionCodec {
if (uncompressedLength == 0L) {
// shortcut for empty buffer
ArrowBuf compressedBuffer =
allocator.buffer(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH);
- compressedBuffer.setLong(0, 0);
+ compressedBuffer.setLong(0, CompressionUtil.NO_COMPRESSION_LENGTH);
compressedBuffer.writerIndex(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH);
return compressedBuffer;
}
