This is an automated email from the ASF dual-hosted git repository. iemejia pushed a commit to branch fuzzer in repository https://gitbox.apache.org/repos/asf/avro.git
commit eb1f80f80f729b393a77be6b6b355ff0cdfd722e Author: Ismaël Mejía <[email protected]> AuthorDate: Fri Apr 3 20:45:32 2026 +0000 Extend validate-before-allocate to stream-backed BinaryDecoder InputStreamByteSource.remainingBytes() now returns buffered bytes plus InputStream.available(), which is exact for the finite, in-memory streams used by DataFileReader and DataFileStream. This closes the remaining DoS vector where crafted Avro container files could trigger OOM via DataFileStream.initialize() header parsing. Network/RPC streams use DirectBinaryDecoder (source is null, check is a no-op), so this change has no effect on streaming use cases. Verified: 3415 tests pass, DataFileReaderFuzzer 684K+ iterations clean, JsonDecodingFuzzer 1M+ iterations clean. --- .../main/java/org/apache/avro/io/BinaryDecoder.java | 20 +++++++++++++------- lang/java/fuzz/REPORT.md | 7 +++++-- .../crash-1390b486c8848ccc4abe28242001cdcfd72328ea | Bin 0 -> 611 bytes .../crash-1cd7a7859acc957f54b8d4101764b36b80d75e4d | Bin 0 -> 21 bytes .../crash-54530daabe9701b2d430798d3c1122d70d319e4c | Bin 0 -> 18 bytes .../crash-927a991d3c7382b78a3e7f4b91ae2e129da6b93a | Bin 0 -> 176 bytes .../crash-dd8935f92653f1de660716935886011cf1a8284b | Bin 0 -> 96 bytes .../crash-f927b39629ad65f816f19100f7916b644dc1c7b4 | Bin 0 -> 218 bytes .../crash-a50126cc2d6c726de0ca203c3b659f658d356173 | 1 + .../crash-a50c2ced384d2f9cffd6c2646ab8ea661b344bbc | 1 + 10 files changed, 20 insertions(+), 9 deletions(-) diff --git a/lang/java/avro/src/main/java/org/apache/avro/io/BinaryDecoder.java b/lang/java/avro/src/main/java/org/apache/avro/io/BinaryDecoder.java index ca6537f98e..d17589da0a 100644 --- a/lang/java/avro/src/main/java/org/apache/avro/io/BinaryDecoder.java +++ b/lang/java/avro/src/main/java/org/apache/avro/io/BinaryDecoder.java @@ -546,9 +546,9 @@ public class BinaryDecoder extends Decoder { * too few bytes. * <p> * For byte-array-backed sources the remaining count is exact, so this check is - * definitive. For stream-backed sources the remaining count is unknown and this - * method is a no-op (the subsequent read will still throw {@link EOFException} - * if the stream is too short, but only after the allocation has occurred). + * definitive. For stream-backed sources the check uses buffered bytes plus + * {@link InputStream#available()}, which is a reliable lower bound for the + * finite streams used by {@code DataFileReader} and {@code DataFileStream}. * * @param length the number of bytes about to be allocated * @throws EOFException if the source is known to have fewer bytes remaining @@ -700,9 +700,11 @@ public class BinaryDecoder extends Decoder { /** * Returns the total number of bytes remaining that can be read from this source * (including any buffered bytes), or {@code -1} if the total is unknown. - * Byte-array-backed sources return an exact count; stream-backed sources return - * {@code -1} because {@link InputStream#available()} does not reliably report - * the total remaining bytes. + * Byte-array-backed sources return an exact count. Stream-backed sources return + * the sum of buffered bytes and {@link InputStream#available()}, which is a + * reliable lower bound for the finite, in-memory or file-backed streams used by + * {@code DataFileReader} and {@code DataFileStream}. Returns {@code -1} only + * when {@code available()} throws. */ protected abstract int remainingBytes(); @@ -954,7 +956,11 @@ public class BinaryDecoder extends Decoder { @Override protected int remainingBytes() { - return -1; // unknown for stream-backed sources + try { + return (ba.getLim() - ba.getPos()) + in.available(); + } catch (IOException e) { + return -1; + } } @Override diff --git a/lang/java/fuzz/REPORT.md b/lang/java/fuzz/REPORT.md index 783154dd32..5fe90b3e48 100644 --- a/lang/java/fuzz/REPORT.md +++ b/lang/java/fuzz/REPORT.md @@ -81,6 +81,7 @@ Implementation details: - The check is called in: `readString()`, `readBytes()`, `readArrayStart()`, `arrayNext()`, `readMapStart()`, `mapNext()` - For `DirectBinaryDecoder`, `source` is always null, so the check is a no-op (correct — it has no buffered byte source) - For arrays/maps, each item must be at least 1 byte, so item count serves as a valid lower bound for bytes needed +- `InputStreamByteSource.remainingBytes()` returns `buffered + in.available()`, which is exact for the finite, in-memory streams used by `DataFileReader` and `DataFileStream` (`SeekableInputStream.available()` and `ByteArrayInputStream.available()` both report exact remaining). When `available()` throws, returns -1 (check skipped). This is safe because the buffering `BinaryDecoder` is only used for file/container reading — network/RPC streams use `DirectBinaryDecoder` which bypasses this [...] This approach was chosen over lowering the default `SystemLimitException` limits because it does not change behavior for production users — only clearly impossible allocations (where the claimed length exceeds the actual data) are rejected. @@ -92,13 +93,15 @@ This approach was chosen over lowering the default `SystemLimitException` limits ## Verification ### Test suite -All 3,415 tests in the `avro` module pass after both fixes. +All 3,415 tests in the `avro` module pass after all fixes. ### Fuzzer re-runs (post-fix) - **BinaryDecodingFuzzer**: ~974K iterations, 0 crashes (previously crashed within ~4K iterations with OOM) - **SchemaFuzzer**: ~1M iterations, 0 crashes (previously found NPE) +- **DataFileReaderFuzzer**: ~684K iterations, 0 crashes (previously crashed within ~4K iterations with OOM via `DataFileStream.initialize()`) +- **JsonDecodingFuzzer**: ~1M iterations, 0 crashes -Both bugs are confirmed fixed. +All bugs are confirmed fixed. ## What Each Fuzz Test Does diff --git a/lang/java/fuzz/src/test/resources/org/apache/avro/fuzz/DataFileReaderFuzzerInputs/fuzzDataFileReader/crash-1390b486c8848ccc4abe28242001cdcfd72328ea b/lang/java/fuzz/src/test/resources/org/apache/avro/fuzz/DataFileReaderFuzzerInputs/fuzzDataFileReader/crash-1390b486c8848ccc4abe28242001cdcfd72328ea new file mode 100644 index 0000000000..6a2b66fe0e Binary files /dev/null and b/lang/java/fuzz/src/test/resources/org/apache/avro/fuzz/DataFileReaderFuzzerInputs/fuzzDataFileReader/crash-1390b486c8848ccc4abe28242001cdcfd72328ea differ diff --git a/lang/java/fuzz/src/test/resources/org/apache/avro/fuzz/DataFileReaderFuzzerInputs/fuzzDataFileReader/crash-1cd7a7859acc957f54b8d4101764b36b80d75e4d b/lang/java/fuzz/src/test/resources/org/apache/avro/fuzz/DataFileReaderFuzzerInputs/fuzzDataFileReader/crash-1cd7a7859acc957f54b8d4101764b36b80d75e4d new file mode 100644 index 0000000000..e194c8382b Binary files /dev/null and b/lang/java/fuzz/src/test/resources/org/apache/avro/fuzz/DataFileReaderFuzzerInputs/fuzzDataFileReader/crash-1cd7a7859acc957f54b8d4101764b36b80d75e4d differ diff --git a/lang/java/fuzz/src/test/resources/org/apache/avro/fuzz/DataFileReaderFuzzerInputs/fuzzDataFileReader/crash-54530daabe9701b2d430798d3c1122d70d319e4c b/lang/java/fuzz/src/test/resources/org/apache/avro/fuzz/DataFileReaderFuzzerInputs/fuzzDataFileReader/crash-54530daabe9701b2d430798d3c1122d70d319e4c new file mode 100644 index 0000000000..90aadb186f Binary files /dev/null and b/lang/java/fuzz/src/test/resources/org/apache/avro/fuzz/DataFileReaderFuzzerInputs/fuzzDataFileReader/crash-54530daabe9701b2d430798d3c1122d70d319e4c differ diff --git a/lang/java/fuzz/src/test/resources/org/apache/avro/fuzz/DataFileReaderFuzzerInputs/fuzzDataFileReader/crash-927a991d3c7382b78a3e7f4b91ae2e129da6b93a b/lang/java/fuzz/src/test/resources/org/apache/avro/fuzz/DataFileReaderFuzzerInputs/fuzzDataFileReader/crash-927a991d3c7382b78a3e7f4b91ae2e129da6b93a new file mode 100644 index 0000000000..469cf8e82e Binary files /dev/null and b/lang/java/fuzz/src/test/resources/org/apache/avro/fuzz/DataFileReaderFuzzerInputs/fuzzDataFileReader/crash-927a991d3c7382b78a3e7f4b91ae2e129da6b93a differ diff --git a/lang/java/fuzz/src/test/resources/org/apache/avro/fuzz/DataFileReaderFuzzerInputs/fuzzDataFileReader/crash-dd8935f92653f1de660716935886011cf1a8284b b/lang/java/fuzz/src/test/resources/org/apache/avro/fuzz/DataFileReaderFuzzerInputs/fuzzDataFileReader/crash-dd8935f92653f1de660716935886011cf1a8284b new file mode 100644 index 0000000000..6678a26b7b Binary files /dev/null and b/lang/java/fuzz/src/test/resources/org/apache/avro/fuzz/DataFileReaderFuzzerInputs/fuzzDataFileReader/crash-dd8935f92653f1de660716935886011cf1a8284b differ diff --git a/lang/java/fuzz/src/test/resources/org/apache/avro/fuzz/DataFileReaderFuzzerInputs/fuzzDataFileReader/crash-f927b39629ad65f816f19100f7916b644dc1c7b4 b/lang/java/fuzz/src/test/resources/org/apache/avro/fuzz/DataFileReaderFuzzerInputs/fuzzDataFileReader/crash-f927b39629ad65f816f19100f7916b644dc1c7b4 new file mode 100644 index 0000000000..730cb5b27c Binary files /dev/null and b/lang/java/fuzz/src/test/resources/org/apache/avro/fuzz/DataFileReaderFuzzerInputs/fuzzDataFileReader/crash-f927b39629ad65f816f19100f7916b644dc1c7b4 differ diff --git a/lang/java/fuzz/src/test/resources/org/apache/avro/fuzz/JsonDecodingFuzzerInputs/fuzzJsonDecoding/crash-a50126cc2d6c726de0ca203c3b659f658d356173 b/lang/java/fuzz/src/test/resources/org/apache/avro/fuzz/JsonDecodingFuzzerInputs/fuzzJsonDecoding/crash-a50126cc2d6c726de0ca203c3b659f658d356173 new file mode 100644 index 0000000000..8eeebd0b47 --- /dev/null +++ b/lang/java/fuzz/src/test/resources/org/apache/avro/fuzz/JsonDecodingFuzzerInputs/fuzzJsonDecoding/crash-a50126cc2d6c726de0ca203c3b659f658d356173 @@ -0,0 +1 @@ +r2 \ No newline at end of file diff --git a/lang/java/fuzz/src/test/resources/org/apache/avro/fuzz/JsonDecodingFuzzerInputs/fuzzJsonDecoding/crash-a50c2ced384d2f9cffd6c2646ab8ea661b344bbc b/lang/java/fuzz/src/test/resources/org/apache/avro/fuzz/JsonDecodingFuzzerInputs/fuzzJsonDecoding/crash-a50c2ced384d2f9cffd6c2646ab8ea661b344bbc new file mode 100644 index 0000000000..bc5d2876f2 --- /dev/null +++ b/lang/java/fuzz/src/test/resources/org/apache/avro/fuzz/JsonDecodingFuzzerInputs/fuzzJsonDecoding/crash-a50c2ced384d2f9cffd6c2646ab8ea661b344bbc @@ -0,0 +1 @@ +{}{}{} \ No newline at end of file
