[
https://issues.apache.org/jira/browse/AVRO-4247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18080091#comment-18080091
]
ASF subversion and git services commented on AVRO-4247:
-------------------------------------------------------
Commit d28279dace6ffd2deaf6a66e5e82f81a582c3252 in avro's branch
refs/heads/main from Steve Loughran
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=d28279dace ]
AVRO-4247: Enforce decompression size limits (#3745)
* [AVRO-4081] Add decompression size limit to prevent decompression bomb DoS
Add maximum decompression size limit in DeflateCodec to prevent
OutOfMemoryError when processing maliciously crafted Avro files
with high compression ratios (decompression bombs).
The limit defaults to 200MB and can be configured via system property:
org.apache.avro.limits.decompress.maxLength
* Update lang/java/avro/src/main/java/org/apache/avro/file/DeflateCodec.java
Thanks!
Co-authored-by: Martin Grigorov <[email protected]>
* Update lang/java/avro/src/main/java/org/apache/avro/file/DeflateCodec.java
Co-authored-by: Martin Grigorov <[email protected]>
* Address code review feedback for decompression bomb fix
- Move MAX_DECOMPRESS_LENGTH initialization to static block (read once at class
load)
- Add WARNING log for invalid property values (NumberFormatException)
- Validate negative and zero values, reject with warning
- Add "(bytes)" to error message for clarity
- Add quotes around property name in error message
Test command:
java -Xmx64m -Dorg.apache.avro.limits.decompress.maxLength=1048576 \
-jar avro-tools-1.13.0-SNAPSHOT.jar tojson poc.avro
Expected behavior:
Exception in thread "main" org.apache.avro.AvroRuntimeException:
Decompressed size 1056768 (bytes) exceeds maximum allowed size 1048576.
This can be configured by setting the system property
'org.apache.avro.limits.decompress.maxLength'
* [AVRO-4] Move limit checks into NonCopyingByteArrayOutputStream
* Automatically available to all codecs
* Does need an explicit constructor with no limit, used in DataFileWriter
* No tests, though that new constructor makes it trivial
Note: merged in main as DataFileWriter changes would otherwise stop merging
Change-Id: Ifc5b8921a00425df331a4889472b3e78c6677bde
* [AVRO-4247] review feedback
* Tests
* Option read moved to SystemLimitException
Change-Id: I01d4203ad65e0e09dde88b33f603879323b06425
* [AVRO-4247] Rework invocation
* NonCopyingByteArrayOutputStream ctor reverts to unlimited
* moved the check logic to SystemLimitException
* new static method restrictedCapacityOutputStream() is invoked to
get capacity limited streams
* All Codecs use this
* Except Snappy which calls the SystemLimitException check directly
* And there's a test for that too
Change-Id: I0aa355e258c03d452c4e5a4a18605e8e2b357b2d
* spotless
Change-Id: Ie76e91f9e00ff799d2c0d82258802e6a7f4ef756
* getLongLimitFromProperty() to use parameters over inline constants
Change-Id: I5905dac4ac8a9f838cd7b3025906ef278a5cfb58
* change static factory method to "capacityLimitedOutputStream()"
Change-Id: I24dcd2a023c544370c347bdcd17e6d68abeaf260
* Apply compressor changes from iemejia
+ limit the size of a single output stream to max memory/4.
Change-Id: I0344e4c586f982f37eead7d4a6246bd1af5b4e81
* Beautify with spotless
---------
Co-authored-by: OwenSanzas <[email protected]>
Co-authored-by: Ze Sheng <[email protected]>
Co-authored-by: Martin Grigorov <[email protected]>
Co-authored-by: Ryan Skraba <[email protected]>
> Avro compression codecs should verify decompression size
> --------------------------------------------------------
>
> Key: AVRO-4247
> URL: https://issues.apache.org/jira/browse/AVRO-4247
> Project: Apache Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.12.1
> Reporter: Ismaël Mejía
> Priority: Blocker
> Labels: pull-request-available
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> {color:#1f2328}All Avro Java compression codecs (Deflate, Zstandard, XZ,
> BZip2, Snappy) decompress data without any size limit, allowing an attacker
> to craft a small Avro file (~50KB) that expands to an extremely large size
> (~50MB+), causing {color}{{OutOfMemoryError}}{color:#1f2328} and crashing the
> JVM.{color}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
