[
https://issues.apache.org/jira/browse/AVRO-4324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18102494#comment-18102494
]
ASF subversion and git services commented on AVRO-4324:
-------------------------------------------------------
Commit 769f9cec066f4f759621e6386782faecbada1615 in avro's branch
refs/heads/branch-1.12 from Ismaël Mejía
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=769f9cec06 ]
AVRO-4324: [Java] Align ReflectDatumReader.readArray with GenericDatumReader
eager-allocation guards (#3920)
* AVRO-4324: [Java] Guard ReflectDatumReader.readArray eager allocation
ReflectDatumReader.readArray allocated the backing Java array for the declared
array block count (Array.newInstance) before reading any element, unlike
GenericDatumReader.readArray which already validates the count against the bytes
remaining and caps element types whose minimum encoded size is zero.
Apply the same guards (ensureAvailableCollectionBytes plus
checkMaxCollectionAllocation for zero-byte element types) before the eager
allocation, so a malformed or truncated record mapped to a Java array field
(e.g. long[]) fails fast with an EOFException instead of over-allocating. Valid
arrays continue to read unchanged.
* AVRO-4324: Address review: bound cumulative allocation across array blocks
The zero-byte allocation guard was only applied to the first array block, so a
large logical array split across multiple blocks could pass the first check
while the cumulative count still exceeded the heap-aware limit. Mirror
GenericDatumReader by re-validating each continuation block (via a shared
nextArrayBlock helper) against the bytes remaining and, for zero-byte element
types, the cumulative allocation limit, in both readObjectArray and
readCollection.
> [Java] Align ReflectDatumReader.readArray with GenericDatumReader
> eager-allocation guards for malformed input
> -------------------------------------------------------------------------------------------------------------
>
> Key: AVRO-4324
> URL: https://issues.apache.org/jira/browse/AVRO-4324
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
> Reporter: Ismaël Mejía
> Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> {{GenericDatumReader.readArray}} was already made robust against malformed or
> truncated input: it clamps the initial backing allocation
> ({{initialCollectionCapacity}}), validates the declared array block count
> against the bytes actually remaining ({{ensureAvailableCollectionBytes}}),
> and applies a heap-aware bound for element types whose minimum encoded size
> is zero.
> The {{ReflectDatumReader.readArray}} override does not apply these guards. It
> calls {{newArray(old, (int) l, ...)}} -> {{Array.newInstance(elementClass,
> count)}} using the declared array block count directly, so a malformed record
> mapped to a Java array field (for example a {{long[]}}) can drive a large
> eager allocation before a single element is read.
> We should bring {{ReflectDatumReader.readArray}} in line with the generic
> reader by applying the same {{ensureAvailableCollectionBytes}} /
> collection-allocation checks and clamping the initial allocation, so
> malformed input fails fast (e.g. {{EOFException}}) instead of
> over-allocating. A valid array continues to read normally.
> Relevant code:
> *
> {{lang/java/avro/src/main/java/org/apache/avro/reflect/ReflectDatumReader.java}}
> — {{readArray(...)}} and {{newArray(...)}} ({{Array.newInstance}}).
> *
> {{lang/java/avro/src/main/java/org/apache/avro/generic/GenericDatumReader.java}}
> — {{readArray(...)}}, {{ensureAvailableCollectionBytes(...)}},
> {{initialCollectionCapacity(...)}} as the reference implementation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
