[
https://issues.apache.org/jira/browse/AVRO-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18102497#comment-18102497
]
ASF subversion and git services commented on AVRO-4296:
-------------------------------------------------------
Commit e8c69083b8593b1be1539c2fc185f0b45b2e3ac2 in avro's branch
refs/heads/main from Ismaël Mejía
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=e8c69083b8 ]
AVRO-4296: [python] Bound allocation when decoding length-prefixed values and
collections (#3861)
When decoding an array or map, DatumReader.read_array/read_map used the block
count read from the (potentially malformed or truncated) input directly as a
loop counter, and the length-prefixed byte/string readers allocated from an
unchecked length. A small payload could therefore declare a very large count or
length and drive an unbounded allocation before any element bytes were present.
This bounds those allocations, mirroring the Java SDK's two-limit approach:
- Length-prefixed values and collection blocks are validated against the number
of bytes actually remaining in the input (when the reader is seekable) before
allocating, so a truncated payload fails fast instead of over-allocating.
- Element types whose minimum encoded size is zero (null, a zero-length fixed,
or a record whose fields are all zero-byte) cannot be bounded by the bytes
remaining, so their cumulative count is capped by a separate configurable
limit (AVRO_MAX_COLLECTION_ITEMS). A structural cap of Integer.MAX_VALUE - 8
applies to all collections as defense in depth.
It also hardens related decode paths surfaced by this work: overlong/overflowing
varints are rejected, negative skips and out-of-range union/enum indices are
rejected, sized-block skips validate their byte size, and bytes_remaining()
restores the reader position and degrades gracefully on non-seekable readers.
Reading malformed or truncated input now fails fast with a clear
AvroCollectionSizeException or InvalidAvroBinaryEncoding; valid data reads
unchanged.
> [python] Bound allocation when decoding length-prefixed values and collections
> ------------------------------------------------------------------------------
>
> Key: AVRO-4296
> URL: https://issues.apache.org/jira/browse/AVRO-4296
> Project: Apache Avro
> Issue Type: Sub-task
> Components: python
> Affects Versions: 1.11.5, 1.12.1
> Reporter: Ismaël Mejía
> Assignee: Ismaël Mejía
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0, 1.11.6, 1.12.2
>
> Time Spent: 9h 20m
> Remaining Estimate: 0h
>
> A bytes or string value is encoded as a length prefix followed by that many
> bytes of data, and an array or map block is encoded as an element count
> followed by that many items. A malicious or truncated input can declare a
> very large length or count while carrying little or no actual data, causing a
> large allocation before the shortfall is noticed. When the source can report
> how many bytes remain, reject a declared length (or a collection block count)
> that exceeds the bytes actually available before allocating for it. Companion
> to AVRO-4241 (Java).
> BinaryDecoder.bytes_remaining() backs the check; read() consults
> bytes_remaining() directly for a declared length above a threshold, while
> DatumReader.read_array/read_map/skip_array/skip_map consult it via
> min_bytes_per_element(). _ensure_collection_available enforces the per-block
> limits (cumulative across blocks and after normalizing a negative block
> count); maps are additionally bounded by their >=1-byte keys. Rejections
> raise the new AvroCollectionSizeException.
> Zero-byte elements (null, a zero-length fixed, or a record with only
> zero-byte fields) consume no input, so the available-bytes check cannot bound
> their count: a tiny payload such as {"type":"array","items":"null"} declaring
> a block count of 200,000,000 would otherwise drive an unbounded allocation.
> In addition to the available-bytes check this therefore caps the cumulative
> count of zero-byte elements (default 10,000,000), applies a structural cap
> (Integer.MAX_VALUE - 8) to every non-zero-byte-element collection (which also
> covers collections read from a source that cannot report the bytes
> remaining), and bounds the array/map skip paths. When set, the
> AVRO_MAX_COLLECTION_ITEMS environment variable caps both limits. This
> supersedes the separate collection-limit sub-task.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
