[
https://issues.apache.org/jira/browse/AVRO-4290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18102690#comment-18102690
]
ASF subversion and git services commented on AVRO-4290:
-------------------------------------------------------
Commit c50ffa586a7a2f60fe8ed05489d4c7d3a578e092 in avro's branch
refs/heads/main from Ismaël Mejía
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=c50ffa586a ]
AVRO-4290: [python] Enforce a maximum decompressed block size (#3850)
* AVRO-4290: [python] Enforce a maximum decompressed block size
When reading a data file, each block is decompressed according to the
file's codec. A block with a very high compression ratio (or a malformed
block) could expand to far more memory than its compressed size. Enforce a
configurable maximum decompressed size across the deflate, bzip2, snappy and
zstandard codecs, mirroring the Java SDK's decompression limit (AVRO-4247).
The limit defaults to 200 MiB and can be overridden with the
AVRO_MAX_DECOMPRESS_LENGTH environment variable; exceeding it raises
AvroDecompressionSizeException.
Assisted-by: GitHub Copilot:claude-opus-4.8
* AVRO-4290: [python] Address review: avoid deflate copy; zstd
check-before-extend
- DeflateCodec.decompress accumulates into a bytearray so the flush() output is
appended in place instead of creating an extra full-size copy of the already
decompressed data.
- ZstandardCodec.decompress checks len(uncompressed) + len(chunk) before
extending, so the buffer never grows past the configured limit.
Assisted-by: GitHub Copilot:claude-opus-4.8
* AVRO-4290: [python] Verify bzip2 stream EOF and drain fully
BZip2Codec.decompress now loops the BZ2Decompressor: it drains all buffered
output (bounded by the limit), verifies the stream reached EOF (rejecting a
truncated/corrupt block with InvalidAvroBinaryEncoding), and handles
concatenated bzip2 streams as bz2.decompress does. Add a truncated-block test.
Assisted-by: GitHub Copilot:claude-opus-4.8
* AVRO-4290: [python] Reject truncated deflate and short snappy blocks
Address review feedback:
- DeflateCodec: after flushing the decompressor, verify decompressor.eof so a
truncated/incomplete raw-deflate block is rejected with
InvalidAvroBinaryEncoding (zlib.decompress used to raise for this; the
decompressobj-based size cap otherwise silently accepted partial output).
- SnappyCodec: validate the block length is >= 4 before reading length - 4
bytes, raising a codec-specific InvalidAvroBinaryEncoding instead of falling
through to the generic decoder error.
Assisted-by: GitHub Copilot:claude-opus-4.8
* AVRO-4290: [python] Add a truncated-deflate-block rejection test
Cover the new DeflateCodec behavior (raising InvalidAvroBinaryEncoding when the
end-of-stream marker isn't reached) with a test that truncates a valid deflate
block and asserts decompress() rejects it, mirroring the bzip2 truncation test.
Assisted-by: GitHub Copilot:claude-opus-4.8
* AVRO-4290: [python] Bound the deflate flush() output too
The previous fix bounded decompress(data, limit+1) but then called
decompressor.flush() with no limit, so the residual output (from the
unconsumed input left by max_length) could still expand unbounded. Drain the
unconsumed_tail in a loop with a per-call max_length and pass a bounded length
to flush(), so the accumulated output can never exceed the limit by more than
one byte before being rejected.
Assisted-by: GitHub Copilot:claude-opus-4.8
* AVRO-4290: [python] Clamp AVRO_MAX_DECOMPRESS_LENGTH to sys.maxsize
_max_decompress_length() returned the raw parsed int. An absurdly large override
is passed as max_length to zlib/bz2 decompress(), which raise OverflowError when
it exceeds Py_ssize_t. Clamp to sys.maxsize so an oversized override is honored
as "effectively unbounded" instead of producing a confusing OverflowError.
Assisted-by: GitHub Copilot:claude-opus-4.8
* AVRO-4290: [python] Avoid limit+1 Py_ssize_t overflow at sys.maxsize
After clamping AVRO_MAX_DECOMPRESS_LENGTH to sys.maxsize, the deflate/bz2 paths
still computed `limit + 1` for the decompress() max_length, which overflows
Py_ssize_t (and raises OverflowError) when limit == sys.maxsize. Add
_decompress_read_ceiling(), which returns limit + 1 normally but sys.maxsize
when the limit is already sys.maxsize (no realistic block can exceed it), and
use it for the bounded read/flush in both paths.
Assisted-by: GitHub Copilot:claude-opus-4.8
> [python] Enforce a maximum decompressed block size
> --------------------------------------------------
>
> Key: AVRO-4290
> URL: https://issues.apache.org/jira/browse/AVRO-4290
> Project: Apache Avro
> Issue Type: Sub-task
> Components: python
> Affects Versions: 1.11.5, 1.12.1
> Reporter: Ismaël Mejía
> Assignee: Ismaël Mejía
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.2
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> The codecs in {{lang/py/avro/codecs.py}} decompress each block (deflate via
> {{zlib.decompress}} without a bufsize/limit, plus bzip2/snappy/zstandard)
> with no bound on the output size.
> Enforce a configurable maximum decompressed size and raise a clear error when
> a block would exceed it. Add a test with a high-ratio block.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
