[
https://issues.apache.org/jira/browse/AVRO-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18102493#comment-18102493
]
ASF subversion and git services commented on AVRO-4325:
-------------------------------------------------------
Commit 718e198322785109f4182397fa381fa6671e2d48 in avro's branch
refs/heads/branch-1.12 from Ismaël Mejía
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=718e198322 ]
AVRO-4325: [Trevni] Validate column-file header counts and lengths before
allocating (#3921)
* AVRO-4325: [Trevni] Validate header counts and lengths before allocating
The Trevni readers sized several allocations directly from values read from the
file header/metadata without validating them against the input available. For a
malformed, corrupted, or truncated file these counts/lengths could greatly
exceed the bytes present, driving oversized allocations, or overflow to a
negative size.
Add a shared InputBuffer.checkLength/remaining helper that rejects a negative
value and one that could not be backed by the bytes remaining, and apply it to:
- ColumnFileReader.readHeader (column count)
- ColumnDescriptor.ensureBlocksRead (block count)
- InputBuffer.readBytes/readString (length-prefixed byte arrays)
- ColumnValues.startBlock (compressed block size), which now also uses
Math.addExact for the checksum size to guard against integer overflow.
Reading a malformed file now fails fast with a clear IOException; valid files
read unchanged.
* AVRO-4325: Address review: validate compressed size plus checksum; derive
test offset
ColumnValues.startBlock now validates the combined
compressed-block-plus-checksum
length against the bytes remaining (computed in long to avoid overflow) and
rejects a negative or overflowing size with an IOException, instead of using
Math.addExact (which could throw an unchecked ArithmeticException) and
validating
only the compressed size. The TestColumnFile column-count offset is now derived
from ColumnFileWriter.MAGIC.length + Long.BYTES rather than a hard-coded 12.
> [Trevni] Validate column-file header counts and lengths before allocating in
> the Trevni readers
> -----------------------------------------------------------------------------------------------
>
> Key: AVRO-4325
> URL: https://issues.apache.org/jira/browse/AVRO-4325
> Project: Apache Avro
> Issue Type: Improvement
> Components: trevni
> Reporter: Ismaël Mejía
> Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The Trevni columnar reader sizes several allocations directly from values
> read from the file header/metadata without validating them against the input
> actually available. For a malformed, corrupted, or truncated Trevni file
> these counts/lengths may greatly exceed the bytes actually present, leading
> to oversized eager allocations or an unexpected
> {{NegativeArraySizeException}}:
> * {{ColumnFileReader.readHeader}} allocates {{new
> ColumnDescriptor[columnCount]}} (and a {{HashMap}} sized to {{columnCount}})
> from the 32-bit column count before any column metadata is read.
> * {{ColumnDescriptor.ensureBlocksRead}} allocates {{new
> BlockDescriptor[blockCount]}} from the block count.
> * {{InputBuffer.readBytes}}/{{readString}} allocate {{new byte[readInt()]}}
> from a length prefix.
> * {{ColumnValues.startBlock}} computes {{new byte[compressedSize +
> checksumSize]}}; the addition can overflow to a negative size
> ({{NegativeArraySizeException}}).
> Unlike the Java SDK, Trevni has no shared limit/validation layer. We should:
> * validate these counts/lengths against the remaining input (and a sane
> maximum) before allocating,
> * adopt a consistent bound across the Trevni readers ({{ColumnFileReader}},
> {{ColumnDescriptor}}, {{InputBuffer}}, {{ColumnValues}}), similar in spirit
> to the Java SDK's {{SystemLimits}}, and
> * guard the {{compressedSize + checksum}} addition against overflow,
> so that reading a malformed file fails fast with a clear exception. Valid
> files continue to read normally.
> Relevant code:
> *
> {{lang/java/trevni/core/src/main/java/org/apache/trevni/ColumnFileReader.java}}
> — {{readHeader()}}.
> *
> {{lang/java/trevni/core/src/main/java/org/apache/trevni/ColumnDescriptor.java}}
> — {{ensureBlocksRead()}}.
> * {{lang/java/trevni/core/src/main/java/org/apache/trevni/InputBuffer.java}}
> — {{readBytes()}}/{{readString()}}.
> * {{lang/java/trevni/core/src/main/java/org/apache/trevni/ColumnValues.java}}
> — {{startBlock()}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
