[
https://issues.apache.org/jira/browse/AVRO-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismaël Mejía updated AVRO-4325:
-------------------------------
Fix Version/s: 1.12.2
> [Trevni] Validate column-file header counts and lengths before allocating in
> the Trevni readers
> -----------------------------------------------------------------------------------------------
>
> Key: AVRO-4325
> URL: https://issues.apache.org/jira/browse/AVRO-4325
> Project: Apache Avro
> Issue Type: Improvement
> Components: trevni
> Reporter: Ismaël Mejía
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.2
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The Trevni columnar reader sizes several allocations directly from values
> read from the file header/metadata without validating them against the input
> actually available. For a malformed, corrupted, or truncated Trevni file
> these counts/lengths may greatly exceed the bytes actually present, leading
> to oversized eager allocations or an unexpected
> {{NegativeArraySizeException}}:
> * {{ColumnFileReader.readHeader}} allocates {{new
> ColumnDescriptor[columnCount]}} (and a {{HashMap}} sized to {{columnCount}})
> from the 32-bit column count before any column metadata is read.
> * {{ColumnDescriptor.ensureBlocksRead}} allocates {{new
> BlockDescriptor[blockCount]}} from the block count.
> * {{InputBuffer.readBytes}}/{{readString}} allocate {{new byte[readInt()]}}
> from a length prefix.
> * {{ColumnValues.startBlock}} computes {{new byte[compressedSize +
> checksumSize]}}; the addition can overflow to a negative size
> ({{NegativeArraySizeException}}).
> Unlike the Java SDK, Trevni has no shared limit/validation layer. We should:
> * validate these counts/lengths against the remaining input (and a sane
> maximum) before allocating,
> * adopt a consistent bound across the Trevni readers ({{ColumnFileReader}},
> {{ColumnDescriptor}}, {{InputBuffer}}, {{ColumnValues}}), similar in spirit
> to the Java SDK's {{SystemLimits}}, and
> * guard the {{compressedSize + checksum}} addition against overflow,
> so that reading a malformed file fails fast with a clear exception. Valid
> files continue to read normally.
> Relevant code:
> *
> {{lang/java/trevni/core/src/main/java/org/apache/trevni/ColumnFileReader.java}}
> — {{readHeader()}}.
> *
> {{lang/java/trevni/core/src/main/java/org/apache/trevni/ColumnDescriptor.java}}
> — {{ensureBlocksRead()}}.
> * {{lang/java/trevni/core/src/main/java/org/apache/trevni/InputBuffer.java}}
> — {{readBytes()}}/{{readString()}}.
> * {{lang/java/trevni/core/src/main/java/org/apache/trevni/ColumnValues.java}}
> — {{startBlock()}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
