[
https://issues.apache.org/jira/browse/AVRO-4323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismaël Mejía resolved AVRO-4323.
--------------------------------
Resolution: Fixed
> [Java] Bound DataFileStream block size against available input before
> allocating the block buffer
> -------------------------------------------------------------------------------------------------
>
> Key: AVRO-4323
> URL: https://issues.apache.org/jira/browse/AVRO-4323
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
> Reporter: Ismaël Mejía
> Assignee: Ismaël Mejía
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.2
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> When reading an Avro data (container) file, {{DataFileStream}} reads each
> block's declared size as a long and currently validates it only against the
> {{Integer}} range before allocating the block {{byte[]}} buffer (in
> {{DataFileStream.DataBlock}}). For a malformed, corrupted, or truncated file,
> the declared block size can be much larger than the number of bytes actually
> present, so the reader eagerly allocates a very large buffer on the first
> {{hasNext()}}/{{next()}} call before any block byte has been read.
> We should make block reading more robust for malformed input:
> * validate the declared block size against the number of bytes actually
> remaining in the input (when that is known, e.g. a seekable/known-length
> source), and/or
> * route the block-size allocation through the existing {{SystemLimits}}
> configuration already used for bytes/collection sizing elsewhere in the
> reader,
> so that reading a malformed file fails fast with a clear {{IOException}}
> instead of attempting an oversized allocation.
> This can be reproduced with a small hand-crafted file whose block header
> declares a size near {{Integer.MAX_VALUE}} while carrying no block bytes. A
> valid file continues to read normally.
> Relevant code:
> * {{lang/java/avro/src/main/java/org/apache/avro/file/DataFileStream.java}} —
> {{hasNextBlock()}} (block-size read/validation) and {{DataBlock}} constructor
> ({{new byte[blockSize]}}).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
