Ramin Gharib created FLINK-41361:
------------------------------------
Summary: Bound VARIANT object and array sizes by the value length
before allocating
Key: FLINK-41361
URL: https://issues.apache.org/jira/browse/FLINK-41361
Project: Flink
Issue Type: Bug
Components: API / Core
Reporter: Ramin Gharib
A malformed VARIANT value whose object or array header claims more elements
than its bytes hold makes readers allocate arrays of that size.
Example: metadata {{01 00 00}} and value {{42 ff ff ff 7f}}. The value is 5
bytes, but its header declares an object with 2,147,483,647 fields.
{{toJson()}}, {{toString()}} and {{JSON_STRING}} allocate {{new
int[2147483647]}} and throw an OutOfMemoryError. A VariantTypeException is the
expected failure.
A value can reach these readers without being decoded first through formats
that store the binary encoding, such as {{'raw.variant-encoding' = 'binary'}}.
*Where*
- {{BinaryVariantUtil.handleObject}} (L790) and {{handleArray}} (L843) read the
element count from the header and pass it to the handler as is.
- Allocations sized by that count: {{JsonVariantFormatter}} L126, L127, L146;
{{BinaryVariant.getFieldNames}} L313; {{BinaryVariantInternalBuilder}}
L734-735, when it appends an existing variant.
*Proposal*
In {{handleObject}} and {{handleArray}}, check that the id and offset tables
fit in the value before calling the handler, computed in long arithmetic, and
throw a VariantTypeException such as "The VARIANT object header declares more
fields than the value holds." Every element takes at least one id byte and one
offset byte, so this bounds every allocation by the value length. Values are
still read as is, and the check runs only when a reader walks the container.
Spark has the same {{handleObject}}. Its {{toJson}} streams fields with
bounds-checked reads and fails with MALFORMED_VARIANT. Its
{{VariantBuilder.appendVariant}} has the same {{new ArrayList<>(size)}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
