Robert Yokota created AVRO-4361:
-----------------------------------
Summary: [Java] Collection size guard rejects valid data when the
reader's element schema encodes larger than the writer's
Key: AVRO-4361
URL: https://issues.apache.org/jira/browse/AVRO-4361
Project: Apache Avro
Issue Type: Bug
Components: java
Affects Versions: 1.12.2
Reporter: Robert Yokota
Assignee: Robert Yokota
Fix For: 1.13.0, 1.12.3
The collection size guard added by AVRO-4241 (and extended to the fast reader
by AVRO-4300 and to ReflectDatumReader by AVRO-4324) computes the minimum
bytes
per array element, or map value, from the reader's schema. But the elements
were encoded with the writer's schema. When the reader's element type has a
larger minimum encoded size than the writer's, the guard rejects valid data
with an EOFException.
Schema resolution allows this in several compatible cases:
- The reader's record adds a field with a default. Example: the writer has
Item {id: int} and the reader has Item {id: int, qty: int (default 0)}.
- Type promotion: int, long or float read as double (8 bytes in the guard),
or int or long read as float (4 bytes).
- A non-union writer's element read as a branch of a reader's union. Example:
array<null>, which is 0 bytes per element, read as array<["null","int"]>,
which the guard counts as at least 1 byte.
- The same cases for map values.
These are backward-compatible schema changes, and 1.12.1 decodes the same
bytes. The guard is only active when the decoder knows how many bytes remain
(e.g. BinaryDecoder over a byte array), which is the common case for
message-oriented use such as Kafka.
Repro (classic and fast reader both fail):
```
String w =
"{\"type\":\"record\",\"name\":\"Order\",\"fields\":[{\"name\":\"items\","
+
"\"type\":{\"type\":\"array\",\"items\":{\"type\":\"record\",\"name\":\"Item\","
+ "\"fields\":[{\"name\":\"id\",\"type\":\"int\"}]}}}]}";
String r = w.replace("{\"name\":\"id\",\"type\":\"int\"}",
"{\"name\":\"id\",\"type\":\"int\"},{\"name\":\"qty\",\"type\":\"int\",\"default\":0}");
Schema writer = new Schema.Parser().parse(w);
Schema reader = new Schema.Parser().parse(r);
// write an Order with 100 Items, each with id = 0 (1 byte per element)
...
new GenericDatumReader<>(writer, reader).read(null,
DecoderFactory.get().binaryDecoder(bytes, null));
```
1.12.1: decodes 100 items.
1.12.2: java.io.EOFException: Collection claims 100 elements with at least 2
bytes
each, but only 101 bytes are available
Cause:
- GenericDatumReader.readArray/readMap and ReflectDatumReader.readArray pass
the
reader's element schema (expected.getElementType()) to
ensureAvailableCollectionBytes.
- FastReaderBuilder.createArrayReader does the same with
readerSchema.getElementType(), although the writer's schema is available
there as action.writer.
- The zero-byte allocation cap (isZeroByteSchema) is chosen from the reader's
schema in the same way.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
