We've recently upgraded our Avro dependency to 1.7.7 from 1.7.5. The avdl
for Avro used to look like this:
record Event{
long creationTime;
/**
* The optional payload of the event.
*/
union{null, bytes} value = null;
/**
* Optional properties of the event.
*/
map<string> properties = null;
}
When we upgraded we started seeing warnings like this:
[WARNING] Avro: Invalid default for field properties: null not a
{"type":"map","values":"string"}
So we converted the file to be this:
record Event{
long creationTime;
/**
* The optional payload of the event.
*/
union{null, bytes} value = null;
/**
* Optional properties of the event.
*/
map<string> properties = {};
}
We also had the need to add a new field to the record and thought we could
do so passively like so:
record Event{
long creationTime;
/**
* The optional payload of the event.
*/
union{null, bytes} value = null;
/**
* Optional properties of the event.
*/
map<string> properties = {};
/**
* Type of operation
*/
union{null, string} operation = null;
}
However when we then read data that was written with the very first schema
we get an EOFException.
Caused by: java.io.EOFException
at org.apache.avro.io.BinaryDecoder.ensureBounds(BinaryDecoder.java:473)
at org.apache.avro.io.BinaryDecoder.readInt(BinaryDecoder.java:128)
at org.apache.avro.io.BinaryDecoder.readIndex(BinaryDecoder.java:423)
at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:290)
at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:267)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:155)
at
org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
at
org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
We are reading with a BufferedBinaryDecoder and using the new schema as
both the written and reader schema because the written schema is not
preserved with the payload so it is not easy to retrieve.
My questions are:
1. Is the change we made to add a new defaulted union truly non-passive?
2. Is there a workaround so I can continue to evolve my schema?
Thanks for the help,
Micah
