Hello! Schema evolution relies on both the writer and reader schemas being available.
It looks like the allegro tool you are using is using the GenericDatumReader that assumes the reader and writer schema are the same: https://github.com/allegro/json-avro-converter/blob/json-avro-converter-0.2.8/converter/src/main/java/tech/allegro/schema/json2avro/converter/JsonAvroConverter.java#L83 I do not believe that the "default" value is taken into account for data that is strictly missing from the binary input, just when a field is known to be in the reader schema but missing from the original writer. You may have more luck reading the GenericRecord with a GenericDatumReader with both schemas, and using the `convertToJson(record)`. I hope this is useful -- Ryan On Tue, Jul 30, 2019 at 10:20 AM Martin Mucha <alfon...@gmail.com> wrote: > > Hi, > > I've got some issues/misunderstanding of AVRO schema evolution. > > When reading through avro documentation, for example [1], I understood, that > schema evolution is supported, and if I added column with specified default, > it should be backwards compatible (and even forward when I remove it again). > Sounds great, so I added column defined as: > > { > "name": "newColumn", > "type": ["null","string"], > "default": null, > "doc": "something wrong" > } > > and try to consumer some topic having this schema from beginning, it fails > with message: > > Caused by: java.lang.ArrayIndexOutOfBoundsException: 5 > at > org.apache.avro.io.parsing.Symbol$Alternative.getSymbol(Symbol.java:424) > at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:290) > at org.apache.avro.io.parsing.Parser.advance(Parser.java:88) > at > org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:267) > at > org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153) > at > org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:232) > at > org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:222) > at > org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:175) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153) > at > org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153) > at > org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:232) > at > org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:222) > at > org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:175) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:145) > at > tech.allegro.schema.json2avro.converter.JsonAvroConverter.convertToJson(JsonAvroConverter.java:83) > to give a little bit more information. Avro schema defines one top level > type, having 2 fields. String describing type of message, and union of N > types. All N-1, non-modified types can be read, but one updated with > optional, default-having column cannot be read. I'm not sure if this design > is strictly speaking correct, but that's not the point (feel free to > criticise and recommend better approach!). I'm after schema evolution, which > seems not to be working. > > > And if we alter type definition to: > > "type": "string", > "default": "" > it still does not work and generated error is: > > Caused by: org.apache.avro.AvroRuntimeException: Malformed data. Length is > negative: -1 > at org.apache.avro.io.BinaryDecoder.doReadBytes(BinaryDecoder.java:336) > at org.apache.avro.io.BinaryDecoder.readString(BinaryDecoder.java:263) > at > org.apache.avro.io.ResolvingDecoder.readString(ResolvingDecoder.java:201) > at > org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:422) > at > org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:414) > at > org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:181) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153) > at > org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:232) > at > org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:222) > at > org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:175) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153) > at > org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153) > at > org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:232) > at > org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:222) > at > org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:175) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:145) > at > tech.allegro.schema.json2avro.converter.JsonAvroConverter.convertToJson(JsonAvroConverter.java:83) > > Am I doing something wrong? > > thanks, > Martin. > > [1] > https://docs.oracle.com/database/nosql-12.1.3.4/GettingStartedGuide/schemaevolution.html#changeschema-rules
