Re: Issue with reading old data with a new Avro Schema

[Lukas Steiblys]

The schema is not valid JSON. Maybe you forgot the “[“ after “fields:”?

Lukas
From: Nicolas Phung 
Sent: Wednesday, April 8, 2015 9:45 AM
To: user@avro.apache.org 
Subject: Issue with reading old data with a new Avro Schema

Hello, 

I'm trying to read old avro binary data with a new schema (I add a new field).


This is the Avro Schema (OLD) I was using to write Avro binary data before:
{
    "namespace": "com.hello.world",
    "type": "record",
    "name": "Toto",
    "fields": 
        {
            "name": "a",
            "type": [
                "string",
                "null"
            ]
        },
        {
            "name": "b",
            "type": "string"
        }
    ]
}

This is the Avro Schema (NEW) I'm using to read the Avro binary data :

{
    "namespace": "com.hello.world",
    "type": "record",
    "name": "Toto",
    "fields": 
        {
            "name": "a",
            "type": [
                "string",
                "null"
            ]
        },
        {
            "name": "b",
            "type": "string"
        },
        {
            "name": "c",
            "type": "string",
            "default": "na"
        }
    ]
}

However, I can't read the old data with the new Schema. I've got the following 
errors :

15/04/08 17:32:22 ERROR executor.Executor: Exception in task 0.0 in stage 3.0 
(TID 3)
java.io.EOFException
at org.apache.avro.io.BinaryDecoder.ensureBounds(BinaryDecoder.java:473)
at org.apache.avro.io.BinaryDecoder.readInt(BinaryDecoder.java:128)
at org.apache.avro.io.BinaryDecoder.readString(BinaryDecoder.java:259)
at org.apache.avro.io.BinaryDecoder.readString(BinaryDecoder.java:272)
at org.apache.avro.io.ValidatingDecoder.readString(ValidatingDecoder.java:113)
at 
org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:353)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:157)
at 
org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
at 
org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
at com.miguno.kafka.avro.AvroDecoder.fromBytes(AvroDecoder.scala:31)

>From my understanding, I should be able to read the old data with the new 
>schema that contains a new field with a default value. But it doesn't seem to 
>work. Am I doing something wrong ?

I have posted a report https://issues.apache.org/jira/browse/AVRO-1661

Regards,
Nicolas PHUNG