Re: Avro schema doesn't honor backward compatibilty

2016-02-02 Thread Raghvendra Singh 
Hi Ryan

Thanks for your answer. Here is what i am doing in my environment

1. Write the data using the old schema

*SpecificDatumWriter datumWriter = new
SpecificDatumWriter<>(SCHEMA_V1)*

2. Now trying to read the data written by the old schema using the new
schema

*DatumReader payloadReader = new SpecificDatumReader<>(*
*SCHEMA_V2**)*

In this case *SCHEMA_V1 *is the old schema which doesn't have the field
while SCHEMA_V2 is the new one which has the extra field.

Your suggestion *"You should run setSchema on your SpecificDatumReader to
set the schema the data was written with"*  is kind of work around where i
have to read the data with the schema it was written with and hence this is
not exactly backward compatible. Note that if i do this then i have to
maintain all the schemas while reading and somehow know which version the
data was written with and hence this will make schema evolution pretty
painful.

Please let me know if i didn't understand your email correctly or their is
something i missed.

-raghu

On Tue, Feb 2, 2016 at 9:19 AM, Ryan Blue  wrote:

> Hi Raghvendra,
>
> It looks like the problem is that you're using the new schema in place of
> the schema that the data was written with.  You should run setSchema on
> your SpecificDatumReader to set the schema the data was written with.
>
> What's happening is that the schema you're using, the new one, has the new
> field so Avro assumes it is present and tries to read it. By setting the
> schema that the data was actually written with, the datum reader will know
> that it isn't present and will use your default instead. When you read data
> encoded with the new schema, you need to use it as the written schema
> instead so the datum reader knows that the field should be read.
>
> Does that make sense?
>
> rb
>
> On 02/01/2016 12:31 PM, Raghvendra Singh wrote:
>
>> down votefavorite
>> <
>> http://stackoverflow.com/questions/34733604/avro-schema-doesnt-honor-backward-compatibilty#
>> >
>>
>>
>> I have this avro schema
>>
>> {
>>   "namespace": "xx..x.x",
>>   "type": "record",
>>   "name": "MyPayLoad",
>>   "fields": [
>>   {"name": "filed1",  "type": "string"},
>>   {"name": "filed2", "type": "long"},
>>   {"name": "filed3",  "type": "boolean"},
>>   {
>>"name" : "metrics",
>>"type":
>>{
>>   "type" : "array",
>>   "items":
>>   {
>>   "name": "MyRecord",
>>   "type": "record",
>>   "fields" :
>>   [
>> {"name": "min", "type": "long"},
>> {"name": "max", "type": "long"},
>> {"name": "sum", "type": "long"},
>> {"name": "count", "type": "long"}
>>   ]
>>   }
>>}
>>   }
>>]}
>>
>> Here is the code which we use to parse the data
>>
>> public static final MyPayLoad parseBinaryPayload(byte[] payload) {
>>  DatumReader payloadReader = new
>> SpecificDatumReader<>(MyPayLoad.class);
>>  Decoder decoder = DecoderFactory.get().binaryDecoder(payload,
>> null);
>>  MyPayLoad myPayLoad = null;
>>  try {
>>  myPayLoad = payloadReader.read(null, decoder);
>>  } catch (IOException e) {
>>  logger.log(Level.SEVERE, e.getMessage(), e);
>>  }
>>
>>  return myPayLoad;
>>  }
>>
>> Now i want to add one more field int the schema so the schema looks like
>> below
>>
>>   {
>>   "namespace": "xx..x.x",
>>   "type": "record",
>>   "name": "MyPayLoad",
>>   "fields": [
>>   {"name": "filed1",  "type": "string"},
>>   {"name": "filed2", "type": "long"},
>>   {"name": "filed3",  "type": "boolean"},
>>   {
>>"name" : "metrics",
>>"type":
>>{
>>   "type" : "array",
>>   "items":
>>   {
>>   "name": "MyRecord",
>>   "type": "record",
>>   "fields" :
>>   [
>> {"name": "min", "type": "long"},
>> {"name": "max", "type": "long"},
>> {"name": "sum", "type": "long"},
>> {"name": "count", "type": "long"}
>>   ]
>>   }
>>}
>>   }
>>   {"name": "agentType",  "type": ["null", "string"], "default":
>> "APP_AGENT"}
>>]}
>>
>> Note the filed added and also the default is defined. The problem is that
>> if we receive the data which was written using the older schema i get this
>> error
>>
>> java.io.EOFException: null
>>  at org.apache.avro.io
>> .BinaryDecoder.ensureBounds(BinaryDecoder.java:473)
>> ~[avro-1.7.4.jar:1.7.4]
>>  at org.apache.avro.io.BinaryDecoder.readInt(BinaryDecoder.java:128)
>> ~[avro-1.7.4.jar:1.7.4]
>>  at

Re: Avro schema doesn't honor backward compatibilty

2016-02-02 Thread Raghvendra Singh 
Great, Thank you very much guys, this works. Very much appreciated.

On Tue, Feb 2, 2016 at 12:46 PM, kulkarni.swar...@gmail.com <
kulkarni.swar...@gmail.com> wrote:

> Raghvendra,
>
> You need to use
>
> *DatumReader payloadReader = new
> SpecificDatumReader<>(SCHEMA_V1, **SCHEMA_V2**)*
>
> So you provide both writer(SCHEMA_V1) and reader(SCHMEA_V2) to avro. In
> your current case avro is assuming both to be the same which is certainly
> not the case and hence it is failing. I think this is what Ryan was
> referring to as well.
>
> Hope that helps.
>
>
>
> On Tue, Feb 2, 2016 at 1:44 PM, Raghvendra Singh 
> wrote:
>
>> Hi Ryan
>>
>> Thanks for your answer. Here is what i am doing in my environment
>>
>> 1. Write the data using the old schema
>>
>> *SpecificDatumWriter datumWriter = new
>> SpecificDatumWriter<>(SCHEMA_V1)*
>>
>> 2. Now trying to read the data written by the old schema using the new
>> schema
>>
>> *DatumReader payloadReader = new
>> SpecificDatumReader<>(**SCHEMA_V2**)*
>>
>> In this case *SCHEMA_V1 *is the old schema which doesn't have the field
>> while SCHEMA_V2 is the new one which has the extra field.
>>
>> Your suggestion *"You should run setSchema on your SpecificDatumReader
>> to set the schema the data was written with"*  is kind of work around
>> where i have to read the data with the schema it was written with and hence
>> this is not exactly backward compatible. Note that if i do this then i have
>> to maintain all the schemas while reading and somehow know which version
>> the data was written with and hence this will make schema evolution pretty
>> painful.
>>
>> Please let me know if i didn't understand your email correctly or their
>> is something i missed.
>>
>> -raghu
>>
>> On Tue, Feb 2, 2016 at 9:19 AM, Ryan Blue  wrote:
>>
>>> Hi Raghvendra,
>>>
>>> It looks like the problem is that you're using the new schema in place
>>> of the schema that the data was written with.  You should run setSchema on
>>> your SpecificDatumReader to set the schema the data was written with.
>>>
>>> What's happening is that the schema you're using, the new one, has the
>>> new field so Avro assumes it is present and tries to read it. By setting
>>> the schema that the data was actually written with, the datum reader will
>>> know that it isn't present and will use your default instead. When you read
>>> data encoded with the new schema, you need to use it as the written schema
>>> instead so the datum reader knows that the field should be read.
>>>
>>> Does that make sense?
>>>
>>> rb
>>>
>>> On 02/01/2016 12:31 PM, Raghvendra Singh wrote:
>>>
 down votefavorite
 <
 http://stackoverflow.com/questions/34733604/avro-schema-doesnt-honor-backward-compatibilty#
 >


 I have this avro schema

 {
   "namespace": "xx..x.x",
   "type": "record",
   "name": "MyPayLoad",
   "fields": [
   {"name": "filed1",  "type": "string"},
   {"name": "filed2", "type": "long"},
   {"name": "filed3",  "type": "boolean"},
   {
"name" : "metrics",
"type":
{
   "type" : "array",
   "items":
   {
   "name": "MyRecord",
   "type": "record",
   "fields" :
   [
 {"name": "min", "type": "long"},
 {"name": "max", "type": "long"},
 {"name": "sum", "type": "long"},
 {"name": "count", "type": "long"}
   ]
   }
}
   }
]}

 Here is the code which we use to parse the data

 public static final MyPayLoad parseBinaryPayload(byte[] payload) {
  DatumReader payloadReader = new
 SpecificDatumReader<>(MyPayLoad.class);
  Decoder decoder = DecoderFactory.get().binaryDecoder(payload,
 null);
  MyPayLoad myPayLoad = null;
  try {
  myPayLoad = payloadReader.read(null, decoder);
  } catch (IOException e) {
  logger.log(Level.SEVERE, e.getMessage(), e);
  }

  return myPayLoad;
  }

 Now i want to add one more field int the schema so the schema looks like
 below

   {
   "namespace": "xx..x.x",
   "type": "record",
   "name": "MyPayLoad",
   "fields": [
   {"name": "filed1",  "type": "string"},
   {"name": "filed2", "type": "long"},
   {"name": "filed3",  "type": "boolean"},
   {
"name" : "metrics",
"type":
{
   "type" : "array",
   "items":
   {
   "name": "MyRecord",

Re: Avro schema doesn't honor backward compatibilty

2016-02-02 Thread Ryan Blue 

Hi Raghvendra,

It looks like the problem is that you're using the new schema in place 
of the schema that the data was written with.  You should run setSchema 
on your SpecificDatumReader to set the schema the data was written with.


What's happening is that the schema you're using, the new one, has the 
new field so Avro assumes it is present and tries to read it. By setting 
the schema that the data was actually written with, the datum reader 
will know that it isn't present and will use your default instead. When 
you read data encoded with the new schema, you need to use it as the 
written schema instead so the datum reader knows that the field should 
be read.


Does that make sense?

rb

On 02/01/2016 12:31 PM, Raghvendra Singh wrote:

down votefavorite


I have this avro schema

{
  "namespace": "xx..x.x",
  "type": "record",
  "name": "MyPayLoad",
  "fields": [
  {"name": "filed1",  "type": "string"},
  {"name": "filed2", "type": "long"},
  {"name": "filed3",  "type": "boolean"},
  {
   "name" : "metrics",
   "type":
   {
  "type" : "array",
  "items":
  {
  "name": "MyRecord",
  "type": "record",
  "fields" :
  [
{"name": "min", "type": "long"},
{"name": "max", "type": "long"},
{"name": "sum", "type": "long"},
{"name": "count", "type": "long"}
  ]
  }
   }
  }
   ]}

Here is the code which we use to parse the data

public static final MyPayLoad parseBinaryPayload(byte[] payload) {
 DatumReader payloadReader = new
SpecificDatumReader<>(MyPayLoad.class);
 Decoder decoder = DecoderFactory.get().binaryDecoder(payload, null);
 MyPayLoad myPayLoad = null;
 try {
 myPayLoad = payloadReader.read(null, decoder);
 } catch (IOException e) {
 logger.log(Level.SEVERE, e.getMessage(), e);
 }

 return myPayLoad;
 }

Now i want to add one more field int the schema so the schema looks like
below

  {
  "namespace": "xx..x.x",
  "type": "record",
  "name": "MyPayLoad",
  "fields": [
  {"name": "filed1",  "type": "string"},
  {"name": "filed2", "type": "long"},
  {"name": "filed3",  "type": "boolean"},
  {
   "name" : "metrics",
   "type":
   {
  "type" : "array",
  "items":
  {
  "name": "MyRecord",
  "type": "record",
  "fields" :
  [
{"name": "min", "type": "long"},
{"name": "max", "type": "long"},
{"name": "sum", "type": "long"},
{"name": "count", "type": "long"}
  ]
  }
   }
  }
  {"name": "agentType",  "type": ["null", "string"], "default": "APP_AGENT"}
   ]}

Note the filed added and also the default is defined. The problem is that
if we receive the data which was written using the older schema i get this
error

java.io.EOFException: null
 at org.apache.avro.io.BinaryDecoder.ensureBounds(BinaryDecoder.java:473)
~[avro-1.7.4.jar:1.7.4]
 at org.apache.avro.io.BinaryDecoder.readInt(BinaryDecoder.java:128)
~[avro-1.7.4.jar:1.7.4]
 at org.apache.avro.io.BinaryDecoder.readIndex(BinaryDecoder.java:423)
~[avro-1.7.4.jar:1.7.4]
 at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:229)
~[avro-1.7.4.jar:1.7.4]
 at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
~[avro-1.7.4.jar:1.7.4]
 at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206)
~[avro-1.7.4.jar:1.7.4]
 at 
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:152)
~[avro-1.7.4.jar:1.7.4]
 at 
org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:177)
~[avro-1.7.4.jar:1.7.4]
 at 
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:148)
~[avro-1.7.4.jar:1.7.4]
 at 
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:139)
~[avro-1.7.4.jar:1.7.4]
 at 
com.appdynamics.blitz.shared.util.X.parseBinaryPayload(BlitzAvroSharedUtil.java:38)
~[blitz-shared.jar:na]

What i understood from this

document
that this should have been backward compatible but somehow that doesn't
seem to be the case. Any idea what i am doing wrong?




--
Ryan Blue
Software Engineer
Cloudera, Inc.