Avro versioning and SpecificDatum's

2011-09-19 Thread Alex Holmes
Hi, I'm starting to play with how I can support versioning with Avro. I created an initial schema, code-generated some some Java classes using org.apache.avro.tool.Main compile protocol, and then used the DataFileWriter (with a SpecificDatumWriter) to serialize my objects to a file. I then

Re: Avro versioning and SpecificDatum's

2011-09-19 Thread Chris Wilkes
I'm interested in this as well, for now I've put my versioning in the package namespace of avro definition, ie: com.example.avro.v1.Car com.example.avro.v2.Car After all my documents that had the v1.Car have been reprocessed and are out of use I delete the old definition. Chris On Mon, Sep

Re: Avro versioning and SpecificDatum's

2011-09-19 Thread Doug Cutting
On 09/19/2011 05:12 AM, Alex Holmes wrote: I then modified my original schema by adding, deleting and renaming some fields, creating version 2 of the schema. After re-creating the Java classes I attempted to read the version 1 file using the DataFileStream (with a SpecificDatumReader), and

Re: Avro versioning and SpecificDatum's

2011-09-19 Thread Scott Carey
I version with SpecificDatum objects using avro data files and it works fine. I have seen problems arise if a user is configuring or reconfiguring the schemas on the DatumReader passed into the construction of the DataFileReader. In the case of SpecificDatumReader, it is as simple as:

Re: Avro ruby gem 1.3.3 md5 issues with 1.9

2011-09-19 Thread Doug Cutting
I just got access from Jeff to Avro at RubyGems.org and uploaded the 1.5.4 release. https://rubygems.org/gems/avro I'll make this a part of the release procedure from now on. Doug On 09/15/2011 11:34 PM, Jeremy Hinegardner wrote: On Thu, Sep 15, 2011 at 10:34:11AM -0500, Jeff Hansen wrote: I

Warnings in C++ header files generated by gen-cppcode.py

2011-09-19 Thread Vivek Nadkarni
Hi - The Avro C++ documentation (http://avro.apache.org/docs/current/api/cpp/html/index.html) shows an example of a C++ object generated using the python script gen-cppcode.py, to represent the data for an example schema. I followed the example to generate the example.hh file as described in the

Re: Avro ruby gem 1.3.3 md5 issues with 1.9

2011-09-19 Thread Jeff Hansen
Thanks -- it worked great when I installed it from the gem on github -- it's definitely first instinct to go after the one on rubygems.org though. On Mon, Sep 19, 2011 at 5:25 PM, Doug Cutting cutt...@apache.org wrote: I just got access from Jeff to Avro at RubyGems.org and uploaded the 1.5.4

Re: Avro versioning and SpecificDatum's

2011-09-19 Thread Alex Holmes
I'm trying to put together a simple test case to reproduce the exception. While I was creating the test case, I hit this behavior which doesn't seem right, but maybe it's my misunderstanding on how forward/backward compatibility should work: Schema v1: {name: Record, type: record, fields: [

Re: Avro versioning and SpecificDatum's

2011-09-19 Thread Alex Holmes
OK, I was able to reproduce the exception. v1: {name: Record, type: record, fields: [ {name: name, type: string}, {name: id, type: int} ] } v2: {name: Record, type: record, fields: [ {name: name_rename, type: string, aliases: [name]} ] } Step 1. Write Avro file using v1