Re: Has anyone developed a utility to tell what is missing from a record?

2013-04-06 Thread Scott Carey
Try GenericRecordBuilder. For the Specific API, there are builders that will not let you construct an object that can not be serialized. The Generic API should have the same thing, but I am not 100% sure the builder there covers it. I have always avoided using any API that allows me to create an

Has anyone developed a utility to tell what is missing from a record?

2013-04-04 Thread Jonathan Coveney
I'm working on migrating an internally developed serialization format to Avro. In the process, there have been many cases where I made a mistake migrating the schema (I've automated it), and then avro cries that a record I'm trying to serialize doesn't match the schema. Generally, the error it

Re: Has anyone developed a utility to tell what is missing from a record?

2013-04-04 Thread Jeremy Kahn
I think this would be tremendously useful. I am working - in my copious spare time - on improving schema validation in the Python library, and I think I can see how to improve things there by extending the data/schema parallel recursion to keep track of position in each. Jeremy On Apr 4, 2013

Re: Has anyone developed a utility to tell what is missing from a record?

2013-04-04 Thread Philip Zeyliger
Hi Jonathan, The python implementation is definitely less mature than the Java one. As you run into things, please do file bugs (and, better, yet, fixes!). At one point someone on this list was working on an alternative python implementation that generated python objects to represent the Avro

Re: Has anyone developed a utility to tell what is missing from a record?

2013-04-04 Thread Jonathan Coveney
Ok, cool. I've been using the python implementation pretty heavily and didn't realize that it was less mature. Will definitely work on maturing it where possible :) 2013/4/4 Philip Zeyliger phi...@cloudera.com Hi Jonathan, The python implementation is definitely less mature than the Java

Re: Has anyone developed a utility to tell what is missing from a record?

2013-04-04 Thread Jeremy Kahn
AVRO-1284 [0] has the first patch in improving the schema validation so that schema polymorphism handles validation. Philip and Jonathan, a review would be nice. Upvotes would be nice too, but constructive feedback is probably even better. I think there's a further generalization that allows