Thanks Doug. I'll file enhancement requests within the next few days.
I agree that avrogen or other higher level tools can remedy almost all of the
concerns with respect to schema definition and maintenance. Avrogen is still
very young right now, I have not tried it yet. Based on the discussion around
it in the original ticket I wasn't sure if it was meant to address all of these
sorts of issues or if it was more a parallel track to using json schemas
directly. If it is intended to be the primary place that users of Avro define
and maintain schemas, I'll check it out soon!
On Jan 13, 2010, at 10:52 AM, Doug Cutting wrote:
> Scott Carey wrote:
>> {"name": "ipAddr", "type": [
>> {"name": "IPv4", "type": "fixed", "size": 4 },
>> {"name": "IPv6", "type": "fixed", "size": 16 }
>> ]
>> }
>> This won't work, even though these have their own names and generate their
>> own classes.
>
> That should work: if it doesn't, that's a bug. It should be possible to
> include any two named types in a union, so long as their names differ.
>
Its a bug I guess. I'll file it with the others this week:
org.apache.avro.SchemaParseException: No type:
{"name":"ipAddr","type":[{"name":"IPv4","type":"fixed","size":4},{"name":"IPv6","type":"fixed","size":16}]}
That is on trunk from just before the directory structure change.
>
>> * In Memory data representation
>> Avro is very good at reducing serialized size, but doesn't optimize
>> memory footprint. None of this is a big deal for the typical Hadoop use
>> case, but for my use cases where I want to serialize these things into BDB's
>> or some other key/value store -- in memory footprint is critical. Extra
>> nested object references can easily consume a lot of memory and reduces the
>> effectiveness of in memory caching for key/value stores. Another time you
>> would want to make sure minimum memory is used is in a map side join.
>> Although some can be done to trim up the Specific API, I think that an
>> annotations based approach (with ASM) will be the most flexible and powerful
>> in the long run. For example a fixed size object can just be a byte[] with
>> an annotation, so that ASM knows how to decorate the setter/getter to
>> enforce the size and what the Avro properties for the field are, etc. --
>> rather than having to be its own object that inherits from an abstract fixed
>> type. Much other naming can collapse from objects to methods/annotations
>> this way when generating classes from schemas, and vice-versa to generate
>> schemas from classes without otherwise altering them.
>
> An annotation-based API might indeed be useful.
>
> Also, have you looked at using ResolvingDecoder and ValidatingEncoder?
> These, with little-or-no-performance penalty, allow you to write code
> that uses an arbitrary in-memory representation. As an example of this
> style, look at the code in
> https://issues.apache.org/jira/browse/AVRO-251, notably
> Schema#readSchema(), writeSchema, readJson and writeJson. With
> generated code, folks often need to write wrapper code to convert from a
> possibly pre-existing, manually-maintained representation to the
> generated representation and vice-versa. Writing to the Encoder/Decoder
> API directly and using ResolvingDecoder and ValidatingEncoder requires
> about the same amount of code, is just as safe, handles versioning, and
> bypasses the intermediate representation altogether.
>
I'll definitely check out that API.
Thanks!
-Scott