[ https://issues.apache.org/jira/browse/AVRO-1704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352355#comment-15352355 ]
Ryan Blue commented on AVRO-1704: --------------------------------- [~nielsbasjes], sorry it's taken so long for me to get back to you on this. On the spec: * I think we should go with the header: 0xC3 0x01. The first byte makes it easily recognizable as you suggest and meets my requirement of minimizing the number of non-Avro payloads that match. Using 0x01 makes it easy to see the version and will prevent programs confusing payloads with text as Doug suggests. * I don't see much value in reserving space in the second byte. I don't think there will be many formats for serializing Avro payloads and I don't think we will have problems with collision. I've had a look at your patch and there's a lot in there: an update to the spec, an implementation, an XOR demo, changes to Schema hashing, specific support, and static default classes. I think it would be helpful to get this in by breaking up the work into separate patches, pull requests, or issues. I also think we should simply the API a bit. I'd like to keep it small and grow it was we need to keep the maintenance and compatibility simple. For example, SchemaStorage has open and close methods that are only used in a test. I'd rather not add life-cycle methods like those unless the life-cycle of a SchemaStorage needs to be managed by Avro. To that end, I think we can simplify the API and I propose the following API: {code:lang=java} interface SchemaStore { Schema findByFingerprint(long); } {code} I also think that the message API should be focused around a datum and a buffer or stream. The data model (GenericData instance) and other things can be passed in to create it and then reused for efficiency. I've actually implemented this already for a project that stores Avro-encoded payloads in Parquet so I've [adapted that implementation|https://github.com/apache/avro/pull/103] to look up fingerprints from a SchemaStore. The API is broken into encoder and decoder sides to deal with separate concerns: for the encoder that's how to manage buffers and for the decoder it's how to resolve schemas and datum reuse. {code:lang=java} interface DatumEncoder<D> { DatumDecoder(GenericData model, Schema, boolean copyBuffer); ByteBuffer encode(D datum); // if copyBuffer was true, this is a new buffer void encode(D datum, OutputStream); } interface DatumDecoder<D> { DatumDecoder(GenericData model, Schema, SchemaStore); D decode(ByteBuffer); D decode(ByteBuffer, D reuseDatum); D decode(InputStream); D decode(InputStream, D reuseDatum); } {code} My branch is broken into a few commits. The first two are bug fixes, but the third is [the DatumEncoder implementation, d91b905|https://github.com/apache/avro/pull/103/commits/d91b90544f4486a72da8d3ff5b81dfc3c79d7c2f], and the fourth is [support for the Specific data model, 7fa75aa|https://github.com/apache/avro/pull/103/commits/7fa75aab405c6460077d7cc7e403c664cce84431], based on your patch. I'd like to hear what you think of the DatumEncoder API in that branch. It implements a few things that I think we'll need, like datum reuse, and it reuses encoders, DatumWriters, and buffers. It implements two encoder/decoder pairs, "raw" that is just the datum bytes and "binary" that implements the header and schema lookup. Definitely needs some improvements, like more through tests and better naming, like Doug's suggestion to use "message". > Standardized format for encoding messages with Avro > --------------------------------------------------- > > Key: AVRO-1704 > URL: https://issues.apache.org/jira/browse/AVRO-1704 > Project: Avro > Issue Type: Improvement > Reporter: Daniel Schierbeck > Assignee: Niels Basjes > Attachments: AVRO-1704-2016-05-03-Unfinished.patch, > AVRO-1704-20160410.patch > > > I'm currently using the Datafile format for encoding messages that are > written to Kafka and Cassandra. This seems rather wasteful: > 1. I only encode a single record at a time, so there's no need for sync > markers and other metadata related to multi-record files. > 2. The entire schema is inlined every time. > However, the Datafile format is the only one that has been standardized, > meaning that I can read and write data with minimal effort across the various > languages in use in my organization. If there was a standardized format for > encoding single values that was optimized for out-of-band schema transfer, I > would much rather use that. > I think the necessary pieces of the format would be: > 1. A format version number. > 2. A schema fingerprint type identifier, i.e. Rabin, MD5, SHA256, etc. > 3. The actual schema fingerprint (according to the type.) > 4. Optional metadata map. > 5. The encoded datum. > The language libraries would implement a MessageWriter that would encode > datums in this format, as well as a MessageReader that, given a SchemaStore, > would be able to decode datums. The reader would decode the fingerprint and > ask its SchemaStore to return the corresponding writer's schema. > The idea is that SchemaStore would be an abstract interface that allowed > library users to inject custom backends. A simple, file system based one > could be provided out of the box. -- This message was sent by Atlassian JIRA (v6.3.4#6332)