[ https://issues.apache.org/jira/browse/AVRO-1704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Niels Basjes updated AVRO-1704: ------------------------------- Attachment: AVRO-1704-2016-05-03-Unfinished.patch This patch does just about everything we talked about. Both the schema storage and the serialization of the body are pluggable. I created a single record serializer that does an 'xor' obfuscation on the binary. I see this as enough proof that someone else can later create a proper encryption layer. Main things that still need to be done: # What do we call this? "Single record serializer" ? # The currently generated methods are toBytes and fromBytes. Do we keep those names or should it be more explicit? Like toSingleRecordBytes or toBytesWithSchema or ... # Check the format of the fingerprint in the byte[] (should be little endian) and see if there is an existing method that does this in a performant way (suggestions are welcome). # Naming of packages and classes. I find some of the current names I came up with "sub-optimal". Please review and provide input to the points above. Thanks. > Standardized format for encoding messages with Avro > --------------------------------------------------- > > Key: AVRO-1704 > URL: https://issues.apache.org/jira/browse/AVRO-1704 > Project: Avro > Issue Type: Improvement > Reporter: Daniel Schierbeck > Assignee: Niels Basjes > Attachments: AVRO-1704-2016-05-03-Unfinished.patch, > AVRO-1704-20160410.patch > > > I'm currently using the Datafile format for encoding messages that are > written to Kafka and Cassandra. This seems rather wasteful: > 1. I only encode a single record at a time, so there's no need for sync > markers and other metadata related to multi-record files. > 2. The entire schema is inlined every time. > However, the Datafile format is the only one that has been standardized, > meaning that I can read and write data with minimal effort across the various > languages in use in my organization. If there was a standardized format for > encoding single values that was optimized for out-of-band schema transfer, I > would much rather use that. > I think the necessary pieces of the format would be: > 1. A format version number. > 2. A schema fingerprint type identifier, i.e. Rabin, MD5, SHA256, etc. > 3. The actual schema fingerprint (according to the type.) > 4. Optional metadata map. > 5. The encoded datum. > The language libraries would implement a MessageWriter that would encode > datums in this format, as well as a MessageReader that, given a SchemaStore, > would be able to decode datums. The reader would decode the fingerprint and > ask its SchemaStore to return the corresponding writer's schema. > The idea is that SchemaStore would be an abstract interface that allowed > library users to inject custom backends. A simple, file system based one > could be provided out of the box. -- This message was sent by Atlassian JIRA (v6.3.4#6332)