I've done this in the past, and it worked out well. Stored Avro schema
in ZooKeeper with an integer id and prefixed each message with the id.
You have to make sure when you register a new schema that it resolves
with the current version (ResolvingDecoder helps with this).
-David
On 6/13/13 4:
Thanks Jun & Phil!
Shone
On Thu, Jun 13, 2013 at 12:00 AM, Jun Rao wrote:
> Yes, we just have customized encoder that encodes the first 4 bytes of md5
> of the schema, followed by Avro bytes.
>
> Thanks,
>
> Jun
>
>
> On Wed, Jun 12, 2013 at 9:50 AM, Shone Sadler >wrote:
>
> > Jun,
> > I like
Yes, we just have customized encoder that encodes the first 4 bytes of md5
of the schema, followed by Avro bytes.
Thanks,
Jun
On Wed, Jun 12, 2013 at 9:50 AM, Shone Sadler wrote:
> Jun,
> I like the idea of an explicit version field, if the schema can be derived
> from the topic name itself. T
For one of our key Kafka-based applications, we ensure that all messages in the
stream have a common binary format, which includes (among other things) a
version identifier and a schema identifier. The version refers to the format
itself, and the schema refers to the "payload," which s the data
Jun,
I like the idea of an explicit version field, if the schema can be derived
from the topic name itself. The storage (say 1-4 bytes) would require less
overhead than a 128 bit md5 at the added cost of managing the version#.
Is it correct to assume that your applications are using two schemas th
Actually, currently our schema id is the md5 of the schema itself. Not
fully sure how this compares with an explicit version field in the schema.
Thanks,
Jun
On Wed, Jun 12, 2013 at 8:29 AM, Jun Rao wrote:
> At LinkedIn, we are using option 2.
>
> Thanks,
>
> Jun
>
>
> On Wed, Jun 12, 2013 at
At LinkedIn, we are using option 2.
Thanks,
Jun
On Wed, Jun 12, 2013 at 7:14 AM, Shone Sadler wrote:
> Hello everyone,
>
> After doing some searching on the mailing list for best practices on
> integrating Avro with Kafka there appears to be at least 3 options for
> integrating the Avro Schema
Hello everyone,
After doing some searching on the mailing list for best practices on
integrating Avro with Kafka there appears to be at least 3 options for
integrating the Avro Schema; 1) embedding the entire schema within the
message 2) embedding a unique identifier for the schema in the message