Versioning Schema's

Hello everyone, After doing some searching on the mailing list for best practices on integrating Avro with Kafka there appears to be at least 3 options for integrating the Avro Schema; 1) embedding the entire schema within the message 2) embedding a unique identifier for the schema in the message

kafka 0.8.0 beta release

Hi, Everyone, Joe Stein has kindly agreed to drive this release. We'd like to get KAFKA-937 committed to 0.8 (should happen today) and then call a vote. Thanks, Jun

Re: Producer will pick one of the two brokers, but never the two at same time [0.8]

If the leaders exist in both brokers, the producer should be able to connect to both of them, assuming you don't provide any key when sending the data. Could you try restarting the producer? If there has been broker failures, it may take topic.metadata.refresh.interval.ms for the producer to pick

Re: Versioning Schema's

At LinkedIn, we are using option 2. Thanks, Jun On Wed, Jun 12, 2013 at 7:14 AM, Shone Sadler shone.sad...@gmail.comwrote: Hello everyone, After doing some searching on the mailing list for best practices on integrating Avro with Kafka there appears to be at least 3 options for

Re: Versioning Schema's

Actually, currently our schema id is the md5 of the schema itself. Not fully sure how this compares with an explicit version field in the schema. Thanks, Jun On Wed, Jun 12, 2013 at 8:29 AM, Jun Rao jun...@gmail.com wrote: At LinkedIn, we are using option 2. Thanks, Jun On Wed, Jun

Re: Producer will pick one of the two brokers, but never the two at same time [0.8]

Hi Jun, Thanks for your prompt answer. The producer yields those errors in the beginning, so I think the topic metadata refresh has nothing to do with it. The problem is one of the brokers isn't leader on any partition assigned to it and because topics were created with a replication factor of

Re: Versioning Schema's

Jun, I like the idea of an explicit version field, if the schema can be derived from the topic name itself. The storage (say 1-4 bytes) would require less overhead than a 128 bit md5 at the added cost of managing the version#. Is it correct to assume that your applications are using two schemas

Re: Versioning Schema's

For one of our key Kafka-based applications, we ensure that all messages in the stream have a common binary format, which includes (among other things) a version identifier and a schema identifier. The version refers to the format itself, and the schema refers to the payload, which s the data

Re: Kafka 0.8 Maven and IntelliJ

Thanks Dragos, I've been using that plugin before, that will work on a developer's machine when you try to build and debug the project but I also need this to work with my automated build system. That's why I need maven to work. I've made a bit more progress: cd kafka ./sbt make-pom cd core

One 0.72 ConsumerConnector, multiple threads, 1 blocks. What happens?

Hello -- we're using 0.72. We're looking at the source, but want to be sure. :-) We create a single ConsumerConnector, call createMessageStreams, and hand the streams off to individual threads. If one of those threads calls next() on a stream, gets some messages, and then *blocks* in some

Re: How to compile java files in examples directory bundled in kafka 0.8?

I tried running java examples using READ Me but when I say ./sbt getting following error. Java HotSpot(TM) 64-Bit Server VM warning: Insufficient space for shared memory file: /tmp/hsperfdata_root/1312 Try using the -Djava.io.tmpdir= option to select an alternate temp location. Could you

Re: Producer will pick one of the two brokers, but never the two at same time [0.8]

Any error in state-change.log? Also, are you using the latest code in the 0.8 branch? Thanks, Jun On Wed, Jun 12, 2013 at 9:27 AM, Alexandre Rodrigues alexan...@blismedia.com wrote: Hi Jun, Thanks for your prompt answer. The producer yields those errors in the beginning, so I think the

Re: Versioning Schema's

Yes, we just have customized encoder that encodes the first 4 bytes of md5 of the schema, followed by Avro bytes. Thanks, Jun On Wed, Jun 12, 2013 at 9:50 AM, Shone Sadler shone.sad...@gmail.comwrote: Jun, I like the idea of an explicit version field, if the schema can be derived from the

Re: One 0.72 ConsumerConnector, multiple threads, 1 blocks. What happens?

Yes, when the consumer is consuming multiple topics, if one thread stops consuming topic 1, it can prevent new data getting into the consumer for topic 2. Thanks, Jun On Wed, Jun 12, 2013 at 7:43 PM, Philip O'Toole phi...@loggly.com wrote: Hello -- we're using 0.72. We're looking at the

Re: One 0.72 ConsumerConnector, multiple threads, 1 blocks. What happens?

Jun -- thanks. But if the topic is the same, doesn't each thread get a partition? Isn't that how it works? Philip On Wed, Jun 12, 2013 at 9:08 PM, Jun Rao jun...@gmail.com wrote: Yes, when the consumer is consuming multiple topics, if one thread stops consuming topic 1, it can prevent new

Re: One 0.72 ConsumerConnector, multiple threads, 1 blocks. What happens?

Also, what is it in the ConsumerConnection that causes this behaviour? I'm proposing here that we move to a model of ConsumerConnection per thread. This will decouple the flow for each partition, and allow each to flow, right? We only have one topic on the cluster. Thanks, Philip On Wed, Jun