Re: Kafka compatibility matrix needed

2017-07-19 Thread Sachin Mittal
Hi, Are you referring to this topic level config? message.format.version Specify the message format version the broker will use to append messages to the logs. The value should be a valid ApiVersion. Some examples are: 0.8.2, 0.9.0.0, 0.10.0, check ApiVersion for more details. By setting a partic

Re: Kafka compatibility matrix needed

2017-07-19 Thread Ismael Juma
Also, make On Wed, Jul 19, 2017 at 5:10 AM, Ismael Juma wrote: > Hi Anish, > > What should happen in that case is that the broker down converts the > messages to a format supported by the older clients. We have tests for this > scenario so it would be good to understand what is different in your

Re: Kafka compatibility matrix needed

2017-07-19 Thread Ismael Juma
Hi Sachin, Yes, that should work as the broker will down convert the messages for the older consumer. The one thing to avoid is decreasing the message format version for an existing topic. That is not supported. As usual, you should test this in a non production environment first. Also, one addit

Re: Kafka compatibility matrix needed

2017-07-19 Thread Ismael Juma
Hi Anish, What should happen in that case is that the broker down converts the messages to a format supported by the older clients. We have tests for this scenario so it would be good to understand what is different in your environment. Please file a JIRA and provide as much information as possibl

Re: Kafka compatibility matrix needed

2017-07-19 Thread Sachin Mittal
Hi, So can anyone confirm if messages to topic produced by java client of 0.10.2 can be consumed by java client 0.8.2. Perhaps brokers support older versions of client, but would need some information on this case as it is not very clear. In our case we have control over the producer but not consu

Re: Kafka compatibility matrix needed

2017-07-19 Thread Anish Mashankar
After upgrading Kafka 0.10.0 to 0.11.0, and changing the Message protocol to 0.11 on brokers, the consumers with version 0.8.2.1 started reporting Invalid message error logs. On Tue, Jul 18, 2017 at 6:37 PM Ismael Juma wrote: > Hi all, > > 0.8.x clients should work with 0.9.x, 0.10.x and 0.11.x

Re: Kafka compatibility matrix needed

2017-07-18 Thread Ismael Juma
Hi all, 0.8.x clients should work with 0.9.x, 0.10.x and 0.11.x brokers. We have system tests for all the relevant combinations. One thing to be careful about is that Scala consumers and Java consumers store offsets and group management information differently and the Java consumer was only introd

Re: Kafka compatibility matrix needed

2017-07-18 Thread John Yost
Hi Everyone, I personally found that the 0.8.x clients do not work with 0.10.0. We upgraded our clients (KafkaSpout and custom consumers) to 0.9.0.1 and then Kafka produce/consume worked fine. --John On Tue, Jul 18, 2017 at 6:36 AM, Sachin Mittal wrote: > OK. > > Just a doubt I have is that my

Re: Kafka compatibility matrix needed

2017-07-18 Thread Sachin Mittal
OK. Just a doubt I have is that my broker is 0.10.2 and producer is also of the same version and writes to a topic. Then I have a client from version 0.8.2 trying to fetch these messages and what I see is that all messages are getting dropped. However since this is an older client I tried to fetc

Re: Kafka compatibility matrix needed

2017-07-18 Thread Tom Crayford
All broker versions support all older client versions On Tue, Jul 18, 2017 at 10:15 AM, Sachin Mittal wrote: > Hi, > This gives me some information but still not the complete picture. > > It says: > 0.10.2, Java clients have acquired the ability to communicate with older > brokers. > > It also s

Re: Kafka compatibility matrix needed

2017-07-18 Thread Sachin Mittal
Hi, This gives me some information but still not the complete picture. It says: 0.10.2, Java clients have acquired the ability to communicate with older brokers. It also says Version 0.11.0 brokers support 0.8.x and newer clients Question is does 0.10.2 broker support 0.8.x clients? This may so

Re: Kafka compatibility matrix needed

2017-07-18 Thread Michal Borowiecki
Have you seen this: http://kafka.apache.org/documentation.html#upgrade Starting with version 0.10.2, Java clients (producer and consumer) have acquired the ability to communicate with older brokers. Version 0.11.0 clients can talk to version 0.10.0 or newer brokers. However, if your brokers ar

Kafka compatibility matrix needed

2017-07-18 Thread Sachin Mittal
Hi, I would like some help/information on what client versions are compatible with what broker versions in kafka. Some table like this would be good server client 0.80.9 0.10 0.11 0.8 yes ? ?? 0.9 ?yes ?? 0.10