New Consumer APIs

Hi, Would like to if New Consumer API is GAed. Regards, Rajeswari

Re: [VOTE] 0.10.0.0 RC4

Thanks for finding and reporting, Liquan. I'll wait a day or two for more testing and roll out a new RC. In other news: We keep running into last minute issues with our shell scripts, because we have zero automated testing for them. Contribution of automated tests for our scripts will be super

Re: [VOTE] 0.10.0.0 RC4

We found a blocking issue on the release https://issues.apache.org/jira/browse/KAFKA-3692. This may cause the external CLASSPATH not be included in the final CLASSPATH in kafka-run-class.sh. There is no easy work around of this and we need a new RC. Thanks, Liquan On Mon, May 9, 2016 at 6:49 PM,

Re: ERROR Processor got uncaught exception. (kafka.network.Processor)

Thanks. I had a quick look at the code and it's not obvious how this could happen. Let's see how your testing goes. :) Ismael On Wed, May 11, 2016 at 2:04 AM, Ramanan, Buvana (Nokia - US) < buvana.rama...@nokia.com> wrote: > Ismael, > > We are setting up a 0.10.0 test cluster now. I will report

Kafka Partition Reassignment service (based on an adoption marketplace model)

Hello all, I have a design for a solution to the problem of "partition imbalances in Kafka clusters". It would be great to get some feedback on it. https://soumyajitsahu.wordpress.com/2016/05/11/kafka-partition-reassignment-service-using-an-adoption-marketplace-model/ I have also put a

RE: ERROR Processor got uncaught exception. (kafka.network.Processor)

Ismael, We are setting up a 0.10.0 test cluster now. I will report on whether this bug springs up in that cluster or not after a week or so. Glad to hear that SocketServer bugs are being taken care of in 0.10.0 and hope this issue is ironed out as a result. Regards, Buvana -Original

Re: Kafka 9 version offset storage mechanism changes

And where is the documentation for this topic: "__consumers_offsets" On Tue, May 10, 2016 at 1:16 AM, Spico Florin wrote: > Hi! > Yes both are possible. The new versions 0.9 and above store the offsets in > a special Kafka topic named __consumers_offsets. > Regards, >

Re: Zookeeper dies ... Kafka server unable to connect

Or retry with a volumeMount/persistentVolume for your single ZK pod. On Tue, May 10, 2016 at 9:01 AM, Paolo Patierno wrote: > Ok .. thanks. > I'll retry with a zookeeper cluster. > > Paolo. > > Paolo PatiernoSenior Software Engineer (IoT) @ Red Hat > Microsoft MVP on Windows

Re: New broker as follower for a topic replication

Hi Paolo, The best way to do this would be to have broker3 start up with the same broker id as the failed broker2. broker3 will then rejoin the cluster, begin catching up with broker1, and eventually rejoin the ISR. If it starts up with a new broker id, you'll need to run the partition

Re: Consumer stopped after reading some messages

Hi Sahitya, I wonder if your consumers are experiencing soft failures because they're busy processing a large collection of messages and not calling poll() within session.timeout.ms? In this scenario, the group coordinator (a broker) would not receive a heartbeat within session.timeout.ms and

Re: Backing up Kafka data and using it later?

You may find this interesting, although I don't believe it's exactly what you're looking for: https://github.com/pinterest/secor I'm not sure how stable and commonly used it is. Additionally, I see a lot of users use MirrorMaker for a "backup," where MirrorMaker copies all topics from one Kafka

Re: ERROR Processor got uncaught exception. (kafka.network.Processor)

Thanks Buvana. Is this happening in production only or can you also reproduce it in a test cluster? If the latter, would you be able to test the latest 0.10.0.0 release candidate? We fixed a few issues in the SocketServer. Ismael On Tue, May 10, 2016 at 8:27 PM, Ramanan, Buvana (Nokia - US) <

RE: ERROR Processor got uncaught exception. (kafka.network.Processor)

Ismael, Created bug: https://issues.apache.org/jira/browse/KAFKA-3689 Hope to get a quick resolution. Thanks, Buvana -Original Message- From: isma...@gmail.com [mailto:isma...@gmail.com] On Behalf Of EXT Ismael Juma Sent: Tuesday, May 10, 2016 10:54 AM To: users@kafka.apache.org

Consumer group rebalance taking too long

Hi team, I want to know what would happen if the consumer group rebalance takes long time like longer than the session timeout? For example I have two consumers A and B using the same group id. For some reasons during rebalance consumer A takes long time to finish onPartitionsRevoked what would

RE: Zookeeper dies ... Kafka server unable to connect

Ok .. thanks. I'll retry with a zookeeper cluster. Paolo. Paolo PatiernoSenior Software Engineer (IoT) @ Red Hat Microsoft MVP on Windows Embedded & IoTMicrosoft Azure Advisor Twitter : @ppatierno Linkedin : paolopatierno Blog : DevExperience > Date: Tue, 10 May 2016 17:59:16 +0200 > From:

RE: Zookeeper dies ... Kafka server unable to connect

Yes correct ... the new restarted zookeeper instance is completely new ... it has no information about previous topics and brokers of course. Paolo PatiernoSenior Software Engineer (IoT) @ Red Hat Microsoft MVP on Windows Embedded & IoTMicrosoft Azure Advisor Twitter : @ppatierno Linkedin :

RE: Zookeeper dies ... Kafka server unable to connect

This is what Kubernetes says me ... Name:zookeeper Namespace:default Labels: Selector:name=zookeeper Type:ClusterIP IP:10.0.0.184 Port:zookeeper2181/TCP Endpoints:172.17.0.4:2181 Session Affinity:None So

Re: Zookeeper dies ... Kafka server unable to connect

Are you sure you’re getting the same IP address? Regarding zookeeper connection being closed, is kubernetes doing a soft shutdown of your container? If so, zookeeper is asked politely to stop. – Best regards,
 Radek Gruchalski 
radek@gruchalski.commailto:ra...@gruchalski.com

Zookeeper dies ... Kafka server unable to connect

Hi all, experiencing with Kafka on Kubernetes I have the following error on Kafka server reconnection ... A cluster with one zookeeper and two kafka server ... I turn off the zookeeper pod but kubernetes restart it and guaratees the same IP address for it but the kafka server starts to retry

Re: ERROR Processor got uncaught exception. (kafka.network.Processor)

OK, thanks. I suggest filing a bug in JIRA and please provide as much information as possible (steps to reproduce would be ideal, but sometimes that is hard to do). It does look like a Kafka bug. Ismael On Tue, May 10, 2016 at 2:45 PM, Ramanan, Buvana (Nokia - US) < buvana.rama...@nokia.com>

RE: ERROR Processor got uncaught exception. (kafka.network.Processor)

Ismael, Version 0.9.0.1 Do you have any idea how to prevent this from happening? Is it a Kafka issue? -Buvana -Original Message- From: isma...@gmail.com [mailto:isma...@gmail.com] On Behalf Of EXT Ismael Juma Sent: Monday, May 09, 2016 8:02 PM To: users@kafka.apache.org Subject: Re:

Re: Topic description by zookeeper after a kafka server dies

Hi Paolo, We just jump on the box (don't use kuberenetes) and change the metadata.properties file manually, then restart. We are only doing a small amount of non prod traffic so it's easy for us to manage. Thanks, Ben On Tuesday, 10 May 2016, Paolo Patierno wrote: > Hi

RE: Topic description by zookeeper after a kafka server dies

Hi Ben, in order to avoid conflicts I have auto generation for broker id in the server.properties file. Of course it's a workaround but I'm so curious how do you tell Kubernetes to start a new Kafka pod with a previous broker id ? Btw, waiting for an answer from the Kafka team. Paolo. Paolo

Re: Topic description by zookeeper after a kafka server dies

Hi Paolo, Would be interested to hear the answer also. We've been getting around this ourselves by setting the new broker that comes up back to id 1001. Thanks, Ben On Tuesday, 10 May 2016, Paolo Patierno wrote: > Hello, > > I'm experiencing the usage of Kafka with

Re: Kafka 9 version offset storage mechanism changes

Hi folks, Zookeeper is not required for storing offsets with the new consumer, but it still has a variety of other uses for Kafka and will always be required for coordination purposes. Please see Gwen's answer here as it's a pretty nice write up:

Re: Kafka rebalance at the group name level?

I haven't browsed the source for the rebalance algorithm but anecdotally It appears this is the case. In our system we have a consumer group whose application instances are not only scaled but also split by topics (some topics have much higher message rates). When we perform a deployment of one

ZK path name mentioned in the online Replication Design document seems to be incorrect

I am trying to understand the replication procedure and saw this document https://cwiki.apache.org/confluence/display/KAFKA/kafka+Detailed+Replication+Design+V3 describing: LeaderAndISR path: stores leader and ISR of a partition /brokers/topics/[topic]/[partition_id]/leaderAndISR -->

Re: Kafka 9 version offset storage mechanism changes

But if we set autocommit to false and fetch data using simple consumer, will it still use zookeeper for any purpose? - Original Message - From: "Spico Florin" To: users@kafka.apache.org Sent: Tuesday, May 10, 2016 1:55:39 PM Subject: Re: Kafka 9 version offset

Kafka rebalance at the group name level?

I have 3 topics A,B,C with same number of partitions. I use the same group name for all the consumers to this topics. My questions are: 1. If a consumer for one of the topics/partitions will rebalance be triggered for the other two topics consumers? 2. Same if adding a new partition for one

Re: Tool to look at what's stored in internal topics on kafka 0.9

If you want to consume __consume_offset topic. we have to use this configuration also exclude.internal.topics=false in consumer properties Regards, Ashiq On Tue, May 10, 2016 at 9:50 AM, Henry Cai wrote: > Which deserializer class to use? > > On Mon, May 9, 2016

Re: Kafka 9 version offset storage mechanism changes

Hi! Is just a guess (perhaps someone will correct me if I'm wrong). It depends on the API you are using for consumers: - simple API uses ZK for storring the offsets - high level API stores the offests in Kafka Broker __consumer_offests topic. I hope it help. Florin On Tue, May 10, 2016 at

Re: Kafka 9 version offset storage mechanism changes

so zookeeper not needed anymore? > On May 10, 2016, at 1:46 PM, Spico Florin wrote: > > Hi! > Yes both are possible. The new versions 0.9 and above store the offsets in > a special Kafka topic named __consumers_offsets. > Regards, > florin > > On Tue, May 10, 2016 at

Re: Kafka 9 version offset storage mechanism changes

Hi! Yes both are possible. The new versions 0.9 and above store the offsets in a special Kafka topic named __consumers_offsets. Regards, florin On Tue, May 10, 2016 at 8:33 AM, Gerard Klijs wrote: > Both are possible, but the 'new' consumer stores the offset in an