Re: Re: Re: ZK and Kafka failover testing

2017-04-20 Thread Jun Rao
Hi, Shri, So, it seems that the duplicates are introduced by the publisher. Currently, producer retries can introduce duplicates. We are trying to eliminate that in the EOS work (KIP-98). We plan to have a beta version of that available in 0.11.0. Hi, Jeff, We have a known issue of recovering fr

Re: Re: Re: ZK and Kafka failover testing

2017-04-19 Thread Hans Jespersen
The kafka-console-producer.sh defaults to acks=1 so just be careful with using those tools for too much debugging. Your output is helpful though. https://github.com/apache/kafka/blob/5a2fcdd6d480e9f003cc49a59d5952ba4c515a71/core/src/main/scala/kafka/tools/ConsoleProducer.scala#L185 -hans On Wed,

RE: Re: Re: ZK and Kafka failover testing

2017-04-19 Thread Shrikant Patel
Just to add, I see below behavior repeat with even command line console producer and consumer that come with Kafka. Thanks, Shri __ Shrikant Patel | 817.367.4302 Enterprise Architecture Team PDX-NHIN -Original Message- From: Shrikant Pat

RE: Re: Re: ZK and Kafka failover testing

2017-04-19 Thread Shrikant Patel
Thanks Jeff, Onur, Jun, Hans. I am learning a lot from your response. Just to summarize briefly my steps, 5 node Kafka and ZK cluster. 1. ZK cluster has all node working. Consumer is down. 2. Bring down majority of ZK nodes. 3. Thing are functional no issue (no dup or lost message) 4. Now first ka