Re: Kafka 0.8's ConsumerConnector.close() hangs if ZK is unavailable and autocommit is enabled

2014-05-13 Thread Jun Rao
In 0.8, ZK is critical for managing the brokers. So, we do expect that ZK service to be always available. Thanks, Jun On Mon, May 12, 2014 at 9:59 AM, Yury Ruchin yuri.ruc...@gmail.com wrote: Hi all, I'm using Kafka 0.8 and I've ran into an issue with ConsumerConnector. Steps to

Re: Kafka 0.8's ConsumerConnector.close() hangs if ZK is unavailable and autocommit is enabled

2014-05-13 Thread Guozhang Wang
Hi Yury, When auto-commit is turned on, before the consumer shuts down completely it needs to commit its last offset to ZK to avoid any duplicates consumed when it starts over. If the ZK is not up running then the shutdown would rather stuck than let it go and incur duplicates. This behavior is

Kafka 0.8's ConsumerConnector.close() hangs if ZK is unavailable and autocommit is enabled

2014-05-12 Thread Yury Ruchin
Hi all, I'm using Kafka 0.8 and I've ran into an issue with ConsumerConnector. Steps to reproduce: 1. Start single-broker Kafka cluster with auto.create.topic.enable set to true 2. Start ConsumerConnector on topic (which does not yet exist) with auto.offset.reset set to smallest. 3. Produce some