Re: Repeated failures due to ConsumerRebalanceFailedException

2015-03-08 Thread Ashwin Jayaprakash
Apologies for not replying sooner. (I have digest subscription and I cannot reply to the email chain I started! So I had not received any direct replies either). @Mayuresh and @Jiangjie, there was nothing very indicative in the Kafka logs. But I'm writing to tell you that the issue was "resolved"

Re: Repeated failures due to ConsumerRebalanceFailedException

2015-02-27 Thread Mayuresh Gharat
Do you see "zookeeper state changed (Expired)" in your logs? On Fri, Feb 27, 2015 at 10:12 AM, Jiangjie Qin wrote: > Can you paste the error log for each rebalance try? > You may search for keyword ³exception during rebalance². > > On 2/26/15, 7:41 PM, "Ashwin Jayaprakash" > wrote: > > >Just gi

Re: Repeated failures due to ConsumerRebalanceFailedException

2015-02-27 Thread Jiangjie Qin
Can you paste the error log for each rebalance try? You may search for keyword ³exception during rebalance². On 2/26/15, 7:41 PM, "Ashwin Jayaprakash" wrote: >Just give you some more debugging context, we noticed that the "consumers" >path becomes empty after all the JVMs have exited because of

Re: Repeated failures due to ConsumerRebalanceFailedException

2015-02-26 Thread Ashwin Jayaprakash
Just give you some more debugging context, we noticed that the "consumers" path becomes empty after all the JVMs have exited because of this error. So, when we restart, there are no visible entries in ZK. On Thu, Feb 26, 2015 at 6:04 PM, Ashwin Jayaprakash < ashwin.jayaprak...@gmail.com> wrote: >

Re: Repeated failures due to ConsumerRebalanceFailedException

2015-02-26 Thread Jiangjie Qin
Some times an ephemeral ZK path does not go away after a consumer is closed. You can check the log for each rebalance to see if it complains some conflict data of ZK Path. If all the complaints are pointing to the same consumer, bounce that consumer. Otherwise you can try to remove the ZK path manu

Repeated failures due to ConsumerRebalanceFailedException

2015-02-26 Thread Ashwin Jayaprakash
Hello, we have a set of JVMs that consume messages from Kafka topics. Each JVM creates 4 ConsumerConnectors that are used by 4 separate threads. These JVMs also create and use the CuratorFramework's Path children cache to watch and keep a sub-tree of the ZooKeeper in sync with other JVMs. This path