Thank you Joel. In a different but related thread, somebody is asking to rename the exception as NoBrokerAvailableExcption. But given the description above, the exception seems to be named appropriately.
Regards, Vaibhav On Tue, Jul 16, 2013 at 12:05 AM, Joel Koshy <jjkosh...@gmail.com> wrote: > Yes - rebalance => consumers trying to coordinate through ZK. > Rebalances can happen when one or more of the following happen: > - a consumed topic partition appears or disappears - i.e., if a broker > comes or goes. > - a consumer instance in the group comes or goes > "goes" could also be triggered by session expirations in zookeeper - > typically caused by client-side GC or flaky connections to zookeeper. > > On Mon, Jul 15, 2013 at 10:15 AM, Vaibhav Puranik <vpura...@gmail.com> > wrote: > > Hi all, > > > > We have a small Kafka cluster (0.7.1 - 3 nodes) in EC2. The load is about > > 200 million events per day, each being few kilobytes. We have a single > node > > zookeeper. > > > > Yesterday suddenly our Kafka clients started throwing the following > > exception: > > java.lang.RuntimeException: > kafka.common.ConsumerRebalanceFailedException: > > CONSUMER_GROUP_NAME_ip-00-00-00-00.ec2.internal-1373821190828-5f78e9af > > can't rebalance after 4 retries > > at > > > com.gumgum.kafka.consumer.KafkaTemplate.executeWithBatch(KafkaTemplate.java:59) > > at > > > com.gumgum.storm.fileupload.GenericKafkaSpout.nextTuple(GenericKafkaSpout.java:73) > > at > > > backtype.storm.daemon.executor$fn__3968$fn__4009$fn__4010.invoke(executor.clj:433) > > at backtype.storm.util$async_loop$fn__465.invoke(util.clj:377) > > > > None of the Kafka clients (ConsumerConenctor class) would start. They > would > > fail with the exception. > > > > We tried restarting the clilents, restarting the zookeeper as well. But > > finally it all started working when we restarted all of our kafka > brokers. > > We didn't lose any data because producers (going directly to the brokers > > through a load balancer) were working fine. > > > > I tried googling this issue and looks like lot of people have faced it, > but > > couldn't get anything concrete. > > > > Given this, I have two questions: > > > > It will be nice if you can tell me why this can happen or point me to a > > link where I can understand it better. What does Consumer Rebalancing > mean? > > Does that mean consumers are trying to coordinate amongst themselves > using > > Zookeeper? > > > > On a separate note, are there any JMX parameters I need to be monitoring > to > > make sure that my kafka cluster is healthy? How can I keep watch on my > > kafka cluster? > > > > Regards, > > Vaibhav Puranik > > GumGum >
