A number of our developers are seeing errors like the one below in their console when running a consumer on their laptop. The error is always followed by logging indicating that the local consumer is rebalancing, and in the meantime we are not making much progress.
I'm reading this as the consumer trying to read a ZK node for another consumer in the same group (running on a different machine,) but the node is no longer there. I can't tell if that is triggering a rebalance, or if it's just coincident. In our dev environment, we have a lot (hundreds) of consumers coming and going from the same consumer group, but they are mostly subscribed to different topics. Is this setup (sharing a consumer group across topics) potentially causing more rebalances than we would otherwise need? Or is something else entirely going on? LOG: INFO [2015-10-14 20:32:49,138] kafka.consumer.ZookeeperConsumerConnector: [real-time-updates_Noahs-MacBook-Pro.local-1444853969114-7b52ecb5], exception during rebalance ! org.I0Itec.zkclient.exception.ZkNoNodeException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /consumers/real-time-updates/ids/real-time-updates_infra-buildagent-06-1444854764478-4dd4d6af ! at org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47) ~[zkclient-0.3.jar:0.3] ! at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:685) ~[zkclient-0.3.jar:0.3] ! at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:766) ~[zkclient-0.3.jar:0.3] ! at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:761) ~[zkclient-0.3.jar:0.3] ! at kafka.utils.ZkUtils$.readData(ZkUtils.scala:443) ~[kafka_2.10-0.8.2.1.jar:na] ! at kafka.consumer.TopicCount$.constructTopicCount(TopicCount.scala:61) ~[kafka_2.10-0.8.2.1.jar:na] ! at kafka.utils.ZkUtils$$anonfun$getConsumersPerTopic$1.apply(ZkUtils.scala:665) ~[kafka_2.10-0.8.2.1.jar:na] ! at kafka.utils.ZkUtils$$anonfun$getConsumersPerTopic$1.apply(ZkUtils.scala:664) ~[kafka_2.10-0.8.2.1.jar:na] ! at scala.collection.Iterator$class.foreach(Iterator.scala:727) ~[scala-library-2.10.4.jar:na] ! at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) ~[scala-library-2.10.4.jar:na] ! at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) ~[scala-library-2.10.4.jar:na] ! at scala.collection.AbstractIterable.foreach(Iterable.scala:54) ~[scala-library-2.10.4.jar:na] ! at kafka.utils.ZkUtils$.getConsumersPerTopic(ZkUtils.scala:664) ~[kafka_2.10-0.8.2.1.jar:na] ! at kafka.consumer.AssignmentContext.<init>(PartitionAssignor.scala:52) ~[kafka_2.10-0.8.2.1.jar:na] ! at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance(ZookeeperConsumerConnector.scala:659) [kafka_2.10-0.8.2.1.jar:na] ! at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1$$anonfun$apply$mcV$sp$1.apply$mcVI$sp(ZookeeperConsumerConnector.scala:608) ~[kafka_2.10-0.8.2.1.jar:na] ! at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141) [scala-library-2.10.4.jar:na] ! at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply$mcV$sp(ZookeeperConsumerConnector.scala:602) [kafka_2.10-0.8.2.1.jar:na] ! at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply(ZookeeperConsumerConnector.scala:599) [kafka_2.10-0.8.2.1.jar:na] ! at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply(ZookeeperConsumerConnector.scala:599) [kafka_2.10-0.8.2.1.jar:na] ! at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33) [kafka_2.10-0.8.2.1.jar:na] ! at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:598) [kafka_2.10-0.8.2.1.jar:na] ! at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anon$1.run(ZookeeperConsumerConnector.scala:551) [kafka_2.10-0.8.2.1.jar:na] Caused by: ! org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /consumers/real-time-updates/ids/real-time-updates_infra-buildagent-06-1444854764478-4dd4d6af ! at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) ~[zookeeper-3.4.6.jar:3.4.6-1569965] ! at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.6.jar:3.4.6-1569965] ! at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) ~[zookeeper-3.4.6.jar:3.4.6-1569965] ! at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1184) ~[zookeeper-3.4.6.jar:3.4.6-1569965] ! at org.I0Itec.zkclient.ZkConnection.readData(ZkConnection.java:103) ~[zkclient-0.3.jar:0.3] ! at org.I0Itec.zkclient.ZkClient$9.call(ZkClient.java:770) ~[zkclient-0.3.jar:0.3] ! at org.I0Itec.zkclient.ZkClient$9.call(ZkClient.java:766) ~[zkclient-0.3.jar:0.3] ! at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675) ~[zkclient-0.3.jar:0.3] !... 21 common frames omitted INFO [2015-10-14 20:32:49,139] kafka.consumer.ZookeeperConsumerConnector: [real-time-updates_Noahs-MacBook-Pro.local-1444853969114-7b52ecb5], end rebalancing consumer real-time-updates_Noahs-MacBook-Pro.local-1444853969114-7b52ecb5 try #0
