Using kafka 0.8.1.1, the cluster had been healthy with producers and consumers being able to function well. After a restart of the cluster, it looks like consumers are locked out.
When I try to consume from a topic, I get this warning: [2015-01-23 07:48:50,667] WARN [console-consumer-32626_kafka-node-1.abc.net-1421999330230-a3d2b9e1], no brokers found when trying to rebalance. (kafka.consumer.ZookeeperConsumerConnector) I don't see any errors in server.log on the kafka nodes and there aren't any firewalls between the hosts (brokers, consumers and producers). I can query the topic state: $ /opt/cloudera/parcels/CLABS_KAFKA/bin/kafka-topics --zookeeper zookeeper-node-1:2181/kafka --describe --topic rawunstruct Topic:rawunstruct PartitionCount:5 ReplicationFactor:3 Configs: Topic: rawunstruct Partition: 0 Leader: 328 Replicas: 328,327,329 Isr: 328,327 Topic: rawunstruct Partition: 1 Leader: 328 Replicas: 329,328,327 Isr: 328,327 Topic: rawunstruct Partition: 2 Leader: 328 Replicas: 327,329,328 Isr: 328,327 Topic: rawunstruct Partition: 3 Leader: 328 Replicas: 328,329,327 Isr: 328,327 Topic: rawunstruct Partition: 4 Leader: 328 Replicas: 329,327,328 Isr: 328,327 However, when I list /kafka/brokers/ids on any of the ZK servers, I don't see any brokers ids. I did upgrade from CDH 5.2 to 5.3 but other components in the stack seem to be able to talk to Zookeeper just fine. Any pointers for troubleshooting? Thanks
