Hi, We are doing load testing. Ran into some exception - need some help clarifying what the problem is and how to bring the broker back on-line (hoping w/o the need to shutdown other brokers). Each broker was handling about 100k msgs/sec. Payload is small - less than 500 bytes About 2 hours into our run, one of the brokers stopped. It looked like an update to the zookeeper failed - conflict in /controller data’s version. It seems to happen when a new leader is chosen. How do we prevent and recover?
Sorry for the long post. Thanks for the help. Broker version kafka_2.10-0.8.1.1 jvm 1.7 We have 3 brokers running on 3 virtual machines. 1 topic, 6 partitions, 2 replicas 3 producing apps (6 producer threads each) 3 consuming apps (2 consumer each) The partitions and consumers were properly load-balanced across 3 brokers. [2014-05-30 09:42:21,122] ERROR Conditional update of path /brokers/topics/topic1/partitions/2/state with data {"controller_epoch":27,"leader":1,"version":1,"leader_epoch":0,"isr":[1]} and expected version 1960 failed due to org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode = BadVersion for /brokers/topics/topic1/partitions/2/state (kafka.utils.ZkUtils$) [2014-05-30 09:42:21,123] INFO Closing socket connection to /10.74.9.97. (kafka.network.Processor) [2014-05-30 09:42:21,147] INFO Partition [topic1,2] on broker 1: Cached zkVersion [1960] not equal to that in zookeeper, skip updating ISR (kafka.cluster.Partition) [2014-05-30 09:42:21,147] INFO Partition [topic1,5] on broker 1: Shrinking ISR for partition [topic1,5] from 1,3 to 1 (kafka.cluster.Partition) [2014-05-30 09:42:21,149] ERROR Conditional update of path /brokers/topics/topic1/partitions/5/state with data {"controller_epoch":27,"leader":1,"version":1,"leader_epoch":0,"isr":[1]} and expected version 2394 failed due to org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode = BadVersion for /brokers/topics/topic1/partitions/5/state (kafka.utils.ZkUtils$) [2014-05-30 09:42:21,149] INFO Partition [topic1,5] on broker 1: Cached zkVersion [2394] not equal to that in zookeeper, skip updating ISR (kafka.cluster.Partition) [2014-05-30 09:42:21,182] INFO conflict in /controller data: {"version":1,"brokerid":1,"timestamp":"1401442941179"} stored data: {"version":1,"brokerid":3,"timestamp":"1401442940062"} (kafka.utils.ZkUtils$) [2014-05-30 09:42:21,259] INFO re-registering broker info in ZK for broker 1 (kafka.server.KafkaHealthcheck) [2014-05-30 09:42:21,267] INFO Registered broker 1 at path /brokers/ids/1 with address host01:9092. (kafka.utils.ZkUtils$) [2014-05-30 09:42:21,267] INFO done re-registering broker (kafka.server.KafkaHealthcheck) [2014-05-30 09:42:21,299] INFO Subscribing to /brokers/topics path to watch for new topics (kafka.server.KafkaHealthcheck) We then saw bunch of producer errors - indicating the target partition is not local. [2014-05-30 09:42:21,482] WARN [KafkaApi-1] Produce request with correlation id 277356381 from client pooledKafkaMsgProducer on partition [topic1,2] failed due to Leader not local for partition [topic1,2] on broker 1 (kafka.server.KafkaApis) [2014-05-30 09:42:21,482] WARN [KafkaApi-1] Produce request with correlation id 272337409 from client pooledKafkaMsgProducer on partition [topic1,2] failed due to Leader not local for partition [topic1,2] on broker 1 (kafka.server.KafkaApis) We saw a new leader is elected. [2014-05-30 09:42:21,523] INFO [ReplicaFetcherManager on broker 1] Removed fetcher for partitions [topic1,5],[topic1,2] (kafka.server.ReplicaFetcherManager) [2014-05-30 09:42:21,523] INFO Truncating log topic1-5 to offset 2304238765. (kafka.log.Log) [2014-05-30 09:42:21,549] INFO New leader is 3 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener) [2014-05-30 09:42:21,652] INFO Truncating log topic1-2 to offset 2304238451. (kafka.log.Log) [2014-05-30 09:42:21,659] INFO [ReplicaFetcherManager on broker 1] Added fetcher for partitions ArrayBuffer([[topic1,2], initOffset 2304238451 to broker id:2,host:host02:9092] , [[topic1,5], initOffset 2304238765 to broker id:3,host03:9092] ) (kafka.server.ReplicaFetcherManager) [2014-05-30 09:42:21,688] WARN [KafkaApi-1] Fetch request with correlation id 114863 from client host02-2254-1-ConsumerFetcherThread-g1_host02-1401389340410-4d2e28ab-0-1 on partition [topic1,2] failed due to Leader not local for partition [topic1,2] on broker 1 (kafka.server.KafkaApis) [2014-05-30 09:42:21,690] WARN [KafkaApi-1] Fetch request with correlation id 114444 from client host03-5415-1-ConsumerFetcherThread-g1-host03-1401389345244-70fc9378-0-1 on partition [topic1,5] failed due to Leader not local for partition [topic1,5] on broker 1 (kafka.server.KafkaApis) [2014-05-30 09:42:21,789] WARN [KafkaApi-1] Fetch request with correlation id 114863 from client host02-2254-1-ConsumerFetcherThread-g1_host02-1401389340410-4d2e28ab-0-1 on partition [topic1,2] failed due to Leader not local for partition [topic1,2] on broker 1 (kafka.server.KafkaApis) [2014-05-30 09:42:21,790] WARN [KafkaApi-1] Fetch request with correlation id 114444 from client host03-5415-1-ConsumerFetcherThread-g1_host03-1401389345244-70fc9378-0-1 on partition [topic1,5] failed due to Leader not local for partition [topic1,5] on broker 1 (kafka.server.KafkaApis) [2014-05-30 09:42:21,793] INFO Closing socket connection to /10.74.41.157. (kafka.network.Processor) [2014-05-30 09:42:21,794] INFO Closing socket connection to /10.74.37.171. (kafka.network.Processor) [2014-05-30 09:42:21,794] INFO Closing socket connection to /10.74.41.157. (kafka.network.Processor) [2014-05-30 09:42:21,794] INFO Closing socket connection to /10.74.41.157. (kafka.network.Processor)