Hello, We've an issue on our Kafka production. (Kafka 0.8.1.1 on 9 Debian Wheezy servers). We tried to move a topic from a server to another, but no data was created on the target.
After some tries when decidec to stop Kafka on this server (kafka3), removed all topic's data on /var/lib/kafka/* and start again kafka daemon. Now no data are written at all on kafka3 server. On kafka1 controller.log we had these logs when whe restarted kafka3's kafka daemon: [2015-02-16 11:11:05,822] INFO [BrokerChangeListener on Controller 1]: Broker change listener fired for path /brokers/ids with children 3,2,1,6,5,4,9,8 (kafka.controller.ReplicaStateMachine$BrokerChangeListener) [2015-02-16 11:11:05,836] INFO [BrokerChangeListener on Controller 1]: Newly added brokers: 3, deleted brokers: , all live brokers: 5,1,6,9,2,3,8,4 (kafka.controller.ReplicaStateMachine$BrokerChangeListener) [2015-02-16 11:11:05,836] DEBUG [Channel manager on controller 1]: Controller 1 trying to connect to broker 3 (kafka.controller.ControllerChannelManager) [2015-02-16 11:11:05,836] INFO [Controller-1-to-broker-3-send-thread], Controller 1 connected to id:3,host:172.16.23.3,port:9092 for sending state change requests (kafka.controller.RequestSendThread) [2015-02-16 11:11:05,836] INFO [Controller 1]: New broker startup callback for 3 (kafka.controller.KafkaController) [2015-02-16 11:11:05,836] INFO [Controller-1-to-broker-3-send-thread], Starting (kafka.controller.RequestSendThread) [2015-02-16 11:11:05,837] ERROR [BrokerChangeListener on Controller 1]: Error while handling broker changes (kafka.controller.ReplicaStateMachine$BrokerChangeListener) java.lang.IllegalStateException: Controller to broker state change requests batch is not empty while creating a new one. Some UpdateMetadata state changes Map(3 -> Map([AdBidderZoneSecondKey,0] -> (LeaderAndIsrInfo:(Leader:2,ISR:2,LeaderEpoch:6,ControllerEpoch:5),ReplicationFactor:2),AllReplicas:2,3), [AdBidderZoneSourceAdApprovalDailyKey,0] -> (LeaderAndIsrInfo:(Leader:1,ISR:1,2,LeaderEpoch:2,ControllerEpoch:3),ReplicationFactor:2),AllReplicas:2,1), [AdBidderZoneDealDailyKey,0] -> (LeaderAndIsrInfo:(Leader:4,ISR:4,5,LeaderEpoch:45,ControllerEpoch:5),ReplicationFactor:2),AllReplicas:4,5), [Views,2] -> (LeaderAndIsrInfo:(Leader:1,ISR:1,4,LeaderEpoch:5,ControllerEpoch:4),ReplicationFactor:2),AllReplicas:1,4), [AdBidderZoneTrackIdsDailyKey,0] -> (LeaderAndIsrInfo:(Leader:2,ISR:2,LeaderEpoch:7,ControllerEpoch:5),ReplicationFactor:2),AllReplicas:2,3))) might be lost at kafka.controller.ControllerBrokerRequestBatch.newBatch(ControllerChannelManager.scala:202) at kafka.controller.KafkaController.sendUpdateMetadataRequest(KafkaController.scala:939) at kafka.controller.KafkaController.onBrokerStartup(KafkaController.scala:386) at kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ReplicaStateMachine.scala:342) at kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply(ReplicaStateMachine.scala:330) at kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply(ReplicaStateMachine.scala:330) at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33) at kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply$mcV$sp(ReplicaStateMachine.scala:329) at kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply(ReplicaStateMachine.scala:328) at kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply(ReplicaStateMachine.scala:328) at kafka.utils.Utils$.inLock(Utils.scala:538) at kafka.controller.ReplicaStateMachine$BrokerChangeListener.handleChildChange(ReplicaStateMachine.scala:327) at org.I0Itec.zkclient.ZkClient$7.run(ZkClient.java:568) at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) Any help would be most welcome. -- Anthony Pastor. <http://www.stickyads.tv/> [image: http://www.stickyads.tv/wp-content/uploads/2013/03/StickyADS.tv-Black-Logo-PNG-300x46.png] <http://www.stickyads.tv/> PARIS | *MONTPELLIER* | MADRID | MILAN | HAMBURG | LONDON