You are not supposed to run an even number of zookeepers. Fix that first On Apr 26, 2017 20:59, "Abhit Kalsotra" <abhit...@gmail.com> wrote:
> Any pointers please.... > > > Abhi > > On Wed, Apr 26, 2017 at 11:03 PM, Abhit Kalsotra <abhit...@gmail.com> > wrote: > > > Hi * > > > > My kafka setup > > > > > > **OS: Windows Machine*6 broker nodes , 4 on one Machine and 2 on other > > Machine* > > > > **ZK instance on (4 broker nodes Machine) and another ZK on (2 broker > > nodes machine)* > > ** 2 Topics with partition size = 50 and replication factor = 3* > > > > I am producing on an average of around 500 messages / sec with each > > message size close to 98 bytes... > > > > More or less the message rate stays constant throughout, but after > running > > the setup for close to 2 weeks , my Kafka cluster broke and this happened > > twice in a month. Not able to understand what's the issue, Kafka gurus > > please do share your inputs... > > > > the controlle.log file at the time of Kafka broken looks like > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *[2017-04-26 12:03:34,998] INFO [Controller 0]: Broker failure callback > > for 0,1,3,5,6 (kafka.controller.KafkaController)[2017-04-26 > 12:03:34,998] > > INFO [Controller 0]: Removed ArrayBuffer() from list of shutting down > > brokers. (kafka.controller.KafkaController)[2017-04-26 12:03:34,998] > INFO > > [Partition state machine on Controller 0]: Invoking state change to > > OfflinePartition for partitions > > [__consumer_offsets,19],[mytopic,11],[__consumer_ > offsets,30],[mytopicOLD,18],[mytopic,13],[__consumer_ > offsets,47],[mytopicOLD,26],[__consumer_offsets,29],[ > mytopicOLD,0],[__consumer_offsets,41],[mytopic,44],[ > mytopicOLD,38],[mytopicOLD,2],[__consumer_offsets,17],[__ > consumer_offsets,10],[mytopic,20],[mytopic,23],[mytopic,30], > [__consumer_offsets,14],[__consumer_offsets,40],[mytopic, > 31],[mytopicOLD,43],[mytopicOLD,19],[mytopicOLD,35] > ,[__consumer_offsets,18],[mytopic,43],[__consumer_offsets,26],[__consumer_ > offsets,0],[mytopic,32],[__consumer_offsets,24],[ > mytopicOLD,3],[mytopic,2],[mytopic,3],[mytopicOLD,45],[ > mytopic,35],[__consumer_offsets,20],[mytopic,1],[ > mytopicOLD,33],[__consumer_offsets,5],[mytopicOLD,47],[__ > consumer_offsets,22],[mytopicOLD,8],[mytopic,33],[ > mytopic,36],[mytopicOLD,11],[mytopic,47],[mytopicOLD,20],[ > mytopic,48],[__consumer_offsets,12],[mytopicOLD,32],[_ > _consumer_offsets,8],[mytopicOLD,39],[mytopicOLD,27] > ,[mytopicOLD,49],[mytopicOLD,42],[mytopic,21],[mytopicOLD, > 31],[mytopic,29],[__consumer_offsets,23],[mytopicOLD,21],[_ > _consumer_offsets,48],[__consumer_offsets,11],[mytopic, > 18],[__consumer_offsets,13],[mytopic,45],[mytopic,5],[ > mytopicOLD,25],[mytopic,6],[mytopicOLD,23],[mytopicOLD,37] > ,[__consumer_offsets,6],[__consumer_offsets,49],[ > mytopicOLD,13],[__consumer_offsets,28],[__consumer_offsets,4],[__consumer_ > offsets,37],[mytopic,12],[mytopicOLD,30],[__consumer_ > offsets,31],[__consumer_offsets,44],[mytopicOLD,15],[ > mytopicOLD,29],[mytopic,37],[mytopic,38],[__consumer_ > offsets,42],[mytopic,27],[mytopic,26],[mytopic,15],[__ > consumer_offsets,34],[mytopic,42],[__consumer_offsets,46],[ > mytopic,14],[mytopicOLD,12],[mytopicOLD,1],[mytopic,7],[__ > consumer_offsets,25],[mytopicOLD,24],[mytopicOLD,44] > ,[mytopicOLD,14],[__consumer_offsets,32],[mytopic,0],[__ > consumer_offsets,43],[mytopic,39],[mytopicOLD,5],[mytopic,9] > ,[mytopic,24],[__consumer_offsets,36],[mytopic,25],[ > mytopicOLD,36],[mytopic,19],[__consumer_offsets,35],[__ > consumer_offsets,7],[mytopic,8],[__consumer_offsets,38],[ > mytopicOLD,48],[mytopicOLD,9],[__consumer_offsets,1],[ > mytopicOLD,6],[mytopic,41],[mytopicOLD,41],[mytopicOLD,7], > [mytopic,17],[mytopicOLD,17],[mytopic,49],[__consumer_ > offsets,16],[__consumer_offsets,2] > > (kafka.controller.PartitionStateMachine)[2017-04-26 12:03:35,045] INFO > > [SessionExpirationListener on 1], ZK expired; shut down all controller > > components and try to re-elect > > (kafka.controller.KafkaController$SessionExpirationListener)[2017-04-26 > > 12:03:35,045] DEBUG [Controller 1]: Controller resigning, broker id 1 > > (kafka.controller.KafkaController)[2017-04-26 12:03:35,045] DEBUG > > [Controller 1]: De-registering IsrChangeNotificationListener > > (kafka.controller.KafkaController)[2017-04-26 12:03:35,060] INFO > [Partition > > state machine on Controller 1]: Stopped partition state machine > > (kafka.controller.PartitionStateMachine)[2017-04-26 12:03:35,060] INFO > > [Replica state machine on controller 1]: Stopped replica state machine > > (kafka.controller.ReplicaStateMachine)[2017-04-26 12:03:35,060] INFO > > [Controller 1]: Broker 1 resigned as the controller > > (kafka.controller.KafkaController)[2017-04-26 12:03:36,013] DEBUG > > [OfflinePartitionLeaderSelector]: No broker in ISR is alive for > > [__consumer_offsets,19]. Pick the leader from the alive assigned > replicas: > > (kafka.controller.OfflinePartitionLeaderSelector)[2017-04-26 > 12:03:36,029] > > DEBUG [OfflinePartitionLeaderSelector]: No broker in ISR is alive for > > [mytopic,11]. Pick the leader from the alive assigned replicas: > > (kafka.controller.OfflinePartitionLeaderSelector)[2017-04-26 > 12:03:36,029] > > DEBUG [OfflinePartitionLeaderSelector]: No broker in ISR is alive for > > [__consumer_offsets,30]. Pick the leader from the alive assigned > replicas: > > (kafka.controller.OfflinePartitionLeaderSelector)[2017-04-26 > 12:03:37,811] > > DEBUG [OfflinePartitionLeaderSelector]: Some broker in ISR is alive for > > [mytopicOLD,18]. Select 2 from ISR 2 to be the leader. > > (kafka.controller.OfflinePartitionLeaderSelector)* > > > > Typical broker config attached.. Please do share some valid inputs... > > > > Abhi > > !wq > > > > > > *-- * > > If you can't succeed, call it version 1.0 > > > > > > -- > If you can't succeed, call it version 1.0 >
