Hi Mohammed, thanks for your answer. The ZK cluster is not located in the servers where Kafka runs but in other 3 different machines. This ZK cluster is used by several other services which are not reporting problems. As you suggested, I haven't tried restarting the kafka-server processes because there's no leader for topic partitions, so I don't know what will happen. Never been in a similar situation with Kafka after some years using it.
On 1 June 2017 at 16:29, Mohammed Manna <[email protected]> wrote: > Hi Alberto, > > Usually this means that the leader election/replica syncing couldn't be > successful and the zookeeper logs should be able to show this information > too. The leader -1 is what worries me. For your case (3 broker cluster), I > am assuming you have done the cluster configuration to have 1 > broker-zookeeper setup ? > If that's the case, you should be able to bounce 1 zookeeper at a time and > see if that resolves the issue. > > That said, have you restarted your servers since this issue surfaced? > > On 1 June 2017 at 14:11, Del Barrio, Alberto < > [email protected]> wrote: > > > Hi all, > > > > I'm experiencing an issue which I don't know how to solve, so I'm trying > to > > find some guidance on the topic. > > > > I have a cluster composed by 3 servers, one broker per server running > Kafka > > 0.10.0.1-1 which runs in production with around 100 topics, most of them > > divided in several partitions and replicated always between 2 servers. > > Suddenly I've notice when looking at my topics (with kafka-topics tool) > > that no one of them have a leader (Leader: -1) and the list of ISR > appears > > empty for all the topics. > > So they look something like: > > > > Topic:mytopic PartitionCount:3 ReplicationFactor:2 Configs: > > retention.ms=86400000 > > Topic: mytopic Partition: 0 Leader: -1 Replicas: 30,10 > Isr: > > Topic: mytopic Partition: 1 Leader: -1 Replicas: 10,20 > Isr: > > Topic: mytopic Partition: 2 Leader: -1 Replicas: 20,30 > Isr: > > > > However the applications using it are running normally, consumers as well > > as producers. > > The logs are not showing errors or weird messages with the exceptions of > > some > > Failed to rename [/var/log/kafka/log-cleaner.log] to > > [/var/log/kafka/log-cleaner.log.2017-05-31-17] > > which appear each few days. > > > > Now I would like to bring back to cluster to a good state. I'm afraid of > > restarting brokers because all of them are supposed to be leaders for > some > > partitions, so if I restart them and there's no leader I might experience > > data loss. > > > > Have you face any similar situation? Can someone give me a hint? > > > > Thanks in advance, > > Alberto. > > > -- News, jobs, product releases, events. Follow 360dialog on LinkedIn <https://www.linkedin.com/company/360dialog> and Twitter <https://twitter.com/360dialog>. Subscribe to our newsletter <http://www.360dialog.com/newsletter/>. *Alberto del Barrio*DevOps Engineer <http://www.360dialog.com?utm_campaign=email-signature&utm_content=-&utm_medium=email&utm_source=signature&utm_term=logo> *Contact 360dialog*www.360dialog.com <http://www.360dialog.com/?utm_campaign=email-signature&utm_content=-&utm_medium=email&utm_source=signature&utm_term=url> [email protected] +49-(0)30-6098-5953-0 360dialog GmbH, Saarbrücker Str. 36-38, 10405 Berlin, Germany Managing director: Roland Siebert Commercial register: Charlottenbug, HRB 144188 B VAT ID: DE815382679
