Hi all,
I'm experiencing an issue which I don't know how to solve, so I'm trying to
find some guidance on the topic.
I have a cluster composed by 3 servers, one broker per server running Kafka
0.10.0.1-1 which runs in production with around 100 topics, most of them
divided in several partitions and replicated always between 2 servers.
Suddenly I've notice when looking at my topics (with kafka-topics tool)
that no one of them have a leader (Leader: -1) and the list of ISR appears
empty for all the topics.
So they look something like:
Topic:mytopic PartitionCount:3 ReplicationFactor:2 Configs:
retention.ms=86400000
Topic: mytopic Partition: 0 Leader: -1 Replicas: 30,10 Isr:
Topic: mytopic Partition: 1 Leader: -1 Replicas: 10,20 Isr:
Topic: mytopic Partition: 2 Leader: -1 Replicas: 20,30 Isr:
However the applications using it are running normally, consumers as well
as producers.
The logs are not showing errors or weird messages with the exceptions of
some
Failed to rename [/var/log/kafka/log-cleaner.log] to
[/var/log/kafka/log-cleaner.log.2017-05-31-17]
which appear each few days.
Now I would like to bring back to cluster to a good state. I'm afraid of
restarting brokers because all of them are supposed to be leaders for some
partitions, so if I restart them and there's no leader I might experience
data loss.
Have you face any similar situation? Can someone give me a hint?
Thanks in advance,
Alberto.
