Hi all,
I reconfigured my kafka cluster, changing:
- default replication factor from 1 to 3 and also
- changing the location of the kafka data dir on disk
So after restarting all nodes, the cluster seemed ok but then I noticed
all the topics are failing to come online. In the logs there are messages
like this for each topic:
state-change.log: [2018-02-01 12:41:42,176] ERROR Controller 826437096
epoch 19 initiated state change for partition [filedrop,0] from
OfflinePartition to OnlinePartition failed (state.change.logger)
So none of the topics are usable; Listing topics with kafkacat -L -b shows
leaders not availabil
---
Metadata for all topics (from broker -1: lol-045:9092/bootstrap):
7 brokers:
broker 826437096 at lol-044:9092
broker 746155422 at lol-047:9092
broker 651737161 at lol-046:9092
broker 728512596 at lol-048:9092
broker 213763378 at lol-045:9092
broker 622553932 at lol-049:9092
broker 746727274 at lol-050:9092
14 topics:
topic "lol.stripped" with 3 partitions:
partition 2, leader -1, replicas: , isrs: , Broker: Leader not available
partition 1, leader -1, replicas: , isrs: , Broker: Leader not available
partition 0, leader -1, replicas: , isrs: , Broker: Leader not available
---
However, newly created topics are correctly replicated and healthy
---
topic "lol-kafka-health" with 3 partitions:
partition 2, leader 622553932, replicas: 622553932,213763378,651737161,
isrs: 622553932,213763378,651737161
partition 1, leader 213763378, replicas: 622553932,213763378,826437096,
isrs: 213763378,826437096,622553932
partition 0, leader 826437096, replicas: 213763378,746727274,826437096,
isrs: 826437096,746727274,213763378
---
So I think some kind of metadata corruption happened during the reconfigure
My question is:
- Is there any way I can get these topic partitions online again ?
Given that:
- the broker ids were changed during the reconfigure
- the zookeeper cluster for kafka went down temporarily during the
reconfig
In addition, are there some procedures I can use to investigate how
recoverable these topics are?
Many thanks in advance!
Traiano
