Hi,
I'm experimenting with the following scenario:
- 3 brokers are running (0,1 and 2) -- Kafka version 0.8.2.0
- Continuously: restart broker number 0 by triggering controlled shutdown.
Sleep rand(10) seconds. repeat.
- Continuously: create 'simple-test-topic' (RF=2), write and read messages,
then delete the topic. repeat.
After a while, broker 0 doesn't come back up any more due to "corrupt
index" error (but that's not my question for the moment). Looking at the
state of the topics:
Topic:simple-test-topic PartitionCount:8 ReplicationFactor:2
Configs:
Topic: simple-test-topic Partition: 0 Leader: -1 Replicas: 1,2
Isr: 1
Topic: simple-test-topic Partition: 1 Leader: -1 Replicas: 2,0
Isr: 2
Topic: simple-test-topic Partition: 2 Leader: -1 Replicas: 0,1
Isr: 1
Topic: simple-test-topic Partition: 3 Leader: -1 Replicas: 1,0
Isr: 1
Topic: simple-test-topic Partition: 4 Leader: -1 Replicas: 2,1
Isr: 1
Topic: simple-test-topic Partition: 5 Leader: -1 Replicas: 0,2
Isr: 2
Topic: simple-test-topic Partition: 6 Leader: -1 Replicas: 1,2
Isr: 1
Topic: simple-test-topic Partition: 7 Leader: -1 Replicas: 2,0
Isr: 2
Topic:test PartitionCount:8 ReplicationFactor:3 Configs:
Topic: test Partition: 0 Leader: 1 Replicas: 1,2,0 Isr: 2,1
Topic: test Partition: 1 Leader: 2 Replicas: 2,0,1 Isr: 2,1
Topic: test Partition: 2 Leader: 2 Replicas: 0,1,2 Isr: 2,1
Topic: test Partition: 3 Leader: 1 Replicas: 1,0,2 Isr: 2,1
Topic: test Partition: 4 Leader: 2 Replicas: 2,1,0 Isr: 2,1
Topic: test Partition: 5 Leader: 2 Replicas: 0,2,1 Isr: 2,1
Topic: test Partition: 6 Leader: 1 Replicas: 1,2,0 Isr: 2,1
Topic: test Partition: 7 Leader: 2 Replicas: 2,0,1 Isr: 2,1
.. at which point:
- All 'simple-test-topic' partitions are leaderless.
- It's not possible to delete "simple-test-topic" any more.
- Calling 'kafka-preferred-replica-election.sh' successfully starts
election but doesn't have any effect.
The other topic, named "test" (RF 3), is just sitting there and not
actively participating in the test.
Now I'm wondering:
- Which of the steps above could have caused "simple-test-topic" partitions
to become leaderless?
- How to recover in such situation in cases where broker 0 can or cannot be
recovered?
Thanks,
Omid
