Replica not coming up for a partition after restarting broker | Error NOT_LEADER_FOR_PARTITION

[Manjunath N]

Hi,

Problem: A replica for a partition is not coming up. Steps below;

I have a zookeeper cluster with three machines and a kafka cluster with 3 
brokers on the same machine. 

I created a topic test as below.
Operation: kafka-topics.sh --zookeeper rh3:2181/kafka --create --topic test 
--replication-factor 2 --partitions 1

Step 1

Status: kafka-topics.sh --zookeeper rh3:2181/kafka --describe
                Topic:test      PartitionCount:1        ReplicationFactor:2     
Configs:
                        Topic: test     Partition: 0    Leader: 5       
Replicas: 5,4   Isr: 5,4

Later i altered the topic to add a new partition: 

Step 2:
Operation: kafka-topics.sh --zookeeper rh3:2181/kafka --alter --topic test 
--partitions 2

Status: kafka-topics.sh --zookeeper rh3:2181/kafka --describe
                Topic:test      PartitionCount:2        ReplicationFactor:2     
Configs:
                        Topic: test     Partition: 0    Leader: 5       
Replicas: 5,4   Isr: 5,4
                        Topic: test     Partition: 1    Leader: 3       
Replicas: 3,4   Isr: 3,4

Step 3: now i shutdown broker 4

Status: kafka-topics.sh --zookeeper rh3:2181/kafka --describe
                        Topic:test      PartitionCount:2        
ReplicationFactor:2     Configs:
                                Topic: test     Partition: 0    Leader: 5       
Replicas: 5,4   Isr: 5
                                Topic: test     Partition: 1    Leader: 3       
Replicas: 3,4   Isr: 3

Steep 4: Later, when i got broker up and running. The replica for partition 1 
is not syncing up. Here is the status.

Status: kafka-topics.sh --zookeeper rh3:2181/kafka --describe
                        Topic:test      PartitionCount:2        
ReplicationFactor:2     Configs:
                                Topic: test     Partition: 0    Leader: 5       
Replicas: 5,4   Isr: 5,4
                                Topic: test     Partition: 1    Leader: 3       
Replicas: 3,4   Isr: 3

Log in broker 4 shows me this error. could you please let me know what could be 
done restore the replica for partition 1.

[2018-08-01 06:02:18,685] INFO [ReplicaFetcher replicaId=4, leaderId=3, 
fetcherId=0] Retrying leaderEpoch request for partition test-1 as the leader 
reported an error: NOT_LEADER_FOR_PARTITION (kafka.server.ReplicaFetcherThread)
[2018-08-01 06:02:19,690] INFO [ReplicaFetcher replicaId=4, leaderId=3, 
fetcherId=0] Retrying leaderEpoch request for partition test-1 as the leader 
reported an error: NOT_LEADER_FOR_PARTITION (kafka.server.ReplicaFetcherThread)

Thanks
Manjunath