Hi group,
I've been through some data loss on kafka cluster, one case maybe caused by
the auto preferred leader election.
Here is the situation: 3 brokers = {0,1,2}, 1 partition with 3 replicas on
0/1/2, all in sync while 0 is leader and 1 is controller, current offset is
100.
And here is my hypothesis:
a. Leader 0 is temporary gone due to instablity connection with ZK
b. Controller 1 found that 0 has gone then do a election which let 1(in
ISR) to be leader
c. A producer send 1 message to new leader 1, then offset is 101
d. Old leader 0 is back to cluster(*STILL IN ISR* because the lag not
exceed the *replica.lag.time.max.ms <http://replica.lag.time.max.ms>* and
*replica.lag.max.messages*)
e. Coincidentally, controller 1 start to checkAndTriggerPartitionRebalance
then decide 0 is more preferred, so let 0 to be back to leader
f. Broker 1 become to follower then found HW to be 100, so truncate to 100
that lead to lost newest message.
With this situation, using most reliable settings(broker side:
unclean.leader.election.enable=false, min.insync.replicas=2; producer side
acks=all) is useless. Am I correct or just paranoia?
Below is some real logs from production.
I*n controller.log:*
> *// broker 6 tempraroy gone*
>
[2015-09-09 15:24:42,206] INFO [BrokerChangeListener on Controller 3]:
> Newly added brokers: , deleted brokers: 6, all live brokers: 0,5,1,2,7,3,4
> (kafka.controller.ReplicaStateMachine$BrokerChangeListener)
> [2015-09-09 15:24:42,461] INFO [Controller 3]: Broker failure callback for
> 6 (kafka.controller.KafkaController)
> [2015-09-09 15:24:42,464] INFO [Controller 3]: Removed ArrayBuffer() from
> list of shutting down brokers. (kafka.controller.KafkaController)
> [2015-09-09 15:24:42,466] INFO [Partition state machine on Controller 3]:
> Invoking state change to OfflinePartition for partitions
> [SOME_TOPIC_NAME,1] (kafka.controller.PartitionStateMachine)
>
> *// elect 3 which in ISR to be leader*
> [2015-09-09 15:24:43,182] DEBUG [OfflinePartitionLeaderSelector]: Some
> broker in ISR is alive for [SOME_TOPIC_NAME,1]. Select 3 from ISR 3,4 to be
> the leader. (kafka.controller.OfflinePartitionLeaderSelector)
> [2015-09-09 15:24:43,182] INFO [OfflinePartitionLeaderSelector]: Selected
> new leader and ISR {"leader":3,"leader_epoch":45,"isr":[3,4]} for offline
> partition [SOME_TOPIC_NAME,1]
> (kafka.controller.OfflinePartitionLeaderSelector)
> [2015-09-09 15:24:43,928] DEBUG [Controller 3]: Removing replica 6 from
> ISR 3,4 for partition [SOME_TOPIC_NAME,1].
> (kafka.controller.KafkaController)
> [2015-09-09 15:24:43,929] WARN [Controller 3]: Cannot remove replica 6
> from ISR of partition [SOME_TOPIC_NAME,1] since it is not in the ISR.
> Leader = 3 ; ISR = List(3, 4) (kafka.controller.KafkaController)
>
> *// broker 6 back*
>
[2015-09-09 15:24:44,575] INFO [BrokerChangeListener on Controller 3]:
> Newly added brokers: 6, deleted brokers: , all live brokers:
> 0,5,1,6,2,7,3,4 (kafka.controller.ReplicaStateMachine$BrokerChangeListener)
>
> *// broker 6 is elected as leader by auto preferred leader election*
>
[2015-09-09 15:24:50,939] INFO [Controller 3]: Starting preferred replica
> leader election for partitions [SOME_TOPIC_NAME,1]
> (kafka.controller.KafkaController)
> [2015-09-09 15:24:50,945] INFO [PreferredReplicaPartitionLeaderSelector]:
> Current leader 3 for partition [SOME_TOPIC_NAME,1] is not the preferred
> replica. Trigerring preferred replica leader election
> (kafka.controller.PreferredReplicaPartitionLeaderSelector)
>
>
*And in server.log:*
> *// broker 3 truncating, lossing data*
2015-09-09 15:24:50,964] INFO Truncating log SOME_TOPIC_NAME-1 to offset
> 420549. (kafka.log.Log)
>
