Running Kafka 0.9.0.1 (Producers, Brokers and Consumers).
I'm getting sporadic ISR shrinks and expanding of the [__consumer_offsets] topic which seems to cause some sort of message duplication in message consumption here and there. I was wondering what could cause this ? I am running 1 ZooKeeper, 2 Brokers , 3 Consumers , around 100K tps. Using replication factor 1 with topics with 4 partitions. Some of the partitions are written to broker 0 and some to broker 1 (no replication at all) I am getting the following on both brokers quite often: 2016-03-21 16:03:34,972] INFO Partition [__consumer_offsets,44] on broker 1: Shrinking ISR for partition [__consumer_offsets,44] from 0,1 to 1 (kafka.cluster.Partition) [2016-03-21 16:03:34,992] INFO Partition [__consumer_offsets,6] on broker 1: Shrinking ISR for partition [__consumer_offsets,6] from 0,1 to 1 (kafka.cluster.Partition) [2016-03-21 16:03:35,039] INFO Partition [__consumer_offsets,10] on broker 1: Shrinking ISR for partition [__consumer_offsets,10] from 1,0 to 1 (kafka.cluster.Partition) [2016-03-21 16:03:35,039] INFO Partition [__consumer_offsets,20] on broker 1: Shrinking ISR for partition [__consumer_offsets,20] from 0,1 to 1 (kafka.cluster.Partition) [2016-03-21 16:03:35,040] INFO Partition [__consumer_offsets,4] on broker 1: Shrinking ISR for partition [__consumer_offsets,4] from 0,1 to 1 (kafka.cluster.Partition) [2016-03-21 16:03:35,041] INFO Partition [__consumer_offsets,28] on broker 1: Shrinking ISR for partition [__consumer_offsets,28] from 0,1 to 1 (kafka.cluster.Partition) [2016-03-21 16:03:35,041] INFO Partition [__consumer_offsets,38] on broker 1: Shrinking ISR for partition [__consumer_offsets,38] from 0,1 to 1 (kafka.cluster.Partition) [2016-03-21 16:03:35,042] INFO Partition [__consumer_offsets,26] on broker 1: Shrinking ISR for partition [__consumer_offsets,26] from 1,0 to 1 (kafka.cluster.Partition) [2016-03-21 16:03:35,042] INFO Partition [__consumer_offsets,22] on broker 1: Shrinking ISR for partition [__consumer_offsets,22] from 0,1 to 1 (kafka.cluster.Partition) [2016-03-21 16:03:35,043] INFO Partition [__consumer_offsets,34] on broker 1: Shrinking ISR for partition [__consumer_offsets,34] from 0,1 to 1 (kafka.cluster.Partition) [2016-03-21 16:03:35,043] INFO Partition [__consumer_offsets,8] on broker 1: Shrinking ISR for partition [__consumer_offsets,8] from 0,1 to 1 (kafka.cluster.Partition) [2016-03-21 16:03:35,044] INFO Partition [__consumer_offsets,30] on broker 1: Shrinking ISR for partition [__consumer_offsets,30] from 1,0 to 1 (kafka.cluster.Partition) [2016-03-21 16:03:35,044] INFO Partition [__consumer_offsets,40] on broker 1: Shrinking ISR for partition [__consumer_offsets,40] from 1,0 to 1 (kafka.cluster.Partition) [2016-03-21 16:03:35,045] INFO Partition [__consumer_offsets,0] on broker 1: Shrinking ISR for partition [__consumer_offsets,0] from 1,0 to 1 (kafka.cluster.Partition) [2016-03-21 16:03:35,174] INFO Partition [__consumer_offsets,32] on broker 1: Expanding ISR for partition [__consumer_offsets,32] from 1 to 1,0 (kafka.cluster.Partition) [2016-03-21 16:03:35,175] INFO Partition [__consumer_offsets,16] on broker 1: Expanding ISR for partition [__consumer_offsets,16] from 1 to 1,0 (kafka.cluster.Partition) [2016-03-21 16:03:35,176] INFO Partition [__consumer_offsets,44] on broker 1: Expanding ISR for partition [__consumer_offsets,44] from 1 to 1,0 (kafka.cluster.Partition) [2016-03-21 16:03:35,177] INFO Partition [__consumer_offsets,28] on broker 1: Expanding ISR for partition [__consumer_offsets,28] from 1 to 1,0 (kafka.cluster.Partition) [2016-03-21 16:03:35,178] INFO Partition [__consumer_offsets,4] on broker 1: Expanding ISR for partition [__consumer_offsets,4] from 1 to 1,0 (kafka.cluster.Partition) [2016-03-21 16:03:35,179] INFO Partition [__consumer_offsets,24] on broker 1: Expanding ISR for partition [__consumer_offsets,24] from 1 to 1,0 (kafka.cluster.Partition) [2016-03-21 16:03:35,180] INFO Partition [__consumer_offsets,0] on broker 1: Expanding ISR for partition [__consumer_offsets,0] from 1 to 1,0 (kafka.cluster.Partition) [2016-03-21 16:03:35,181] INFO Partition [__consumer_offsets,38] on broker 1: Expanding ISR for partition [__consumer_offsets,38] from 1 to 1,0 (kafka.cluster.Partition) [2016-03-21 16:03:35,182] INFO Partition [__consumer_offsets,8] on broker 1: Expanding ISR for partition [__consumer_offsets,8] from 1 to 1,0 (kafka.cluster.Partition) [2016-03-21 16:03:35,183] INFO Partition [__consumer_offsets,36] on broker 1: Expanding ISR for partition [__consumer_offsets,36] from 1 to 1,0 (kafka.cluster.Partition) [2016-03-21 16:03:35,184] INFO Partition [__consumer_offsets,40] on broker 1: Expanding ISR for partition [__consumer_offsets,40] from 1 to 1,0 (kafka.cluster.Partition) [2016-03-21 16:03:35,185] INFO Partition [__consumer_offsets,6] on broker 1: Expanding ISR for partition [__consumer_offsets,6] from 1 to 1,0 (kafka.cluster.Partition) [2016-03-21 16:24:00,002] ERROR [ReplicaFetcherThread-0-1], Error for partition [__consumer_offsets,46] to broker 1:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread) [2016-03-21 16:24:00,002] ERROR [ReplicaFetcherThread-0-1], Error for partition [__consumer_offsets,32] to broker 1:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread) [2016-03-21 16:24:00,002] ERROR [ReplicaFetcherThread-0-1], Error for partition [__consumer_offsets,36] to broker 1:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread) Any idea what can cause this ? Thank you, Jordan.
