Hello Apache Kafka community, Consumer offsets topic default configuration is somewhat different from all other topics for a good reason, e.g. replication factor is 3 compared to 1 for other topics, and number of partitions is 50 compared to 1 for other topic. Can somebody please explain why does consumer offsets topic default to min.insync.replicas of 1, or in other words would it make sense to set it to 2?
I guess that min.insync.replicas of 1 is preferred for higher availability of consumer offsets topic partitions (whichever replica is alive allow it to become leader regardless of how far behind it was) compared to higher consistency across replicas achieved with min.insync.replicas of 2 or higher. Problem I see there, and please correct me if wrong, is that the unclean.leader.election.enable seems to be configurable only on cluster/broker level, one cannot override it on topic level (as documented http://kafka.apache.org/documentation.html#topic-config ). On the other hand one can override min.insync.replicas per topic and have it default to 1 on broker level. So, if one disables unclean leader election, and has min.insync.replicas set to 1 for consumer offsets topic and to 2 for some or all of the other topics, one would actually make consumer offsets topic partitions less available as loosing a lead would make partition unavailable until lead is recovered, since no other replica if it wasn't part of ISR set cannot take over, as by default with min.insync.replicas of 1 only a lead is guaranteed to be part of ISR set. Would it make sense to allow one to override unclean.leader.election.enable on topic level, introduce separate offsets.topic.unclean.leader.election.enable and have it default to true, and introduce offsets.topic.min.insync.replicas defaulting to 1, documenting rationale for all of these, suggesting one to increase offsets.topic.min.insync.replicas if offsets.topic.unclean.leader.election.enable is overridden to false? Kind regards, Stevo Slavic.
