[
https://issues.apache.org/jira/browse/KAFKA-12405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander updated KAFKA-12405:
------------------------------
Summary: Inconsistent broker without active topics still involved in
coordinating consumer group (was: Broker without active topics still involved
in coordinating consumer group)
> Inconsistent broker without active topics still involved in coordinating
> consumer group
> ---------------------------------------------------------------------------------------
>
> Key: KAFKA-12405
> URL: https://issues.apache.org/jira/browse/KAFKA-12405
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 2.3.0
> Reporter: Alexander
> Priority: Major
>
> Short description:
> Broker without active topics and with inconsistent data still involved in
> coordinating consumer group. It leads to constant rebalancing consumer group
>
> Details:
> We have kafka cluster(kafka 2.3.0) of three nodes(1,2,3)
> * After high disk load broker(2) came out of cluster
> * Brokers (1), (3) shortly became leaders of it's topics
> * Broker(2) back to cluster, but have failed to became leader of it's
> preferred topics due to
> "Out of sync replicas". Now it start spamming with
> "Shrinking ISR from 3,2,1 to 2, ... Out of sync replicas"
> "Cached zkVersion [20] not equal to that in zookeeper, skip updating ISR"
> Like in https://issues.apache.org/jira/browse/KAFKA-3042
> But at the same time we start to observing constant rebalancing of consumer
> group "mirror_maker_group" and topic reading within this group was completely
> stucked. Restarting broker(2) fix the problem
> Broker(2) server log:
> {noformat}
> [executor-Produce] [kafka.coordinator.group.GroupCoordinator] INFO
> [GroupCoordinator 2]: Preparing to rebalance group mirror_maker_group in
> state PreparingRebalance with old generation 7139 (__consumer_offsets-30)
> (reason: error when storing group assignment during SyncGroup)"
> [data-plane-kafka-request-handler-6]
> [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]:
> Stabilized group mirror_maker_group generation 7140 (__consumer_offsets-30)
> [data-plane-kafka-request-handler-7]
> [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]:
> Assignment received from leader for group mirror_maker_group for generation
> 7140{noformat}
> The number of log entry "Rebalancing group..." per hour
> {noformat}
> $ for log in `grep -l 'Preparing to rebalance group mirror_maker_group' *`;
> do echo -n "$log "; grep 'Preparing to rebalance group mirror_maker_group'
> $log | wc -l; done
> server.2021-03-02-10.kafka3 5
> server.2021-03-02-11.kafka3 1
> server.2021-03-02-12.kafka3 3
> server.2021-03-02-10.kafka2 1
> server.2021-03-02-11.kafka2 306
> server.2021-03-02-12.kafka2 452
> server.2021-03-02-13.kafka2 1 # Here we rebooted broker(2)
> {noformat}
>
> The questions:
> * Is it a bug or topicless broker should be involved in coordinating
> consumer group?
> * If it's a bug, does this fixed in any later kafka release?
> Thank you.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
