[jira] [Created] (KAFKA-13384) FailedPartitionsCount metric is not updated if a partition log file was corrupted
Alexander created KAFKA-13384: - Summary: FailedPartitionsCount metric is not updated if a partition log file was corrupted Key: KAFKA-13384 URL: https://issues.apache.org/jira/browse/KAFKA-13384 Project: Kafka Issue Type: Bug Affects Versions: 2.6.2 Environment: OS: NAME="Amazon Linux AMI" VERSION="2018.03" ID="amzn" ID_LIKE="rhel fedora" VERSION_ID="2018.03" PRETTY_NAME="Amazon Linux AMI 2018.03" CPE_NAME="cpe:/o:amazon:linux:2018.03:ga" HOME_URL="http://aws.amazon.com/amazon-linux-ami/"; Kafka version: 2.6.2 (Commit:da65af02e5856e34) Reporter: Alexander Attachments: Screenshot 2021-10-19 at 15.28.33.png We found a misbehavior on our Kafka cluster (version: 2.6.2 (Commit:da65af02e5856e34)), `FailedPartitionsCount` metric is not updated if a partition log file was corrupted Steps to reproduce the problem: 1. corrupt a partition log file 2. restart Kafka process After that, you will get a correct log which tells that Kafka marked corrupted partitions as failed ``` 2021-10-19T14:49:31+02:00 [2021-10-19 12:49:30,924] WARN [ReplicaFetcher replicaId=11, leaderId=10, fetcherId=0] Partition test_topic-1 marked as failed (kafka.server.ReplicaFetcherThread) ``` But the value of `FailedPartitionsCount` metric will be 0 (see attached screenshot) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-13384) FailedPartitionsCount metric is not updated if a partition log file was corrupted
[ https://issues.apache.org/jira/browse/KAFKA-13384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander updated KAFKA-13384: -- Description: We found a misbehavior on our Kafka cluster (version: 2.6.2 (Commit:da65af02e5856e34)), `FailedPartitionsCount` metric is not updated if a partition log file was corrupted Steps to reproduce the problem: # corrupt a partition log file # restart Kafka process After that, you will get a correct log which tells that Kafka marked corrupted partitions as failed {code:java} 2021-10-19T14:49:31+02:00 [2021-10-19 12:49:30,924] WARN [ReplicaFetcher replicaId=11, leaderId=10, fetcherId=0] Partition test_topic-1 marked as failed (kafka.server.ReplicaFetcherThread){code} But the value of `FailedPartitionsCount` metric will be 0 (see attached screenshot) was: We found a misbehavior on our Kafka cluster (version: 2.6.2 (Commit:da65af02e5856e34)), `FailedPartitionsCount` metric is not updated if a partition log file was corrupted Steps to reproduce the problem: 1. corrupt a partition log file 2. restart Kafka process After that, you will get a correct log which tells that Kafka marked corrupted partitions as failed ``` 2021-10-19T14:49:31+02:00 [2021-10-19 12:49:30,924] WARN [ReplicaFetcher replicaId=11, leaderId=10, fetcherId=0] Partition test_topic-1 marked as failed (kafka.server.ReplicaFetcherThread) ``` But the value of `FailedPartitionsCount` metric will be 0 (see attached screenshot) > FailedPartitionsCount metric is not updated if a partition log file was > corrupted > - > > Key: KAFKA-13384 > URL: https://issues.apache.org/jira/browse/KAFKA-13384 > Project: Kafka > Issue Type: Bug >Affects Versions: 2.6.2 > Environment: OS: > NAME="Amazon Linux AMI" > VERSION="2018.03" > ID="amzn" > ID_LIKE="rhel fedora" > VERSION_ID="2018.03" > PRETTY_NAME="Amazon Linux AMI 2018.03" > CPE_NAME="cpe:/o:amazon:linux:2018.03:ga" > HOME_URL="http://aws.amazon.com/amazon-linux-ami/"; > Kafka version: > 2.6.2 (Commit:da65af02e5856e34) >Reporter: Alexander >Priority: Major > Attachments: Screenshot 2021-10-19 at 15.28.33.png > > > We found a misbehavior on our Kafka cluster (version: 2.6.2 > (Commit:da65af02e5856e34)), `FailedPartitionsCount` metric is not updated if > a partition log file was corrupted > Steps to reproduce the problem: > # corrupt a partition log file > # restart Kafka process > After that, you will get a correct log which tells that Kafka marked > corrupted partitions as failed > > {code:java} > 2021-10-19T14:49:31+02:00 [2021-10-19 12:49:30,924] WARN [ReplicaFetcher > replicaId=11, leaderId=10, fetcherId=0] Partition test_topic-1 marked as > failed > (kafka.server.ReplicaFetcherThread){code} > > > But the value of `FailedPartitionsCount` metric will be 0 (see attached > screenshot) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-13384) FailedPartitionsCount metric is not updated if a partition log file was corrupted
[ https://issues.apache.org/jira/browse/KAFKA-13384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander updated KAFKA-13384: -- Description: We found a misbehavior on our Kafka cluster (version: 2.6.2 (Commit:da65af02e5856e34)), `FailedPartitionsCount` metric is not updated if a partition log file was corrupted Steps to reproduce the problem: # corrupt a partition log file # restart Kafka process After that, you will get a correct log which tells that Kafka marked corrupted partitions as failed {code:java} 2021-10-19T14:49:31+02:00 [2021-10-19 12:49:30,924] WARN [ReplicaFetcher replicaId=11, leaderId=10, fetcherId=0] Partition test_topic-1 marked as failed (kafka.server.ReplicaFetcherThread){code} But the value of `FailedPartitionsCount` metric will be 0 (see attached screenshot) was: We found a misbehavior on our Kafka cluster (version: 2.6.2 (Commit:da65af02e5856e34)), `FailedPartitionsCount` metric is not updated if a partition log file was corrupted Steps to reproduce the problem: # corrupt a partition log file # restart Kafka process After that, you will get a correct log which tells that Kafka marked corrupted partitions as failed {code:java} 2021-10-19T14:49:31+02:00 [2021-10-19 12:49:30,924] WARN [ReplicaFetcher replicaId=11, leaderId=10, fetcherId=0] Partition test_topic-1 marked as failed (kafka.server.ReplicaFetcherThread){code} But the value of `FailedPartitionsCount` metric will be 0 (see attached screenshot) > FailedPartitionsCount metric is not updated if a partition log file was > corrupted > - > > Key: KAFKA-13384 > URL: https://issues.apache.org/jira/browse/KAFKA-13384 > Project: Kafka > Issue Type: Bug >Affects Versions: 2.6.2 > Environment: OS: > NAME="Amazon Linux AMI" > VERSION="2018.03" > ID="amzn" > ID_LIKE="rhel fedora" > VERSION_ID="2018.03" > PRETTY_NAME="Amazon Linux AMI 2018.03" > CPE_NAME="cpe:/o:amazon:linux:2018.03:ga" > HOME_URL="http://aws.amazon.com/amazon-linux-ami/"; > Kafka version: > 2.6.2 (Commit:da65af02e5856e34) >Reporter: Alexander >Priority: Major > Attachments: Screenshot 2021-10-19 at 15.28.33.png > > > We found a misbehavior on our Kafka cluster (version: 2.6.2 > (Commit:da65af02e5856e34)), `FailedPartitionsCount` metric is not updated if > a partition log file was corrupted > Steps to reproduce the problem: > # corrupt a partition log file > # restart Kafka process > After that, you will get a correct log which tells that Kafka marked > corrupted partitions as failed > > {code:java} > 2021-10-19T14:49:31+02:00 [2021-10-19 12:49:30,924] WARN [ReplicaFetcher > replicaId=11, leaderId=10, fetcherId=0] Partition test_topic-1 marked as > failed > (kafka.server.ReplicaFetcherThread){code} > > But the value of `FailedPartitionsCount` metric will be 0 (see attached > screenshot) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-13384) FailedPartitionsCount metric is not updated if a partition log file was corrupted
[ https://issues.apache.org/jira/browse/KAFKA-13384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander updated KAFKA-13384: -- Description: We found a misbehavior on our Kafka cluster - version: 2.6.2 (Commit:da65af02e5856e34)) _*FailedPartitionsCount*_ metric is not updated if a partition log file was corrupted Steps to reproduce the problem: # corrupt a partition log file # restart Kafka process After that, you will get a correct log which tells that Kafka marked corrupted partitions as failed {code:java} 2021-10-19T14:49:31+02:00 [2021-10-19 12:49:30,924] WARN [ReplicaFetcher replicaId=11, leaderId=10, fetcherId=0] Partition test_topic-1 marked as failed (kafka.server.ReplicaFetcherThread){code} But the value of _*FailedPartitionsCount*_ metric will be 0 (see attached screenshot) was: We found a misbehavior on our Kafka cluster (version: 2.6.2 (Commit:da65af02e5856e34)), `FailedPartitionsCount` metric is not updated if a partition log file was corrupted Steps to reproduce the problem: # corrupt a partition log file # restart Kafka process After that, you will get a correct log which tells that Kafka marked corrupted partitions as failed {code:java} 2021-10-19T14:49:31+02:00 [2021-10-19 12:49:30,924] WARN [ReplicaFetcher replicaId=11, leaderId=10, fetcherId=0] Partition test_topic-1 marked as failed (kafka.server.ReplicaFetcherThread){code} But the value of `FailedPartitionsCount` metric will be 0 (see attached screenshot) > FailedPartitionsCount metric is not updated if a partition log file was > corrupted > - > > Key: KAFKA-13384 > URL: https://issues.apache.org/jira/browse/KAFKA-13384 > Project: Kafka > Issue Type: Bug >Affects Versions: 2.6.2 > Environment: OS: > NAME="Amazon Linux AMI" > VERSION="2018.03" > ID="amzn" > ID_LIKE="rhel fedora" > VERSION_ID="2018.03" > PRETTY_NAME="Amazon Linux AMI 2018.03" > CPE_NAME="cpe:/o:amazon:linux:2018.03:ga" > HOME_URL="http://aws.amazon.com/amazon-linux-ami/"; > Kafka version: > 2.6.2 (Commit:da65af02e5856e34) >Reporter: Alexander >Priority: Major > Attachments: Screenshot 2021-10-19 at 15.28.33.png > > > We found a misbehavior on our Kafka cluster - version: 2.6.2 > (Commit:da65af02e5856e34)) _*FailedPartitionsCount*_ metric is not updated if > a partition log file was corrupted > Steps to reproduce the problem: > # corrupt a partition log file > # restart Kafka process > After that, you will get a correct log which tells that Kafka marked > corrupted partitions as failed > {code:java} > 2021-10-19T14:49:31+02:00 [2021-10-19 12:49:30,924] WARN [ReplicaFetcher > replicaId=11, leaderId=10, fetcherId=0] Partition test_topic-1 marked as > failed > (kafka.server.ReplicaFetcherThread){code} > But the value of _*FailedPartitionsCount*_ metric will be 0 (see attached > screenshot) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-12405) Broker without active topics still involved in coordinating consumer group
Alexander created KAFKA-12405: - Summary: Broker without active topics still involved in coordinating consumer group Key: KAFKA-12405 URL: https://issues.apache.org/jira/browse/KAFKA-12405 Project: Kafka Issue Type: Bug Components: core Affects Versions: 2.3.0 Reporter: Alexander Short description: Broker without active topics and with inconsistent data still involved in coordinating consumer group. It leads to constant rebalancing consumer group Details: We have kafka cluster(kafka 2.3.0) of three nodes(1,2,3) * After high disk load broker(2) came out of cluster * Brokers (1), (3) shortly became leaders of it's topics * Broker(2) back to cluster, but have failed to became leader of it's preferred topics due to "Out of sync replicas". Now it start spamming with "Shrinking ISR from 3,2,1 to 2, ... Out of sync replicas" "Cached zkVersion [20] not equal to that in zookeeper, skip updating ISR" Like in https://issues.apache.org/jira/browse/KAFKA-3042 But at the same time we start to observing constant rebalancing of consumer group "mirror_maker_group" and topic reading within this group was completely stucked. Restarting broker(2) fix the problem Broker(2) server log: {noformat} [executor-Produce] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Preparing to rebalance group mirror_maker_group in state PreparingRebalance with old generation 7139 (__consumer_offsets-30) (reason: error when storing group assignment during SyncGroup)" [data-plane-kafka-request-handler-6] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Stabilized group mirror_maker_group generation 7140 (__consumer_offsets-30) [data-plane-kafka-request-handler-7] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Assignment received from leader for group mirror_maker_group for generation 7140{noformat} {noformat} $ for log in `grep -l 'Preparing to rebalance group mirror_maker_group' *`; do echo -n "$log "; grep 'Preparing to rebalance group mirror_maker_group' $log | wc -l; done server.2021-03-02-10.kafka3 5 server.2021-03-02-11.kafka3 1 server.2021-03-02-12.kafka3 3 server.2021-03-02-10.kafka2 1 server.2021-03-02-11.kafka2 306 server.2021-03-02-12.kafka2 452 server.2021-03-02-13.kafka2 1 # Here we rebooted broker(2) {noformat} The questions: * Is it a bug or topicless broker should be involved in coordinating consumer group? * If it's a bug, does this fixed in any later kafka release? Thank you. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-12405) Broker without active topics still involved in coordinating consumer group
[ https://issues.apache.org/jira/browse/KAFKA-12405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander updated KAFKA-12405: -- Description: Short description: Broker without active topics and with inconsistent data still involved in coordinating consumer group. It leads to constant rebalancing consumer group Details: We have kafka cluster(kafka 2.3.0) of three nodes(1,2,3) * After high disk load broker(2) came out of cluster * Brokers (1), (3) shortly became leaders of it's topics * Broker(2) back to cluster, but have failed to became leader of it's preferred topics due to "Out of sync replicas". Now it start spamming with "Shrinking ISR from 3,2,1 to 2, ... Out of sync replicas" "Cached zkVersion [20] not equal to that in zookeeper, skip updating ISR" Like in https://issues.apache.org/jira/browse/KAFKA-3042 But at the same time we start to observing constant rebalancing of consumer group "mirror_maker_group" and topic reading within this group was completely stucked. Restarting broker(2) fix the problem Broker(2) server log: {noformat} [executor-Produce] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Preparing to rebalance group mirror_maker_group in state PreparingRebalance with old generation 7139 (__consumer_offsets-30) (reason: error when storing group assignment during SyncGroup)" [data-plane-kafka-request-handler-6] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Stabilized group mirror_maker_group generation 7140 (__consumer_offsets-30) [data-plane-kafka-request-handler-7] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Assignment received from leader for group mirror_maker_group for generation 7140{noformat} The number of log entry "Rebalancing group..." per hour {noformat} $ for log in `grep -l 'Preparing to rebalance group mirror_maker_group' *`; do echo -n "$log "; grep 'Preparing to rebalance group mirror_maker_group' $log | wc -l; done server.2021-03-02-10.kafka3 5 server.2021-03-02-11.kafka3 1 server.2021-03-02-12.kafka3 3 server.2021-03-02-10.kafka2 1 server.2021-03-02-11.kafka2 306 server.2021-03-02-12.kafka2 452 server.2021-03-02-13.kafka2 1 # Here we rebooted broker(2) {noformat} The questions: * Is it a bug or topicless broker should be involved in coordinating consumer group? * If it's a bug, does this fixed in any later kafka release? Thank you. was: Short description: Broker without active topics and with inconsistent data still involved in coordinating consumer group. It leads to constant rebalancing consumer group Details: We have kafka cluster(kafka 2.3.0) of three nodes(1,2,3) * After high disk load broker(2) came out of cluster * Brokers (1), (3) shortly became leaders of it's topics * Broker(2) back to cluster, but have failed to became leader of it's preferred topics due to "Out of sync replicas". Now it start spamming with "Shrinking ISR from 3,2,1 to 2, ... Out of sync replicas" "Cached zkVersion [20] not equal to that in zookeeper, skip updating ISR" Like in https://issues.apache.org/jira/browse/KAFKA-3042 But at the same time we start to observing constant rebalancing of consumer group "mirror_maker_group" and topic reading within this group was completely stucked. Restarting broker(2) fix the problem Broker(2) server log: {noformat} [executor-Produce] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Preparing to rebalance group mirror_maker_group in state PreparingRebalance with old generation 7139 (__consumer_offsets-30) (reason: error when storing group assignment during SyncGroup)" [data-plane-kafka-request-handler-6] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Stabilized group mirror_maker_group generation 7140 (__consumer_offsets-30) [data-plane-kafka-request-handler-7] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Assignment received from leader for group mirror_maker_group for generation 7140{noformat} {noformat} $ for log in `grep -l 'Preparing to rebalance group mirror_maker_group' *`; do echo -n "$log "; grep 'Preparing to rebalance group mirror_maker_group' $log | wc -l; done server.2021-03-02-10.kafka3 5 server.2021-03-02-11.kafka3 1 server.2021-03-02-12.kafka3 3 server.2021-03-02-10.kafka2 1 server.2021-03-02-11.kafka2 306 server.2021-03-02-12.kafka2 452 server.2021-03-02-13.kafka2 1 # Here we rebooted broker(2) {noformat} The questions: * Is it a bug or topicless broker should be involved in coordinating consumer group? * If it's a bug, does this fixed in any later kafka release? Thank you. > Broker without active topics still involved in coordinating consumer group > -- > > Key: KAFKA-12405 > URL: https://issues.apache.org/jira/browse/KAFKA-12405 > Proje
[jira] [Updated] (KAFKA-12405) Inconsistent broker without active topics still involved in coordinating consumer group
[ https://issues.apache.org/jira/browse/KAFKA-12405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander updated KAFKA-12405: -- Summary: Inconsistent broker without active topics still involved in coordinating consumer group (was: Broker without active topics still involved in coordinating consumer group) > Inconsistent broker without active topics still involved in coordinating > consumer group > --- > > Key: KAFKA-12405 > URL: https://issues.apache.org/jira/browse/KAFKA-12405 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 2.3.0 >Reporter: Alexander >Priority: Major > > Short description: > Broker without active topics and with inconsistent data still involved in > coordinating consumer group. It leads to constant rebalancing consumer group > > Details: > We have kafka cluster(kafka 2.3.0) of three nodes(1,2,3) > * After high disk load broker(2) came out of cluster > * Brokers (1), (3) shortly became leaders of it's topics > * Broker(2) back to cluster, but have failed to became leader of it's > preferred topics due to > "Out of sync replicas". Now it start spamming with > "Shrinking ISR from 3,2,1 to 2, ... Out of sync replicas" > "Cached zkVersion [20] not equal to that in zookeeper, skip updating ISR" > Like in https://issues.apache.org/jira/browse/KAFKA-3042 > But at the same time we start to observing constant rebalancing of consumer > group "mirror_maker_group" and topic reading within this group was completely > stucked. Restarting broker(2) fix the problem > Broker(2) server log: > {noformat} > [executor-Produce] [kafka.coordinator.group.GroupCoordinator] INFO > [GroupCoordinator 2]: Preparing to rebalance group mirror_maker_group in > state PreparingRebalance with old generation 7139 (__consumer_offsets-30) > (reason: error when storing group assignment during SyncGroup)" > [data-plane-kafka-request-handler-6] > [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: > Stabilized group mirror_maker_group generation 7140 (__consumer_offsets-30) > [data-plane-kafka-request-handler-7] > [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: > Assignment received from leader for group mirror_maker_group for generation > 7140{noformat} > The number of log entry "Rebalancing group..." per hour > {noformat} > $ for log in `grep -l 'Preparing to rebalance group mirror_maker_group' *`; > do echo -n "$log "; grep 'Preparing to rebalance group mirror_maker_group' > $log | wc -l; done > server.2021-03-02-10.kafka3 5 > server.2021-03-02-11.kafka3 1 > server.2021-03-02-12.kafka3 3 > server.2021-03-02-10.kafka2 1 > server.2021-03-02-11.kafka2 306 > server.2021-03-02-12.kafka2 452 > server.2021-03-02-13.kafka2 1 # Here we rebooted broker(2) > {noformat} > > The questions: > * Is it a bug or topicless broker should be involved in coordinating > consumer group? > * If it's a bug, does this fixed in any later kafka release? > Thank you. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-12405) Inconsistent broker without active topics still involved in coordinating consumer group
[ https://issues.apache.org/jira/browse/KAFKA-12405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander updated KAFKA-12405: -- Description: Short description: Broker without active topics and with inconsistent data still involved in coordinating consumer group. It leads to constant rebalancing consumer group Details: We have kafka cluster(kafka 2.3.0) of three nodes(1,2,3) * After high disk load broker(2) came out of cluster * Brokers (1), (3) shortly became leaders of it's topics * Broker(2) back to cluster, but have failed to became leader of it's preferred topics due to "Out of sync replicas" * Now it start spamming with "Shrinking ISR from 3,2,1 to 2, ... Out of sync replicas" "Cached zkVersion [20] not equal to that in zookeeper, skip updating ISR" Like in https://issues.apache.org/jira/browse/KAFKA-3042 But at the same time we start to observing constant rebalancing of consumer group "mirror_maker_group" and topic reading within this group was completely stucked. Restarting broker(2) fix the problem Broker(2) server log: {noformat} [executor-Produce] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Preparing to rebalance group mirror_maker_group in state PreparingRebalance with old generation 7139 (__consumer_offsets-30) (reason: error when storing group assignment during SyncGroup)" [data-plane-kafka-request-handler-6] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Stabilized group mirror_maker_group generation 7140 (__consumer_offsets-30) [data-plane-kafka-request-handler-7] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Assignment received from leader for group mirror_maker_group for generation 7140{noformat} The number of log entry "Rebalancing group..." per hour {noformat} $ for log in `grep -l 'Preparing to rebalance group mirror_maker_group' *`; do echo -n "$log "; grep 'Preparing to rebalance group mirror_maker_group' $log | wc -l; done server.2021-03-02-10.kafka3 5 server.2021-03-02-11.kafka3 1 server.2021-03-02-12.kafka3 3 server.2021-03-02-10.kafka2 1 server.2021-03-02-11.kafka2 306 server.2021-03-02-12.kafka2 452 server.2021-03-02-13.kafka2 1 # Here we rebooted broker(2) {noformat} The questions: * Is it a bug or topicless broker should be involved in coordinating consumer group? * If it's a bug, does this fixed in any later kafka release? Thank you. was: Short description: Broker without active topics and with inconsistent data still involved in coordinating consumer group. It leads to constant rebalancing consumer group Details: We have kafka cluster(kafka 2.3.0) of three nodes(1,2,3) * After high disk load broker(2) came out of cluster * Brokers (1), (3) shortly became leaders of it's topics * Broker(2) back to cluster, but have failed to became leader of it's preferred topics due to "Out of sync replicas". Now it start spamming with "Shrinking ISR from 3,2,1 to 2, ... Out of sync replicas" "Cached zkVersion [20] not equal to that in zookeeper, skip updating ISR" Like in https://issues.apache.org/jira/browse/KAFKA-3042 But at the same time we start to observing constant rebalancing of consumer group "mirror_maker_group" and topic reading within this group was completely stucked. Restarting broker(2) fix the problem Broker(2) server log: {noformat} [executor-Produce] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Preparing to rebalance group mirror_maker_group in state PreparingRebalance with old generation 7139 (__consumer_offsets-30) (reason: error when storing group assignment during SyncGroup)" [data-plane-kafka-request-handler-6] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Stabilized group mirror_maker_group generation 7140 (__consumer_offsets-30) [data-plane-kafka-request-handler-7] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Assignment received from leader for group mirror_maker_group for generation 7140{noformat} The number of log entry "Rebalancing group..." per hour {noformat} $ for log in `grep -l 'Preparing to rebalance group mirror_maker_group' *`; do echo -n "$log "; grep 'Preparing to rebalance group mirror_maker_group' $log | wc -l; done server.2021-03-02-10.kafka3 5 server.2021-03-02-11.kafka3 1 server.2021-03-02-12.kafka3 3 server.2021-03-02-10.kafka2 1 server.2021-03-02-11.kafka2 306 server.2021-03-02-12.kafka2 452 server.2021-03-02-13.kafka2 1 # Here we rebooted broker(2) {noformat} The questions: * Is it a bug or topicless broker should be involved in coordinating consumer group? * If it's a bug, does this fixed in any later kafka release? Thank you. > Inconsistent broker without active topics still involved in coordinating > consumer group > --- > > Key: KAFKA-12405 >
[jira] [Updated] (KAFKA-12405) Inconsistent broker without active topics still involved in coordinating consumer group
[ https://issues.apache.org/jira/browse/KAFKA-12405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander updated KAFKA-12405: -- Description: Short description: Broker without active topics and with inconsistent data still involved in coordinating consumer group. It leads to constant rebalancing consumer group Details: We have kafka cluster(kafka 2.3.0) of three nodes(1,2,3) * After high disk load broker(2) came out of cluster * Brokers (1), (3) shortly became leaders of it's topics * Broker(2) back to cluster, but have failed to became leader of it's preferred topics due to "Out of sync replicas" * Now it start spamming with "Shrinking ISR from 3,2,1 to 2, ... Out of sync replicas" "Cached zkVersion [20] not equal to that in zookeeper, skip updating ISR" Like in https://issues.apache.org/jira/browse/KAFKA-3042 But at the same time we start to observing constant rebalancing of consumer group "mirror_maker_group" and topic reading within this group was completely stucked. Restarting broker(2) fix the problem Broker(2) server log: {noformat} [executor-Produce] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Preparing to rebalance group mirror_maker_group in state PreparingRebalance with old generation 7139 (__consumer_offsets-30) (reason: error when storing group assignment during SyncGroup)" [data-plane-kafka-request-handler-6] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Stabilized group mirror_maker_group generation 7140 (__consumer_offsets-30) [data-plane-kafka-request-handler-7] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Assignment received from leader for group mirror_maker_group for generation 7140{noformat} The number of log entry "Rebalancing group..." per hour {noformat} $ for log in `grep -l 'Preparing to rebalance group mirror_maker_group' *`; do echo -n "$log "; grep 'Preparing to rebalance group mirror_maker_group' $log | wc -l; done server.2021-03-02-10.kafka3 5 server.2021-03-02-11.kafka3 1 server.2021-03-02-12.kafka3 3 server.2021-03-02-10.kafka2 1 server.2021-03-02-11.kafka2 306 server.2021-03-02-12.kafka2 452 server.2021-03-02-13.kafka2 1 # Here we rebooted broker(2) {noformat} The questions: * Is it a bug or topicless broker should be involved in coordinating consumer group? * If it's a bug, was this fixed in any later kafka release? Thank you. was: Short description: Broker without active topics and with inconsistent data still involved in coordinating consumer group. It leads to constant rebalancing consumer group Details: We have kafka cluster(kafka 2.3.0) of three nodes(1,2,3) * After high disk load broker(2) came out of cluster * Brokers (1), (3) shortly became leaders of it's topics * Broker(2) back to cluster, but have failed to became leader of it's preferred topics due to "Out of sync replicas" * Now it start spamming with "Shrinking ISR from 3,2,1 to 2, ... Out of sync replicas" "Cached zkVersion [20] not equal to that in zookeeper, skip updating ISR" Like in https://issues.apache.org/jira/browse/KAFKA-3042 But at the same time we start to observing constant rebalancing of consumer group "mirror_maker_group" and topic reading within this group was completely stucked. Restarting broker(2) fix the problem Broker(2) server log: {noformat} [executor-Produce] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Preparing to rebalance group mirror_maker_group in state PreparingRebalance with old generation 7139 (__consumer_offsets-30) (reason: error when storing group assignment during SyncGroup)" [data-plane-kafka-request-handler-6] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Stabilized group mirror_maker_group generation 7140 (__consumer_offsets-30) [data-plane-kafka-request-handler-7] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Assignment received from leader for group mirror_maker_group for generation 7140{noformat} The number of log entry "Rebalancing group..." per hour {noformat} $ for log in `grep -l 'Preparing to rebalance group mirror_maker_group' *`; do echo -n "$log "; grep 'Preparing to rebalance group mirror_maker_group' $log | wc -l; done server.2021-03-02-10.kafka3 5 server.2021-03-02-11.kafka3 1 server.2021-03-02-12.kafka3 3 server.2021-03-02-10.kafka2 1 server.2021-03-02-11.kafka2 306 server.2021-03-02-12.kafka2 452 server.2021-03-02-13.kafka2 1 # Here we rebooted broker(2) {noformat} The questions: * Is it a bug or topicless broker should be involved in coordinating consumer group? * If it's a bug, does this fixed in any later kafka release? Thank you. > Inconsistent broker without active topics still involved in coordinating > consumer group > --- > > Key: KAFKA-12405 >
[jira] [Updated] (KAFKA-12405) Inconsistent broker without active topics still involved in coordinating consumer group
[ https://issues.apache.org/jira/browse/KAFKA-12405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander updated KAFKA-12405: -- Description: Short description: Broker without active topics and with inconsistent data still involved in coordinating consumer group. It leads to constant rebalancing consumer group Details: We have kafka cluster(kafka 2.3.0) of three nodes(1,2,3) * After high disk load broker(2) came out of cluster * Brokers (1), (3) shortly became leaders of it's topics * Broker(2) back to cluster, but have failed to became leader of it's preferred topics due to "Out of sync replicas" * Now it start spamming with "Shrinking ISR from 3,2,1 to 2, ... Out of sync replicas" "Cached zkVersion [20] not equal to that in zookeeper, skip updating ISR" Like in https://issues.apache.org/jira/browse/KAFKA-3042 But at the same time we start to observing constant rebalancing of consumer group "mirror_maker_group" and topic reading within this group was completely stucked. Restarting broker(2) fix the problem Broker(2) server log: {noformat} 2021-03-02 11:34:38,791 + [executor-Produce] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Preparing to rebalance group mirror_maker_group in state PreparingRebalance with old generation 7139 (__consumer_offsets-30) (reason: error when storing group assignment during SyncGroup (member: mirror_maker_group-0-0668b084-bef9-46f6-aec2-07807f1634a0)) (kafka.coordinator.group.GroupCoordinator) 2021-03-02 11:34:38,837 + [data-plane-kafka-request-handler-6] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Stabilized group mirror_maker_group generation 7140 (__consumer_offsets-30) (kafka.coordinator.group.GroupCoordinator) 2021-03-02 11:34:38,885 + [data-plane-kafka-request-handler-7] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Assignment received from leader for group mirror_maker_group for generation 7140 (kafka.coordinator.group.GroupCoordinator){noformat} The number of log entry "Rebalancing group..." per hour {noformat} $ for log in `grep -l 'Preparing to rebalance group mirror_maker_group' *`; do echo -n "$log "; grep 'Preparing to rebalance group mirror_maker_group' $log | wc -l; done server.2021-03-02-10.kafka3 5 server.2021-03-02-11.kafka3 1 server.2021-03-02-12.kafka3 3 server.2021-03-02-10.kafka2 1 server.2021-03-02-11.kafka2 306 server.2021-03-02-12.kafka2 452 server.2021-03-02-13.kafka2 1 # Here we rebooted broker(2) {noformat} The questions: * Is it a bug or topicless broker should be involved in coordinating consumer group? * If it's a bug, was this fixed in any later kafka release? Thank you. was: Short description: Broker without active topics and with inconsistent data still involved in coordinating consumer group. It leads to constant rebalancing consumer group Details: We have kafka cluster(kafka 2.3.0) of three nodes(1,2,3) * After high disk load broker(2) came out of cluster * Brokers (1), (3) shortly became leaders of it's topics * Broker(2) back to cluster, but have failed to became leader of it's preferred topics due to "Out of sync replicas" * Now it start spamming with "Shrinking ISR from 3,2,1 to 2, ... Out of sync replicas" "Cached zkVersion [20] not equal to that in zookeeper, skip updating ISR" Like in https://issues.apache.org/jira/browse/KAFKA-3042 But at the same time we start to observing constant rebalancing of consumer group "mirror_maker_group" and topic reading within this group was completely stucked. Restarting broker(2) fix the problem Broker(2) server log: {noformat} [executor-Produce] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Preparing to rebalance group mirror_maker_group in state PreparingRebalance with old generation 7139 (__consumer_offsets-30) (reason: error when storing group assignment during SyncGroup)" [data-plane-kafka-request-handler-6] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Stabilized group mirror_maker_group generation 7140 (__consumer_offsets-30) [data-plane-kafka-request-handler-7] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Assignment received from leader for group mirror_maker_group for generation 7140{noformat} The number of log entry "Rebalancing group..." per hour {noformat} $ for log in `grep -l 'Preparing to rebalance group mirror_maker_group' *`; do echo -n "$log "; grep 'Preparing to rebalance group mirror_maker_group' $log | wc -l; done server.2021-03-02-10.kafka3 5 server.2021-03-02-11.kafka3 1 server.2021-03-02-12.kafka3 3 server.2021-03-02-10.kafka2 1 server.2021-03-02-11.kafka2 306 server.2021-03-02-12.kafka2 452 server.2021-03-02-13.kafka2 1 # Here we rebooted broker(2) {noformat} The questions: * Is it a bug or topicless broker should be involved in coordinating consumer group? * If it's a bug,
[jira] [Updated] (KAFKA-12405) Inconsistent broker without active topics still involved in coordinating consumer group
[ https://issues.apache.org/jira/browse/KAFKA-12405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander updated KAFKA-12405: -- Description: Short description: Broker without active topics and with inconsistent data still involved in coordinating consumer group. It leads to constant rebalancing consumer group Details: We have kafka cluster(kafka 2.3.0) of three nodes(1,2,3) * After high disk load broker(2) came out of cluster * Brokers (1), (3) shortly became leaders of it's topics * Broker(2) back to cluster, but have failed to became leader of it's preferred topics due to "Out of sync replicas" * Now it start spamming with "Shrinking ISR from 3,2,1 to 2, ... Out of sync replicas" "Cached zkVersion [20] not equal to that in zookeeper, skip updating ISR" Like in https://issues.apache.org/jira/browse/KAFKA-3042 But at the same time we start to observing constant rebalancing of consumer group "mirror_maker_group" and topic reading within this group was completely stucked. Restarting broker(2) fix the problem Broker(2) server log: {noformat} 2021-03-02 11:34:38,791 + [executor-Produce] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Preparing to rebalance group mirror_maker_group in state PreparingRebalance with old generation 7139 (__consumer_offsets-30) (reason: error when storing group assignment during SyncGroup (member: mirror_maker_group-0-0668b084-bef9-46f6-aec2-07807f1634a0)) (kafka.coordinator.group.GroupCoordinator) 2021-03-02 11:34:38,837 + [data-plane-kafka-request-handler-6] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Stabilized group mirror_maker_group generation 7140 (__consumer_offsets-30) (kafka.coordinator.group.GroupCoordinator) 2021-03-02 11:34:38,885 + [data-plane-kafka-request-handler-7] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Assignment received from leader for group mirror_maker_group for generation 7140 (kafka.coordinator.group.GroupCoordinator){noformat} The number of log entry "Rebalancing group..." per hour {noformat} $ for log in `grep -l 'Preparing to rebalance group mirror_maker_group' *`; do echo -n "$log "; grep 'Preparing to rebalance group mirror_maker_group' $log | wc -l; done server.2021-03-02-10.kafka3 5 # kafka3 - fine node with actvice controller server.2021-03-02-11.kafka3 1 server.2021-03-02-12.kafka3 3 server.2021-03-02-10.kafka2 1 # kafka2 - node with inconsistent broker server.2021-03-02-11.kafka2 306 server.2021-03-02-12.kafka2 452 server.2021-03-02-13.kafka2 1 # Here we rebooted broker(2) {noformat} The questions: * Is it a bug or topicless broker should be involved in coordinating consumer group? * If it's a bug, was this fixed in any later kafka release? Thank you. was: Short description: Broker without active topics and with inconsistent data still involved in coordinating consumer group. It leads to constant rebalancing consumer group Details: We have kafka cluster(kafka 2.3.0) of three nodes(1,2,3) * After high disk load broker(2) came out of cluster * Brokers (1), (3) shortly became leaders of it's topics * Broker(2) back to cluster, but have failed to became leader of it's preferred topics due to "Out of sync replicas" * Now it start spamming with "Shrinking ISR from 3,2,1 to 2, ... Out of sync replicas" "Cached zkVersion [20] not equal to that in zookeeper, skip updating ISR" Like in https://issues.apache.org/jira/browse/KAFKA-3042 But at the same time we start to observing constant rebalancing of consumer group "mirror_maker_group" and topic reading within this group was completely stucked. Restarting broker(2) fix the problem Broker(2) server log: {noformat} 2021-03-02 11:34:38,791 + [executor-Produce] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Preparing to rebalance group mirror_maker_group in state PreparingRebalance with old generation 7139 (__consumer_offsets-30) (reason: error when storing group assignment during SyncGroup (member: mirror_maker_group-0-0668b084-bef9-46f6-aec2-07807f1634a0)) (kafka.coordinator.group.GroupCoordinator) 2021-03-02 11:34:38,837 + [data-plane-kafka-request-handler-6] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Stabilized group mirror_maker_group generation 7140 (__consumer_offsets-30) (kafka.coordinator.group.GroupCoordinator) 2021-03-02 11:34:38,885 + [data-plane-kafka-request-handler-7] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Assignment received from leader for group mirror_maker_group for generation 7140 (kafka.coordinator.group.GroupCoordinator){noformat} The number of log entry "Rebalancing group..." per hour {noformat} $ for log in `grep -l 'Preparing to rebalance group mirror_maker_group' *`; do echo -n "$log "; grep 'Preparing to rebalance group mirror_maker_group' $log | wc -l; done
[jira] [Updated] (KAFKA-12405) Inconsistent broker without active topics still involved in coordinating consumer group
[ https://issues.apache.org/jira/browse/KAFKA-12405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander updated KAFKA-12405: -- Description: Short description: Broker without active topics and with inconsistent data still involved in coordinating consumer group. It leads to constant rebalancing consumer group Details: We have kafka cluster(kafka 2.3.0) of three nodes(1,2,3) All topics with same configuration: Topic:xxx.metrics PartitionCount:1 ReplicationFactor:3 Configs: Topic: xxx.metrics Partition: 0 Leader: 2 Replicas: 2,3,1 Isr: 3,1,2 * After high disk load broker(2) came out of cluster * Brokers (1), (3) shortly became leaders of it's topics * Broker(2) back to cluster, but have failed to became leader of it's preferred topics due to "Out of sync replicas" * Now it start spamming with "Shrinking ISR from 3,2,1 to 2, ... Out of sync replicas" "Cached zkVersion [20] not equal to that in zookeeper, skip updating ISR" Like in https://issues.apache.org/jira/browse/KAFKA-3042 But at the same time we start to observing constant rebalancing of consumer group "mirror_maker_group" and topic reading within this group was completely stucked. Restarting broker(2) fix the problem Broker(2) server log: {noformat} 2021-03-02 11:34:38,791 + [executor-Produce] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Preparing to rebalance group mirror_maker_group in state PreparingRebalance with old generation 7139 (__consumer_offsets-30) (reason: error when storing group assignment during SyncGroup (member: mirror_maker_group-0-0668b084-bef9-46f6-aec2-07807f1634a0)) (kafka.coordinator.group.GroupCoordinator) 2021-03-02 11:34:38,837 + [data-plane-kafka-request-handler-6] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Stabilized group mirror_maker_group generation 7140 (__consumer_offsets-30) (kafka.coordinator.group.GroupCoordinator) 2021-03-02 11:34:38,885 + [data-plane-kafka-request-handler-7] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Assignment received from leader for group mirror_maker_group for generation 7140 (kafka.coordinator.group.GroupCoordinator){noformat} The number of log entry "Rebalancing group..." per hour {noformat} $ for log in `grep -l 'Preparing to rebalance group mirror_maker_group' *`; do echo -n "$log "; grep 'Preparing to rebalance group mirror_maker_group' $log | wc -l; done server.2021-03-02-10.kafka3 5 # kafka3 - fine node with actvice controller server.2021-03-02-11.kafka3 1 server.2021-03-02-12.kafka3 3 server.2021-03-02-10.kafka2 1 # kafka2 - node with inconsistent broker server.2021-03-02-11.kafka2 306 server.2021-03-02-12.kafka2 452 server.2021-03-02-13.kafka2 1 # Here we rebooted broker(2) {noformat} The questions: * Is it a bug or topicless broker should be involved in coordinating consumer group? * If it's a bug, was this fixed in any later kafka release? Thank you. was: Short description: Broker without active topics and with inconsistent data still involved in coordinating consumer group. It leads to constant rebalancing consumer group Details: We have kafka cluster(kafka 2.3.0) of three nodes(1,2,3) * After high disk load broker(2) came out of cluster * Brokers (1), (3) shortly became leaders of it's topics * Broker(2) back to cluster, but have failed to became leader of it's preferred topics due to "Out of sync replicas" * Now it start spamming with "Shrinking ISR from 3,2,1 to 2, ... Out of sync replicas" "Cached zkVersion [20] not equal to that in zookeeper, skip updating ISR" Like in https://issues.apache.org/jira/browse/KAFKA-3042 But at the same time we start to observing constant rebalancing of consumer group "mirror_maker_group" and topic reading within this group was completely stucked. Restarting broker(2) fix the problem Broker(2) server log: {noformat} 2021-03-02 11:34:38,791 + [executor-Produce] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Preparing to rebalance group mirror_maker_group in state PreparingRebalance with old generation 7139 (__consumer_offsets-30) (reason: error when storing group assignment during SyncGroup (member: mirror_maker_group-0-0668b084-bef9-46f6-aec2-07807f1634a0)) (kafka.coordinator.group.GroupCoordinator) 2021-03-02 11:34:38,837 + [data-plane-kafka-request-handler-6] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Stabilized group mirror_maker_group generation 7140 (__consumer_offsets-30) (kafka.coordinator.group.GroupCoordinator) 2021-03-02 11:34:38,885 + [data-plane-kafka-request-handler-7] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Assignment received from leader for group mirror_maker_group for generation 7140 (kafka.coordinator.group.GroupCoordinator){noformat} The number of log entry "Rebalancing group..." per hour {noformat}