[jira] [Created] (KAFKA-13384) FailedPartitionsCount metric is not updated if a partition log file was corrupted
Alexander created KAFKA-13384: - Summary: FailedPartitionsCount metric is not updated if a partition log file was corrupted Key: KAFKA-13384 URL: https://issues.apache.org/jira/browse/KAFKA-13384 Project: Kafka Issue Type: Bug Affects Versions: 2.6.2 Environment: OS: NAME="Amazon Linux AMI" VERSION="2018.03" ID="amzn" ID_LIKE="rhel fedora" VERSION_ID="2018.03" PRETTY_NAME="Amazon Linux AMI 2018.03" CPE_NAME="cpe:/o:amazon:linux:2018.03:ga" HOME_URL="http://aws.amazon.com/amazon-linux-ami/"; Kafka version: 2.6.2 (Commit:da65af02e5856e34) Reporter: Alexander Attachments: Screenshot 2021-10-19 at 15.28.33.png We found a misbehavior on our Kafka cluster (version: 2.6.2 (Commit:da65af02e5856e34)), `FailedPartitionsCount` metric is not updated if a partition log file was corrupted Steps to reproduce the problem: 1. corrupt a partition log file 2. restart Kafka process After that, you will get a correct log which tells that Kafka marked corrupted partitions as failed ``` 2021-10-19T14:49:31+02:00 [2021-10-19 12:49:30,924] WARN [ReplicaFetcher replicaId=11, leaderId=10, fetcherId=0] Partition test_topic-1 marked as failed (kafka.server.ReplicaFetcherThread) ``` But the value of `FailedPartitionsCount` metric will be 0 (see attached screenshot) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-13384) FailedPartitionsCount metric is not updated if a partition log file was corrupted
[
https://issues.apache.org/jira/browse/KAFKA-13384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander updated KAFKA-13384:
--
Description:
We found a misbehavior on our Kafka cluster (version: 2.6.2
(Commit:da65af02e5856e34)), `FailedPartitionsCount` metric is not updated if a
partition log file was corrupted
Steps to reproduce the problem:
# corrupt a partition log file
# restart Kafka process
After that, you will get a correct log which tells that Kafka marked corrupted
partitions as failed
{code:java}
2021-10-19T14:49:31+02:00 [2021-10-19 12:49:30,924] WARN [ReplicaFetcher
replicaId=11, leaderId=10, fetcherId=0] Partition test_topic-1 marked as failed
(kafka.server.ReplicaFetcherThread){code}
But the value of `FailedPartitionsCount` metric will be 0 (see attached
screenshot)
was:
We found a misbehavior on our Kafka cluster (version: 2.6.2
(Commit:da65af02e5856e34)), `FailedPartitionsCount` metric is not updated if a
partition log file was corrupted
Steps to reproduce the problem:
1. corrupt a partition log file
2. restart Kafka process
After that, you will get a correct log which tells that Kafka marked corrupted
partitions as failed
```
2021-10-19T14:49:31+02:00 [2021-10-19 12:49:30,924] WARN [ReplicaFetcher
replicaId=11, leaderId=10, fetcherId=0] Partition test_topic-1 marked as failed
(kafka.server.ReplicaFetcherThread)
```
But the value of `FailedPartitionsCount` metric will be 0 (see attached
screenshot)
> FailedPartitionsCount metric is not updated if a partition log file was
> corrupted
> -
>
> Key: KAFKA-13384
> URL: https://issues.apache.org/jira/browse/KAFKA-13384
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 2.6.2
> Environment: OS:
> NAME="Amazon Linux AMI"
> VERSION="2018.03"
> ID="amzn"
> ID_LIKE="rhel fedora"
> VERSION_ID="2018.03"
> PRETTY_NAME="Amazon Linux AMI 2018.03"
> CPE_NAME="cpe:/o:amazon:linux:2018.03:ga"
> HOME_URL="http://aws.amazon.com/amazon-linux-ami/";
> Kafka version:
> 2.6.2 (Commit:da65af02e5856e34)
>Reporter: Alexander
>Priority: Major
> Attachments: Screenshot 2021-10-19 at 15.28.33.png
>
>
> We found a misbehavior on our Kafka cluster (version: 2.6.2
> (Commit:da65af02e5856e34)), `FailedPartitionsCount` metric is not updated if
> a partition log file was corrupted
> Steps to reproduce the problem:
> # corrupt a partition log file
> # restart Kafka process
> After that, you will get a correct log which tells that Kafka marked
> corrupted partitions as failed
>
> {code:java}
> 2021-10-19T14:49:31+02:00 [2021-10-19 12:49:30,924] WARN [ReplicaFetcher
> replicaId=11, leaderId=10, fetcherId=0] Partition test_topic-1 marked as
> failed
> (kafka.server.ReplicaFetcherThread){code}
>
>
> But the value of `FailedPartitionsCount` metric will be 0 (see attached
> screenshot)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-13384) FailedPartitionsCount metric is not updated if a partition log file was corrupted
[
https://issues.apache.org/jira/browse/KAFKA-13384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander updated KAFKA-13384:
--
Description:
We found a misbehavior on our Kafka cluster (version: 2.6.2
(Commit:da65af02e5856e34)), `FailedPartitionsCount` metric is not updated if a
partition log file was corrupted
Steps to reproduce the problem:
# corrupt a partition log file
# restart Kafka process
After that, you will get a correct log which tells that Kafka marked corrupted
partitions as failed
{code:java}
2021-10-19T14:49:31+02:00 [2021-10-19 12:49:30,924] WARN [ReplicaFetcher
replicaId=11, leaderId=10, fetcherId=0] Partition test_topic-1 marked as failed
(kafka.server.ReplicaFetcherThread){code}
But the value of `FailedPartitionsCount` metric will be 0 (see attached
screenshot)
was:
We found a misbehavior on our Kafka cluster (version: 2.6.2
(Commit:da65af02e5856e34)), `FailedPartitionsCount` metric is not updated if a
partition log file was corrupted
Steps to reproduce the problem:
# corrupt a partition log file
# restart Kafka process
After that, you will get a correct log which tells that Kafka marked corrupted
partitions as failed
{code:java}
2021-10-19T14:49:31+02:00 [2021-10-19 12:49:30,924] WARN [ReplicaFetcher
replicaId=11, leaderId=10, fetcherId=0] Partition test_topic-1 marked as failed
(kafka.server.ReplicaFetcherThread){code}
But the value of `FailedPartitionsCount` metric will be 0 (see attached
screenshot)
> FailedPartitionsCount metric is not updated if a partition log file was
> corrupted
> -
>
> Key: KAFKA-13384
> URL: https://issues.apache.org/jira/browse/KAFKA-13384
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 2.6.2
> Environment: OS:
> NAME="Amazon Linux AMI"
> VERSION="2018.03"
> ID="amzn"
> ID_LIKE="rhel fedora"
> VERSION_ID="2018.03"
> PRETTY_NAME="Amazon Linux AMI 2018.03"
> CPE_NAME="cpe:/o:amazon:linux:2018.03:ga"
> HOME_URL="http://aws.amazon.com/amazon-linux-ami/";
> Kafka version:
> 2.6.2 (Commit:da65af02e5856e34)
>Reporter: Alexander
>Priority: Major
> Attachments: Screenshot 2021-10-19 at 15.28.33.png
>
>
> We found a misbehavior on our Kafka cluster (version: 2.6.2
> (Commit:da65af02e5856e34)), `FailedPartitionsCount` metric is not updated if
> a partition log file was corrupted
> Steps to reproduce the problem:
> # corrupt a partition log file
> # restart Kafka process
> After that, you will get a correct log which tells that Kafka marked
> corrupted partitions as failed
>
> {code:java}
> 2021-10-19T14:49:31+02:00 [2021-10-19 12:49:30,924] WARN [ReplicaFetcher
> replicaId=11, leaderId=10, fetcherId=0] Partition test_topic-1 marked as
> failed
> (kafka.server.ReplicaFetcherThread){code}
>
> But the value of `FailedPartitionsCount` metric will be 0 (see attached
> screenshot)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-13384) FailedPartitionsCount metric is not updated if a partition log file was corrupted
[
https://issues.apache.org/jira/browse/KAFKA-13384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander updated KAFKA-13384:
--
Description:
We found a misbehavior on our Kafka cluster - version: 2.6.2
(Commit:da65af02e5856e34)) _*FailedPartitionsCount*_ metric is not updated if a
partition log file was corrupted
Steps to reproduce the problem:
# corrupt a partition log file
# restart Kafka process
After that, you will get a correct log which tells that Kafka marked corrupted
partitions as failed
{code:java}
2021-10-19T14:49:31+02:00 [2021-10-19 12:49:30,924] WARN [ReplicaFetcher
replicaId=11, leaderId=10, fetcherId=0] Partition test_topic-1 marked as failed
(kafka.server.ReplicaFetcherThread){code}
But the value of _*FailedPartitionsCount*_ metric will be 0 (see attached
screenshot)
was:
We found a misbehavior on our Kafka cluster (version: 2.6.2
(Commit:da65af02e5856e34)), `FailedPartitionsCount` metric is not updated if a
partition log file was corrupted
Steps to reproduce the problem:
# corrupt a partition log file
# restart Kafka process
After that, you will get a correct log which tells that Kafka marked corrupted
partitions as failed
{code:java}
2021-10-19T14:49:31+02:00 [2021-10-19 12:49:30,924] WARN [ReplicaFetcher
replicaId=11, leaderId=10, fetcherId=0] Partition test_topic-1 marked as failed
(kafka.server.ReplicaFetcherThread){code}
But the value of `FailedPartitionsCount` metric will be 0 (see attached
screenshot)
> FailedPartitionsCount metric is not updated if a partition log file was
> corrupted
> -
>
> Key: KAFKA-13384
> URL: https://issues.apache.org/jira/browse/KAFKA-13384
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 2.6.2
> Environment: OS:
> NAME="Amazon Linux AMI"
> VERSION="2018.03"
> ID="amzn"
> ID_LIKE="rhel fedora"
> VERSION_ID="2018.03"
> PRETTY_NAME="Amazon Linux AMI 2018.03"
> CPE_NAME="cpe:/o:amazon:linux:2018.03:ga"
> HOME_URL="http://aws.amazon.com/amazon-linux-ami/";
> Kafka version:
> 2.6.2 (Commit:da65af02e5856e34)
>Reporter: Alexander
>Priority: Major
> Attachments: Screenshot 2021-10-19 at 15.28.33.png
>
>
> We found a misbehavior on our Kafka cluster - version: 2.6.2
> (Commit:da65af02e5856e34)) _*FailedPartitionsCount*_ metric is not updated if
> a partition log file was corrupted
> Steps to reproduce the problem:
> # corrupt a partition log file
> # restart Kafka process
> After that, you will get a correct log which tells that Kafka marked
> corrupted partitions as failed
> {code:java}
> 2021-10-19T14:49:31+02:00 [2021-10-19 12:49:30,924] WARN [ReplicaFetcher
> replicaId=11, leaderId=10, fetcherId=0] Partition test_topic-1 marked as
> failed
> (kafka.server.ReplicaFetcherThread){code}
> But the value of _*FailedPartitionsCount*_ metric will be 0 (see attached
> screenshot)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-12405) Broker without active topics still involved in coordinating consumer group
Alexander created KAFKA-12405:
-
Summary: Broker without active topics still involved in
coordinating consumer group
Key: KAFKA-12405
URL: https://issues.apache.org/jira/browse/KAFKA-12405
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 2.3.0
Reporter: Alexander
Short description:
Broker without active topics and with inconsistent data still involved in
coordinating consumer group. It leads to constant rebalancing consumer group
Details:
We have kafka cluster(kafka 2.3.0) of three nodes(1,2,3)
* After high disk load broker(2) came out of cluster
* Brokers (1), (3) shortly became leaders of it's topics
* Broker(2) back to cluster, but have failed to became leader of it's
preferred topics due to
"Out of sync replicas". Now it start spamming with
"Shrinking ISR from 3,2,1 to 2, ... Out of sync replicas"
"Cached zkVersion [20] not equal to that in zookeeper, skip updating ISR"
Like in https://issues.apache.org/jira/browse/KAFKA-3042
But at the same time we start to observing constant rebalancing of consumer
group "mirror_maker_group" and topic reading within this group was completely
stucked. Restarting broker(2) fix the problem
Broker(2) server log:
{noformat}
[executor-Produce] [kafka.coordinator.group.GroupCoordinator] INFO
[GroupCoordinator 2]: Preparing to rebalance group mirror_maker_group in state
PreparingRebalance with old generation 7139 (__consumer_offsets-30) (reason:
error when storing group assignment during SyncGroup)"
[data-plane-kafka-request-handler-6] [kafka.coordinator.group.GroupCoordinator]
INFO [GroupCoordinator 2]: Stabilized group mirror_maker_group generation 7140
(__consumer_offsets-30)
[data-plane-kafka-request-handler-7] [kafka.coordinator.group.GroupCoordinator]
INFO [GroupCoordinator 2]: Assignment received from leader for group
mirror_maker_group for generation 7140{noformat}
{noformat}
$ for log in `grep -l 'Preparing to rebalance group mirror_maker_group' *`; do
echo -n "$log "; grep 'Preparing to rebalance group mirror_maker_group' $log |
wc -l; done
server.2021-03-02-10.kafka3 5
server.2021-03-02-11.kafka3 1
server.2021-03-02-12.kafka3 3
server.2021-03-02-10.kafka2 1
server.2021-03-02-11.kafka2 306
server.2021-03-02-12.kafka2 452
server.2021-03-02-13.kafka2 1 # Here we rebooted broker(2)
{noformat}
The questions:
* Is it a bug or topicless broker should be involved in coordinating consumer
group?
* If it's a bug, does this fixed in any later kafka release?
Thank you.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-12405) Broker without active topics still involved in coordinating consumer group
[
https://issues.apache.org/jira/browse/KAFKA-12405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander updated KAFKA-12405:
--
Description:
Short description:
Broker without active topics and with inconsistent data still involved in
coordinating consumer group. It leads to constant rebalancing consumer group
Details:
We have kafka cluster(kafka 2.3.0) of three nodes(1,2,3)
* After high disk load broker(2) came out of cluster
* Brokers (1), (3) shortly became leaders of it's topics
* Broker(2) back to cluster, but have failed to became leader of it's
preferred topics due to
"Out of sync replicas". Now it start spamming with
"Shrinking ISR from 3,2,1 to 2, ... Out of sync replicas"
"Cached zkVersion [20] not equal to that in zookeeper, skip updating ISR"
Like in https://issues.apache.org/jira/browse/KAFKA-3042
But at the same time we start to observing constant rebalancing of consumer
group "mirror_maker_group" and topic reading within this group was completely
stucked. Restarting broker(2) fix the problem
Broker(2) server log:
{noformat}
[executor-Produce] [kafka.coordinator.group.GroupCoordinator] INFO
[GroupCoordinator 2]: Preparing to rebalance group mirror_maker_group in state
PreparingRebalance with old generation 7139 (__consumer_offsets-30) (reason:
error when storing group assignment during SyncGroup)"
[data-plane-kafka-request-handler-6] [kafka.coordinator.group.GroupCoordinator]
INFO [GroupCoordinator 2]: Stabilized group mirror_maker_group generation 7140
(__consumer_offsets-30)
[data-plane-kafka-request-handler-7] [kafka.coordinator.group.GroupCoordinator]
INFO [GroupCoordinator 2]: Assignment received from leader for group
mirror_maker_group for generation 7140{noformat}
The number of log entry "Rebalancing group..." per hour
{noformat}
$ for log in `grep -l 'Preparing to rebalance group mirror_maker_group' *`; do
echo -n "$log "; grep 'Preparing to rebalance group mirror_maker_group' $log |
wc -l; done
server.2021-03-02-10.kafka3 5
server.2021-03-02-11.kafka3 1
server.2021-03-02-12.kafka3 3
server.2021-03-02-10.kafka2 1
server.2021-03-02-11.kafka2 306
server.2021-03-02-12.kafka2 452
server.2021-03-02-13.kafka2 1 # Here we rebooted broker(2)
{noformat}
The questions:
* Is it a bug or topicless broker should be involved in coordinating consumer
group?
* If it's a bug, does this fixed in any later kafka release?
Thank you.
was:
Short description:
Broker without active topics and with inconsistent data still involved in
coordinating consumer group. It leads to constant rebalancing consumer group
Details:
We have kafka cluster(kafka 2.3.0) of three nodes(1,2,3)
* After high disk load broker(2) came out of cluster
* Brokers (1), (3) shortly became leaders of it's topics
* Broker(2) back to cluster, but have failed to became leader of it's
preferred topics due to
"Out of sync replicas". Now it start spamming with
"Shrinking ISR from 3,2,1 to 2, ... Out of sync replicas"
"Cached zkVersion [20] not equal to that in zookeeper, skip updating ISR"
Like in https://issues.apache.org/jira/browse/KAFKA-3042
But at the same time we start to observing constant rebalancing of consumer
group "mirror_maker_group" and topic reading within this group was completely
stucked. Restarting broker(2) fix the problem
Broker(2) server log:
{noformat}
[executor-Produce] [kafka.coordinator.group.GroupCoordinator] INFO
[GroupCoordinator 2]: Preparing to rebalance group mirror_maker_group in state
PreparingRebalance with old generation 7139 (__consumer_offsets-30) (reason:
error when storing group assignment during SyncGroup)"
[data-plane-kafka-request-handler-6] [kafka.coordinator.group.GroupCoordinator]
INFO [GroupCoordinator 2]: Stabilized group mirror_maker_group generation 7140
(__consumer_offsets-30)
[data-plane-kafka-request-handler-7] [kafka.coordinator.group.GroupCoordinator]
INFO [GroupCoordinator 2]: Assignment received from leader for group
mirror_maker_group for generation 7140{noformat}
{noformat}
$ for log in `grep -l 'Preparing to rebalance group mirror_maker_group' *`; do
echo -n "$log "; grep 'Preparing to rebalance group mirror_maker_group' $log |
wc -l; done
server.2021-03-02-10.kafka3 5
server.2021-03-02-11.kafka3 1
server.2021-03-02-12.kafka3 3
server.2021-03-02-10.kafka2 1
server.2021-03-02-11.kafka2 306
server.2021-03-02-12.kafka2 452
server.2021-03-02-13.kafka2 1 # Here we rebooted broker(2)
{noformat}
The questions:
* Is it a bug or topicless broker should be involved in coordinating consumer
group?
* If it's a bug, does this fixed in any later kafka release?
Thank you.
> Broker without active topics still involved in coordinating consumer group
> --
>
> Key: KAFKA-12405
> URL: https://issues.apache.org/jira/browse/KAFKA-12405
> Proje
[jira] [Updated] (KAFKA-12405) Inconsistent broker without active topics still involved in coordinating consumer group
[
https://issues.apache.org/jira/browse/KAFKA-12405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander updated KAFKA-12405:
--
Summary: Inconsistent broker without active topics still involved in
coordinating consumer group (was: Broker without active topics still involved
in coordinating consumer group)
> Inconsistent broker without active topics still involved in coordinating
> consumer group
> ---
>
> Key: KAFKA-12405
> URL: https://issues.apache.org/jira/browse/KAFKA-12405
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 2.3.0
>Reporter: Alexander
>Priority: Major
>
> Short description:
> Broker without active topics and with inconsistent data still involved in
> coordinating consumer group. It leads to constant rebalancing consumer group
>
> Details:
> We have kafka cluster(kafka 2.3.0) of three nodes(1,2,3)
> * After high disk load broker(2) came out of cluster
> * Brokers (1), (3) shortly became leaders of it's topics
> * Broker(2) back to cluster, but have failed to became leader of it's
> preferred topics due to
> "Out of sync replicas". Now it start spamming with
> "Shrinking ISR from 3,2,1 to 2, ... Out of sync replicas"
> "Cached zkVersion [20] not equal to that in zookeeper, skip updating ISR"
> Like in https://issues.apache.org/jira/browse/KAFKA-3042
> But at the same time we start to observing constant rebalancing of consumer
> group "mirror_maker_group" and topic reading within this group was completely
> stucked. Restarting broker(2) fix the problem
> Broker(2) server log:
> {noformat}
> [executor-Produce] [kafka.coordinator.group.GroupCoordinator] INFO
> [GroupCoordinator 2]: Preparing to rebalance group mirror_maker_group in
> state PreparingRebalance with old generation 7139 (__consumer_offsets-30)
> (reason: error when storing group assignment during SyncGroup)"
> [data-plane-kafka-request-handler-6]
> [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]:
> Stabilized group mirror_maker_group generation 7140 (__consumer_offsets-30)
> [data-plane-kafka-request-handler-7]
> [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]:
> Assignment received from leader for group mirror_maker_group for generation
> 7140{noformat}
> The number of log entry "Rebalancing group..." per hour
> {noformat}
> $ for log in `grep -l 'Preparing to rebalance group mirror_maker_group' *`;
> do echo -n "$log "; grep 'Preparing to rebalance group mirror_maker_group'
> $log | wc -l; done
> server.2021-03-02-10.kafka3 5
> server.2021-03-02-11.kafka3 1
> server.2021-03-02-12.kafka3 3
> server.2021-03-02-10.kafka2 1
> server.2021-03-02-11.kafka2 306
> server.2021-03-02-12.kafka2 452
> server.2021-03-02-13.kafka2 1 # Here we rebooted broker(2)
> {noformat}
>
> The questions:
> * Is it a bug or topicless broker should be involved in coordinating
> consumer group?
> * If it's a bug, does this fixed in any later kafka release?
> Thank you.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-12405) Inconsistent broker without active topics still involved in coordinating consumer group
[
https://issues.apache.org/jira/browse/KAFKA-12405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander updated KAFKA-12405:
--
Description:
Short description:
Broker without active topics and with inconsistent data still involved in
coordinating consumer group. It leads to constant rebalancing consumer group
Details:
We have kafka cluster(kafka 2.3.0) of three nodes(1,2,3)
* After high disk load broker(2) came out of cluster
* Brokers (1), (3) shortly became leaders of it's topics
* Broker(2) back to cluster, but have failed to became leader of it's
preferred topics due to "Out of sync replicas"
* Now it start spamming with
"Shrinking ISR from 3,2,1 to 2, ... Out of sync replicas"
"Cached zkVersion [20] not equal to that in zookeeper, skip updating ISR"
Like in https://issues.apache.org/jira/browse/KAFKA-3042
But at the same time we start to observing constant rebalancing of consumer
group "mirror_maker_group" and topic reading within this group was completely
stucked. Restarting broker(2) fix the problem
Broker(2) server log:
{noformat}
[executor-Produce] [kafka.coordinator.group.GroupCoordinator] INFO
[GroupCoordinator 2]: Preparing to rebalance group mirror_maker_group in state
PreparingRebalance with old generation 7139 (__consumer_offsets-30) (reason:
error when storing group assignment during SyncGroup)"
[data-plane-kafka-request-handler-6] [kafka.coordinator.group.GroupCoordinator]
INFO [GroupCoordinator 2]: Stabilized group mirror_maker_group generation 7140
(__consumer_offsets-30)
[data-plane-kafka-request-handler-7] [kafka.coordinator.group.GroupCoordinator]
INFO [GroupCoordinator 2]: Assignment received from leader for group
mirror_maker_group for generation 7140{noformat}
The number of log entry "Rebalancing group..." per hour
{noformat}
$ for log in `grep -l 'Preparing to rebalance group mirror_maker_group' *`; do
echo -n "$log "; grep 'Preparing to rebalance group mirror_maker_group' $log |
wc -l; done
server.2021-03-02-10.kafka3 5
server.2021-03-02-11.kafka3 1
server.2021-03-02-12.kafka3 3
server.2021-03-02-10.kafka2 1
server.2021-03-02-11.kafka2 306
server.2021-03-02-12.kafka2 452
server.2021-03-02-13.kafka2 1 # Here we rebooted broker(2)
{noformat}
The questions:
* Is it a bug or topicless broker should be involved in coordinating consumer
group?
* If it's a bug, does this fixed in any later kafka release?
Thank you.
was:
Short description:
Broker without active topics and with inconsistent data still involved in
coordinating consumer group. It leads to constant rebalancing consumer group
Details:
We have kafka cluster(kafka 2.3.0) of three nodes(1,2,3)
* After high disk load broker(2) came out of cluster
* Brokers (1), (3) shortly became leaders of it's topics
* Broker(2) back to cluster, but have failed to became leader of it's
preferred topics due to
"Out of sync replicas". Now it start spamming with
"Shrinking ISR from 3,2,1 to 2, ... Out of sync replicas"
"Cached zkVersion [20] not equal to that in zookeeper, skip updating ISR"
Like in https://issues.apache.org/jira/browse/KAFKA-3042
But at the same time we start to observing constant rebalancing of consumer
group "mirror_maker_group" and topic reading within this group was completely
stucked. Restarting broker(2) fix the problem
Broker(2) server log:
{noformat}
[executor-Produce] [kafka.coordinator.group.GroupCoordinator] INFO
[GroupCoordinator 2]: Preparing to rebalance group mirror_maker_group in state
PreparingRebalance with old generation 7139 (__consumer_offsets-30) (reason:
error when storing group assignment during SyncGroup)"
[data-plane-kafka-request-handler-6] [kafka.coordinator.group.GroupCoordinator]
INFO [GroupCoordinator 2]: Stabilized group mirror_maker_group generation 7140
(__consumer_offsets-30)
[data-plane-kafka-request-handler-7] [kafka.coordinator.group.GroupCoordinator]
INFO [GroupCoordinator 2]: Assignment received from leader for group
mirror_maker_group for generation 7140{noformat}
The number of log entry "Rebalancing group..." per hour
{noformat}
$ for log in `grep -l 'Preparing to rebalance group mirror_maker_group' *`; do
echo -n "$log "; grep 'Preparing to rebalance group mirror_maker_group' $log |
wc -l; done
server.2021-03-02-10.kafka3 5
server.2021-03-02-11.kafka3 1
server.2021-03-02-12.kafka3 3
server.2021-03-02-10.kafka2 1
server.2021-03-02-11.kafka2 306
server.2021-03-02-12.kafka2 452
server.2021-03-02-13.kafka2 1 # Here we rebooted broker(2)
{noformat}
The questions:
* Is it a bug or topicless broker should be involved in coordinating consumer
group?
* If it's a bug, does this fixed in any later kafka release?
Thank you.
> Inconsistent broker without active topics still involved in coordinating
> consumer group
> ---
>
> Key: KAFKA-12405
>
[jira] [Updated] (KAFKA-12405) Inconsistent broker without active topics still involved in coordinating consumer group
[
https://issues.apache.org/jira/browse/KAFKA-12405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander updated KAFKA-12405:
--
Description:
Short description:
Broker without active topics and with inconsistent data still involved in
coordinating consumer group. It leads to constant rebalancing consumer group
Details:
We have kafka cluster(kafka 2.3.0) of three nodes(1,2,3)
* After high disk load broker(2) came out of cluster
* Brokers (1), (3) shortly became leaders of it's topics
* Broker(2) back to cluster, but have failed to became leader of it's
preferred topics due to "Out of sync replicas"
* Now it start spamming with
"Shrinking ISR from 3,2,1 to 2, ... Out of sync replicas"
"Cached zkVersion [20] not equal to that in zookeeper, skip updating ISR"
Like in https://issues.apache.org/jira/browse/KAFKA-3042
But at the same time we start to observing constant rebalancing of consumer
group "mirror_maker_group" and topic reading within this group was completely
stucked. Restarting broker(2) fix the problem
Broker(2) server log:
{noformat}
[executor-Produce] [kafka.coordinator.group.GroupCoordinator] INFO
[GroupCoordinator 2]: Preparing to rebalance group mirror_maker_group in state
PreparingRebalance with old generation 7139 (__consumer_offsets-30) (reason:
error when storing group assignment during SyncGroup)"
[data-plane-kafka-request-handler-6] [kafka.coordinator.group.GroupCoordinator]
INFO [GroupCoordinator 2]: Stabilized group mirror_maker_group generation 7140
(__consumer_offsets-30)
[data-plane-kafka-request-handler-7] [kafka.coordinator.group.GroupCoordinator]
INFO [GroupCoordinator 2]: Assignment received from leader for group
mirror_maker_group for generation 7140{noformat}
The number of log entry "Rebalancing group..." per hour
{noformat}
$ for log in `grep -l 'Preparing to rebalance group mirror_maker_group' *`; do
echo -n "$log "; grep 'Preparing to rebalance group mirror_maker_group' $log |
wc -l; done
server.2021-03-02-10.kafka3 5
server.2021-03-02-11.kafka3 1
server.2021-03-02-12.kafka3 3
server.2021-03-02-10.kafka2 1
server.2021-03-02-11.kafka2 306
server.2021-03-02-12.kafka2 452
server.2021-03-02-13.kafka2 1 # Here we rebooted broker(2)
{noformat}
The questions:
* Is it a bug or topicless broker should be involved in coordinating consumer
group?
* If it's a bug, was this fixed in any later kafka release?
Thank you.
was:
Short description:
Broker without active topics and with inconsistent data still involved in
coordinating consumer group. It leads to constant rebalancing consumer group
Details:
We have kafka cluster(kafka 2.3.0) of three nodes(1,2,3)
* After high disk load broker(2) came out of cluster
* Brokers (1), (3) shortly became leaders of it's topics
* Broker(2) back to cluster, but have failed to became leader of it's
preferred topics due to "Out of sync replicas"
* Now it start spamming with
"Shrinking ISR from 3,2,1 to 2, ... Out of sync replicas"
"Cached zkVersion [20] not equal to that in zookeeper, skip updating ISR"
Like in https://issues.apache.org/jira/browse/KAFKA-3042
But at the same time we start to observing constant rebalancing of consumer
group "mirror_maker_group" and topic reading within this group was completely
stucked. Restarting broker(2) fix the problem
Broker(2) server log:
{noformat}
[executor-Produce] [kafka.coordinator.group.GroupCoordinator] INFO
[GroupCoordinator 2]: Preparing to rebalance group mirror_maker_group in state
PreparingRebalance with old generation 7139 (__consumer_offsets-30) (reason:
error when storing group assignment during SyncGroup)"
[data-plane-kafka-request-handler-6] [kafka.coordinator.group.GroupCoordinator]
INFO [GroupCoordinator 2]: Stabilized group mirror_maker_group generation 7140
(__consumer_offsets-30)
[data-plane-kafka-request-handler-7] [kafka.coordinator.group.GroupCoordinator]
INFO [GroupCoordinator 2]: Assignment received from leader for group
mirror_maker_group for generation 7140{noformat}
The number of log entry "Rebalancing group..." per hour
{noformat}
$ for log in `grep -l 'Preparing to rebalance group mirror_maker_group' *`; do
echo -n "$log "; grep 'Preparing to rebalance group mirror_maker_group' $log |
wc -l; done
server.2021-03-02-10.kafka3 5
server.2021-03-02-11.kafka3 1
server.2021-03-02-12.kafka3 3
server.2021-03-02-10.kafka2 1
server.2021-03-02-11.kafka2 306
server.2021-03-02-12.kafka2 452
server.2021-03-02-13.kafka2 1 # Here we rebooted broker(2)
{noformat}
The questions:
* Is it a bug or topicless broker should be involved in coordinating consumer
group?
* If it's a bug, does this fixed in any later kafka release?
Thank you.
> Inconsistent broker without active topics still involved in coordinating
> consumer group
> ---
>
> Key: KAFKA-12405
>
[jira] [Updated] (KAFKA-12405) Inconsistent broker without active topics still involved in coordinating consumer group
[
https://issues.apache.org/jira/browse/KAFKA-12405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander updated KAFKA-12405:
--
Description:
Short description:
Broker without active topics and with inconsistent data still involved in
coordinating consumer group. It leads to constant rebalancing consumer group
Details:
We have kafka cluster(kafka 2.3.0) of three nodes(1,2,3)
* After high disk load broker(2) came out of cluster
* Brokers (1), (3) shortly became leaders of it's topics
* Broker(2) back to cluster, but have failed to became leader of it's
preferred topics due to "Out of sync replicas"
* Now it start spamming with
"Shrinking ISR from 3,2,1 to 2, ... Out of sync replicas"
"Cached zkVersion [20] not equal to that in zookeeper, skip updating ISR"
Like in https://issues.apache.org/jira/browse/KAFKA-3042
But at the same time we start to observing constant rebalancing of consumer
group "mirror_maker_group" and topic reading within this group was completely
stucked. Restarting broker(2) fix the problem
Broker(2) server log:
{noformat}
2021-03-02 11:34:38,791 + [executor-Produce]
[kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Preparing
to rebalance group mirror_maker_group in state PreparingRebalance with old
generation 7139 (__consumer_offsets-30) (reason: error when storing group
assignment during SyncGroup (member:
mirror_maker_group-0-0668b084-bef9-46f6-aec2-07807f1634a0))
(kafka.coordinator.group.GroupCoordinator)
2021-03-02 11:34:38,837 + [data-plane-kafka-request-handler-6]
[kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]:
Stabilized group mirror_maker_group generation 7140 (__consumer_offsets-30)
(kafka.coordinator.group.GroupCoordinator)
2021-03-02 11:34:38,885 + [data-plane-kafka-request-handler-7]
[kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]:
Assignment received from leader for group mirror_maker_group for generation
7140 (kafka.coordinator.group.GroupCoordinator){noformat}
The number of log entry "Rebalancing group..." per hour
{noformat}
$ for log in `grep -l 'Preparing to rebalance group mirror_maker_group' *`; do
echo -n "$log "; grep 'Preparing to rebalance group mirror_maker_group' $log |
wc -l; done
server.2021-03-02-10.kafka3 5
server.2021-03-02-11.kafka3 1
server.2021-03-02-12.kafka3 3
server.2021-03-02-10.kafka2 1
server.2021-03-02-11.kafka2 306
server.2021-03-02-12.kafka2 452
server.2021-03-02-13.kafka2 1 # Here we rebooted broker(2)
{noformat}
The questions:
* Is it a bug or topicless broker should be involved in coordinating consumer
group?
* If it's a bug, was this fixed in any later kafka release?
Thank you.
was:
Short description:
Broker without active topics and with inconsistent data still involved in
coordinating consumer group. It leads to constant rebalancing consumer group
Details:
We have kafka cluster(kafka 2.3.0) of three nodes(1,2,3)
* After high disk load broker(2) came out of cluster
* Brokers (1), (3) shortly became leaders of it's topics
* Broker(2) back to cluster, but have failed to became leader of it's
preferred topics due to "Out of sync replicas"
* Now it start spamming with
"Shrinking ISR from 3,2,1 to 2, ... Out of sync replicas"
"Cached zkVersion [20] not equal to that in zookeeper, skip updating ISR"
Like in https://issues.apache.org/jira/browse/KAFKA-3042
But at the same time we start to observing constant rebalancing of consumer
group "mirror_maker_group" and topic reading within this group was completely
stucked. Restarting broker(2) fix the problem
Broker(2) server log:
{noformat}
[executor-Produce] [kafka.coordinator.group.GroupCoordinator] INFO
[GroupCoordinator 2]: Preparing to rebalance group mirror_maker_group in state
PreparingRebalance with old generation 7139 (__consumer_offsets-30) (reason:
error when storing group assignment during SyncGroup)"
[data-plane-kafka-request-handler-6] [kafka.coordinator.group.GroupCoordinator]
INFO [GroupCoordinator 2]: Stabilized group mirror_maker_group generation 7140
(__consumer_offsets-30)
[data-plane-kafka-request-handler-7] [kafka.coordinator.group.GroupCoordinator]
INFO [GroupCoordinator 2]: Assignment received from leader for group
mirror_maker_group for generation 7140{noformat}
The number of log entry "Rebalancing group..." per hour
{noformat}
$ for log in `grep -l 'Preparing to rebalance group mirror_maker_group' *`; do
echo -n "$log "; grep 'Preparing to rebalance group mirror_maker_group' $log |
wc -l; done
server.2021-03-02-10.kafka3 5
server.2021-03-02-11.kafka3 1
server.2021-03-02-12.kafka3 3
server.2021-03-02-10.kafka2 1
server.2021-03-02-11.kafka2 306
server.2021-03-02-12.kafka2 452
server.2021-03-02-13.kafka2 1 # Here we rebooted broker(2)
{noformat}
The questions:
* Is it a bug or topicless broker should be involved in coordinating consumer
group?
* If it's a bug,
[jira] [Updated] (KAFKA-12405) Inconsistent broker without active topics still involved in coordinating consumer group
[
https://issues.apache.org/jira/browse/KAFKA-12405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander updated KAFKA-12405:
--
Description:
Short description:
Broker without active topics and with inconsistent data still involved in
coordinating consumer group. It leads to constant rebalancing consumer group
Details:
We have kafka cluster(kafka 2.3.0) of three nodes(1,2,3)
* After high disk load broker(2) came out of cluster
* Brokers (1), (3) shortly became leaders of it's topics
* Broker(2) back to cluster, but have failed to became leader of it's
preferred topics due to "Out of sync replicas"
* Now it start spamming with
"Shrinking ISR from 3,2,1 to 2, ... Out of sync replicas"
"Cached zkVersion [20] not equal to that in zookeeper, skip updating ISR"
Like in https://issues.apache.org/jira/browse/KAFKA-3042
But at the same time we start to observing constant rebalancing of consumer
group "mirror_maker_group" and topic reading within this group was completely
stucked. Restarting broker(2) fix the problem
Broker(2) server log:
{noformat}
2021-03-02 11:34:38,791 + [executor-Produce]
[kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Preparing
to rebalance group mirror_maker_group in state PreparingRebalance with old
generation 7139 (__consumer_offsets-30) (reason: error when storing group
assignment during SyncGroup (member:
mirror_maker_group-0-0668b084-bef9-46f6-aec2-07807f1634a0))
(kafka.coordinator.group.GroupCoordinator)
2021-03-02 11:34:38,837 + [data-plane-kafka-request-handler-6]
[kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]:
Stabilized group mirror_maker_group generation 7140 (__consumer_offsets-30)
(kafka.coordinator.group.GroupCoordinator)
2021-03-02 11:34:38,885 + [data-plane-kafka-request-handler-7]
[kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]:
Assignment received from leader for group mirror_maker_group for generation
7140 (kafka.coordinator.group.GroupCoordinator){noformat}
The number of log entry "Rebalancing group..." per hour
{noformat}
$ for log in `grep -l 'Preparing to rebalance group mirror_maker_group' *`; do
echo -n "$log "; grep 'Preparing to rebalance group mirror_maker_group' $log |
wc -l; done
server.2021-03-02-10.kafka3 5 # kafka3 - fine node with actvice controller
server.2021-03-02-11.kafka3 1
server.2021-03-02-12.kafka3 3
server.2021-03-02-10.kafka2 1 # kafka2 - node with inconsistent broker
server.2021-03-02-11.kafka2 306
server.2021-03-02-12.kafka2 452
server.2021-03-02-13.kafka2 1 # Here we rebooted broker(2)
{noformat}
The questions:
* Is it a bug or topicless broker should be involved in coordinating consumer
group?
* If it's a bug, was this fixed in any later kafka release?
Thank you.
was:
Short description:
Broker without active topics and with inconsistent data still involved in
coordinating consumer group. It leads to constant rebalancing consumer group
Details:
We have kafka cluster(kafka 2.3.0) of three nodes(1,2,3)
* After high disk load broker(2) came out of cluster
* Brokers (1), (3) shortly became leaders of it's topics
* Broker(2) back to cluster, but have failed to became leader of it's
preferred topics due to "Out of sync replicas"
* Now it start spamming with
"Shrinking ISR from 3,2,1 to 2, ... Out of sync replicas"
"Cached zkVersion [20] not equal to that in zookeeper, skip updating ISR"
Like in https://issues.apache.org/jira/browse/KAFKA-3042
But at the same time we start to observing constant rebalancing of consumer
group "mirror_maker_group" and topic reading within this group was completely
stucked. Restarting broker(2) fix the problem
Broker(2) server log:
{noformat}
2021-03-02 11:34:38,791 + [executor-Produce]
[kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Preparing
to rebalance group mirror_maker_group in state PreparingRebalance with old
generation 7139 (__consumer_offsets-30) (reason: error when storing group
assignment during SyncGroup (member:
mirror_maker_group-0-0668b084-bef9-46f6-aec2-07807f1634a0))
(kafka.coordinator.group.GroupCoordinator)
2021-03-02 11:34:38,837 + [data-plane-kafka-request-handler-6]
[kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]:
Stabilized group mirror_maker_group generation 7140 (__consumer_offsets-30)
(kafka.coordinator.group.GroupCoordinator)
2021-03-02 11:34:38,885 + [data-plane-kafka-request-handler-7]
[kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]:
Assignment received from leader for group mirror_maker_group for generation
7140 (kafka.coordinator.group.GroupCoordinator){noformat}
The number of log entry "Rebalancing group..." per hour
{noformat}
$ for log in `grep -l 'Preparing to rebalance group mirror_maker_group' *`; do
echo -n "$log "; grep 'Preparing to rebalance group mirror_maker_group' $log |
wc -l; done
[jira] [Updated] (KAFKA-12405) Inconsistent broker without active topics still involved in coordinating consumer group
[
https://issues.apache.org/jira/browse/KAFKA-12405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander updated KAFKA-12405:
--
Description:
Short description:
Broker without active topics and with inconsistent data still involved in
coordinating consumer group. It leads to constant rebalancing consumer group
Details:
We have kafka cluster(kafka 2.3.0) of three nodes(1,2,3) All topics with same
configuration:
Topic:xxx.metrics PartitionCount:1 ReplicationFactor:3 Configs:
Topic: xxx.metrics Partition: 0 Leader: 2 Replicas: 2,3,1 Isr: 3,1,2
* After high disk load broker(2) came out of cluster
* Brokers (1), (3) shortly became leaders of it's topics
* Broker(2) back to cluster, but have failed to became leader of it's
preferred topics due to "Out of sync replicas"
* Now it start spamming with
"Shrinking ISR from 3,2,1 to 2, ... Out of sync replicas"
"Cached zkVersion [20] not equal to that in zookeeper, skip updating ISR"
Like in https://issues.apache.org/jira/browse/KAFKA-3042
But at the same time we start to observing constant rebalancing of consumer
group "mirror_maker_group" and topic reading within this group was completely
stucked. Restarting broker(2) fix the problem
Broker(2) server log:
{noformat}
2021-03-02 11:34:38,791 + [executor-Produce]
[kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Preparing
to rebalance group mirror_maker_group in state PreparingRebalance with old
generation 7139 (__consumer_offsets-30) (reason: error when storing group
assignment during SyncGroup (member:
mirror_maker_group-0-0668b084-bef9-46f6-aec2-07807f1634a0))
(kafka.coordinator.group.GroupCoordinator)
2021-03-02 11:34:38,837 + [data-plane-kafka-request-handler-6]
[kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]:
Stabilized group mirror_maker_group generation 7140 (__consumer_offsets-30)
(kafka.coordinator.group.GroupCoordinator)
2021-03-02 11:34:38,885 + [data-plane-kafka-request-handler-7]
[kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]:
Assignment received from leader for group mirror_maker_group for generation
7140 (kafka.coordinator.group.GroupCoordinator){noformat}
The number of log entry "Rebalancing group..." per hour
{noformat}
$ for log in `grep -l 'Preparing to rebalance group mirror_maker_group' *`; do
echo -n "$log "; grep 'Preparing to rebalance group mirror_maker_group' $log |
wc -l; done
server.2021-03-02-10.kafka3 5 # kafka3 - fine node with actvice controller
server.2021-03-02-11.kafka3 1
server.2021-03-02-12.kafka3 3
server.2021-03-02-10.kafka2 1 # kafka2 - node with inconsistent broker
server.2021-03-02-11.kafka2 306
server.2021-03-02-12.kafka2 452
server.2021-03-02-13.kafka2 1 # Here we rebooted broker(2)
{noformat}
The questions:
* Is it a bug or topicless broker should be involved in coordinating consumer
group?
* If it's a bug, was this fixed in any later kafka release?
Thank you.
was:
Short description:
Broker without active topics and with inconsistent data still involved in
coordinating consumer group. It leads to constant rebalancing consumer group
Details:
We have kafka cluster(kafka 2.3.0) of three nodes(1,2,3)
* After high disk load broker(2) came out of cluster
* Brokers (1), (3) shortly became leaders of it's topics
* Broker(2) back to cluster, but have failed to became leader of it's
preferred topics due to "Out of sync replicas"
* Now it start spamming with
"Shrinking ISR from 3,2,1 to 2, ... Out of sync replicas"
"Cached zkVersion [20] not equal to that in zookeeper, skip updating ISR"
Like in https://issues.apache.org/jira/browse/KAFKA-3042
But at the same time we start to observing constant rebalancing of consumer
group "mirror_maker_group" and topic reading within this group was completely
stucked. Restarting broker(2) fix the problem
Broker(2) server log:
{noformat}
2021-03-02 11:34:38,791 + [executor-Produce]
[kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Preparing
to rebalance group mirror_maker_group in state PreparingRebalance with old
generation 7139 (__consumer_offsets-30) (reason: error when storing group
assignment during SyncGroup (member:
mirror_maker_group-0-0668b084-bef9-46f6-aec2-07807f1634a0))
(kafka.coordinator.group.GroupCoordinator)
2021-03-02 11:34:38,837 + [data-plane-kafka-request-handler-6]
[kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]:
Stabilized group mirror_maker_group generation 7140 (__consumer_offsets-30)
(kafka.coordinator.group.GroupCoordinator)
2021-03-02 11:34:38,885 + [data-plane-kafka-request-handler-7]
[kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]:
Assignment received from leader for group mirror_maker_group for generation
7140 (kafka.coordinator.group.GroupCoordinator){noformat}
The number of log entry "Rebalancing group..." per hour
{noformat}
