[
https://issues.apache.org/jira/browse/KAFKA-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17472112#comment-17472112
]
Matthias Gabriel edited comment on KAFKA-2729 at 1/10/22, 3:43 PM:
---
Hey [~junrao],
We also have the same issue recurring once a week in version 1.1.0, which is
marked as the "Fix version".
We run a cluster with 3 Kafka Brokers:
Node-1
{code:java}
[2021-12-31 19:12:23,540] INFO [Partition topicXYZ-1 broker=1] Shrinking ISR
from 5,3,1 to 5,1 (kafka.cluster.Partition)
[2021-12-31 19:12:23,544] INFO [Partition topicXYZ-1 broker=1] Cached zkVersion
[326] not equal to that in zookeeper, skip updating ISR
(kafka.cluster.Partition){code}
On Node-2 we do not see any related message for the timeperiod
On Node-3 we have the following message, which we are not sure if its related
at all.
{code:java}
2021-12-31 19:12:23,541 [myid:3] - INFO [ProcessThread(sid:3
cport:-1)::PrepRequestProcessor@653] - Got user-level KeeperException when
processing sessionid:0x1004521e6ed type:setData cxid:0xbca4
zxid:0x3a4a372 txntype:-1 reqpath:n/a Error
Path:/brokers/topics/topicXYZ/partitions/1/state Error:KeeperErrorCode =
BadVersion for /brokers/topics/topicXYZ/partitions/1/state{code}
Do you have any idea what we could do or which data we could deliver to give
you additional insights?
Thanks
Matthias
was (Author: JIRAUSER283346):
Hey [~junrao],
We also have the same issue recurring once a week in version 1.1.0, which is
marked as the "Fix version".
We run a cluster with 3 Kafka Brokers:
Node-1
{code:java}
[2021-12-31 19:12:23,540] INFO [Partition topicXYZ-1 broker=1] Shrinking ISR
from 5,3,1 to 5,1 (kafka.cluster.Partition)
[2021-12-31 19:12:23,544] INFO [Partition topicXYZ-1 broker=1] Cached zkVersion
[326] not equal to that in zookeeper, skip updating ISR
(kafka.cluster.Partition){code}
On Node-2 we do not see any related message for the timeperiod
On Node-3 we have the following message, which we are not sure if its related
at all.
{code:java}
2021-12-31 19:12:23,541 [myid:3] - INFO [ProcessThread(sid:3
cport:-1)::PrepRequestProcessor@653] - Got user-level KeeperException when
processing sessionid:0x1004521e6ed type:setData cxid:0xbca4
zxid:0x3a4a372 txntype:-1 reqpath:n/a Error
Path:/brokers/topics/topicXYZ/partitions/1/state Error:KeeperErrorCode =
BadVersion for /brokers/topics/topicXYZ/partitions/1/state{code}
Do you have any idea what we could do or which data we could deliver to give
you additional insights?
Thanks
Matthias
> Cached zkVersion not equal to that in zookeeper, broker not recovering.
> ---
>
> Key: KAFKA-2729
> URL: https://issues.apache.org/jira/browse/KAFKA-2729
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 0.8.2.1, 0.9.0.0, 0.10.0.0, 0.10.1.0, 0.11.0.0, 2.4.1
>Reporter: Danil Serdyuchenko
>Assignee: Onur Karaman
>Priority: Critical
> Fix For: 1.1.0
>
>
> After a small network wobble where zookeeper nodes couldn't reach each other,
> we started seeing a large number of undereplicated partitions. The zookeeper
> cluster recovered, however we continued to see a large number of
> undereplicated partitions. Two brokers in the kafka cluster were showing this
> in the logs:
> {code}
> [2015-10-27 11:36:00,888] INFO Partition
> [__samza_checkpoint_event-creation_1,3] on broker 5: Shrinking ISR for
> partition [__samza_checkpoint_event-creation_1,3] from 6,5 to 5
> (kafka.cluster.Partition)
> [2015-10-27 11:36:00,891] INFO Partition
> [__samza_checkpoint_event-creation_1,3] on broker 5: Cached zkVersion [66]
> not equal to that in zookeeper, skip updating ISR (kafka.cluster.Partition)
> {code}
> For all of the topics on the effected brokers. Both brokers only recovered
> after a restart. Our own investigation yielded nothing, I was hoping you
> could shed some light on this issue. Possibly if it's related to:
> https://issues.apache.org/jira/browse/KAFKA-1382 , however we're using
> 0.8.2.1.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)