[ https://issues.apache.org/jira/browse/KAFKA-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17472112#comment-17472112 ]
Matthias Gabriel commented on KAFKA-2729: ----------------------------------------- Hey [~junrao], We also have the same issue recurring once a week in version 1.1.0, which is marked as the "Fix version". We run a cluster with 3 Kafka Brokers: Node-1 {code:java} [2021-12-31 19:12:23,540] INFO [Partition topicXYZ-1 broker=1] Shrinking ISR from 5,3,1 to 5,1 (kafka.cluster.Partition) [2021-12-31 19:12:23,544] INFO [Partition topicXYZ-1 broker=1] Cached zkVersion [326] not equal to that in zookeeper, skip updating ISR (kafka.cluster.Partition){code} On Node-2 we do not see any related message for the timeperiod On Node-3 we have the following message, which we are not sure if its related at all. {code:java} 2021-12-31 19:12:23,541 [myid:3] - INFO [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@653] - Got user-level KeeperException when processing sessionid:0x1004521e6ed0000 type:setData cxid:0xbca4 zxid:0x3a40000a372 txntype:-1 reqpath:n/a Error Path:/brokers/topics/topicXYZ/partitions/1/state Error:KeeperErrorCode = BadVersion for /brokers/topics/topicXYZ/partitions/1/state{code} Do you have any idea what we could do or which data we could deliver to give you additional insights? Thanks Matthias > Cached zkVersion not equal to that in zookeeper, broker not recovering. > ----------------------------------------------------------------------- > > Key: KAFKA-2729 > URL: https://issues.apache.org/jira/browse/KAFKA-2729 > Project: Kafka > Issue Type: Bug > Affects Versions: 0.8.2.1, 0.9.0.0, 0.10.0.0, 0.10.1.0, 0.11.0.0, 2.4.1 > Reporter: Danil Serdyuchenko > Assignee: Onur Karaman > Priority: Critical > Fix For: 1.1.0 > > > After a small network wobble where zookeeper nodes couldn't reach each other, > we started seeing a large number of undereplicated partitions. The zookeeper > cluster recovered, however we continued to see a large number of > undereplicated partitions. Two brokers in the kafka cluster were showing this > in the logs: > {code} > [2015-10-27 11:36:00,888] INFO Partition > [__samza_checkpoint_event-creation_1,3] on broker 5: Shrinking ISR for > partition [__samza_checkpoint_event-creation_1,3] from 6,5 to 5 > (kafka.cluster.Partition) > [2015-10-27 11:36:00,891] INFO Partition > [__samza_checkpoint_event-creation_1,3] on broker 5: Cached zkVersion [66] > not equal to that in zookeeper, skip updating ISR (kafka.cluster.Partition) > {code} > For all of the topics on the effected brokers. Both brokers only recovered > after a restart. Our own investigation yielded nothing, I was hoping you > could shed some light on this issue. Possibly if it's related to: > https://issues.apache.org/jira/browse/KAFKA-1382 , however we're using > 0.8.2.1. -- This message was sent by Atlassian Jira (v8.20.1#820001)