RE: leader update partitions fail with KeeperErrorCode = BadVersion,kafka version=0.8.1.1
i create a topic with 72 partitions 2 replicas,then increased to 108,and the cluster is run ok. some days later i find the topic has 2 partition which ISR only include leader,check follow log-segment with partitions,the log-segment does not later. and i can not find more useful logs from kafka logs or zk logs. if restart leader,the follow will become leader,and pre-leader whill back as follow. Thanks, Lax > Date: Mon, 1 Jun 2015 16:14:15 -0400 > Subject: Re: leader update partitions fail with KeeperErrorCode = > BadVersion,kafka version=0.8.1.1 > From: j...@squareup.com > To: users@kafka.apache.org > > I've seen this problem now too with 0.8.2.1. It happened after we had a > disk failure (but the server failed to shutdown: KAFKA-). After that > happened, subsequently, several ISR's underwent I think 'unclean leader > election', but I'm not 100% sure. But I did see lots of those same error > messages: "Cached zkVersion [X] not equal to that in zookeeper, skip > updating ISR...". > > So, I don't know that the issue was fixed in 0.8.1.1. > > Can you describe the circumstances for the errors you saw? > > Jason > > On Fri, May 29, 2015 at 12:17 AM, chenlax wrote: > > > kafka version =0.8.1.1 > > > > i get the error log as follow: > > > > INFO Partition [Topic_Beacon_1,10] on broker 4: Shrinking ISR for > > partition [Topic_Beacon_1,10] from 4,7 to 4 (kafka.cluster.Partition) > > ERROR Conditional update of path > > /brokers/topics/Topic_Beacon_1/partitions/10/state with data > > {"controller_epoch":16,"leader":4,"version":1,"leader_epoch":7,"isr":[4]} > > and expecte > > d version 5032 failed due to > > org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode = > > BadVersion for /brokers/topics/Topic_Beacon_1/partitions/10/state > > (kafka.utils.ZkUtils$) > > INFO Partition [Topic_Beacon_1,10] on broker 4: Cached zkVersion [5032] > > not equal to that in zookeeper, skip updating ISR (kafka.cluster.Partition) > > > > > > Only restart broker,the issue will fix, why the broker can not back to ISR > > for partition? > > > > and i find the same issue, > > http://mail-archives.apache.org/mod_mbox/kafka-users/201404.mbox/%3CCAHwHRrW_vKsSpoAnhEqQUZVBT5_Djx3qbixsH8=6hAe=vg4...@mail.gmail.com%3E > > > > it point out kafka_0.8.1.1 fix the bug,so i want to know, what causes this > > problem. > > > > > > Thanks, > > Lax > >
Re: leader update partitions fail with KeeperErrorCode = BadVersion,kafka version=0.8.1.1
I've seen this problem now too with 0.8.2.1. It happened after we had a disk failure (but the server failed to shutdown: KAFKA-). After that happened, subsequently, several ISR's underwent I think 'unclean leader election', but I'm not 100% sure. But I did see lots of those same error messages: "Cached zkVersion [X] not equal to that in zookeeper, skip updating ISR...". So, I don't know that the issue was fixed in 0.8.1.1. Can you describe the circumstances for the errors you saw? Jason On Fri, May 29, 2015 at 12:17 AM, chenlax wrote: > kafka version =0.8.1.1 > > i get the error log as follow: > > INFO Partition [Topic_Beacon_1,10] on broker 4: Shrinking ISR for > partition [Topic_Beacon_1,10] from 4,7 to 4 (kafka.cluster.Partition) > ERROR Conditional update of path > /brokers/topics/Topic_Beacon_1/partitions/10/state with data > {"controller_epoch":16,"leader":4,"version":1,"leader_epoch":7,"isr":[4]} > and expecte > d version 5032 failed due to > org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode = > BadVersion for /brokers/topics/Topic_Beacon_1/partitions/10/state > (kafka.utils.ZkUtils$) > INFO Partition [Topic_Beacon_1,10] on broker 4: Cached zkVersion [5032] > not equal to that in zookeeper, skip updating ISR (kafka.cluster.Partition) > > > Only restart broker,the issue will fix, why the broker can not back to ISR > for partition? > > and i find the same issue, > http://mail-archives.apache.org/mod_mbox/kafka-users/201404.mbox/%3CCAHwHRrW_vKsSpoAnhEqQUZVBT5_Djx3qbixsH8=6hAe=vg4...@mail.gmail.com%3E > > it point out kafka_0.8.1.1 fix the bug,so i want to know, what causes this > problem. > > > Thanks, > Lax >
leader update partitions fail with KeeperErrorCode = BadVersion,kafka version=0.8.1.1
kafka version =0.8.1.1 i get the error log as follow: INFO Partition [Topic_Beacon_1,10] on broker 4: Shrinking ISR for partition [Topic_Beacon_1,10] from 4,7 to 4 (kafka.cluster.Partition) ERROR Conditional update of path /brokers/topics/Topic_Beacon_1/partitions/10/state with data {"controller_epoch":16,"leader":4,"version":1,"leader_epoch":7,"isr":[4]} and expecte d version 5032 failed due to org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode = BadVersion for /brokers/topics/Topic_Beacon_1/partitions/10/state (kafka.utils.ZkUtils$) INFO Partition [Topic_Beacon_1,10] on broker 4: Cached zkVersion [5032] not equal to that in zookeeper, skip updating ISR (kafka.cluster.Partition) Only restart broker,the issue will fix, why the broker can not back to ISR for partition? and i find the same issue, http://mail-archives.apache.org/mod_mbox/kafka-users/201404.mbox/%3CCAHwHRrW_vKsSpoAnhEqQUZVBT5_Djx3qbixsH8=6hAe=vg4...@mail.gmail.com%3E it point out kafka_0.8.1.1 fix the bug,so i want to know, what causes this problem. Thanks, Lax