Maung, [2014-03-14 12:11:44,310] INFO Partition [nelo2-normal-logs,0] on broker 0: Shrinking ISR for partition [nelo2-normal-logs,0] from 0,1 to 0 (kafka.cluster.Partition) [2014-03-14 12:11:44,313] INFO Partition [nelo2-symbolicated-logs,1] on broker 0: Shrinking ISR for partition [nelo2-symbolicated-logs,1] from 0,2 to 0 (kafka.cluster.Partition)
The above log messages indicate that the leader is trying to remove the follower brokers 1,2 from the ISR for the respective partitions. To prevent such churns in the ISR, you can follow the steps listed here - https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowtoreducechurnsinISR?WhendoesabrokerleavetheISR ? What about the error: "ERROR Conditional update of path” in there. Is that OK? Only if it is transient and recovers on its own. This error is logged on the broker when the controller broker shrinks the ISR for the same partition around the same time. Due to this, the leader's update of the ISR fails momentarily until it refreshes it's view from zookeeper and tries again. I agree that the log messages are confusing and can be improved. Thanks, Neha On Fri, Mar 14, 2014 at 10:37 AM, Maung Than <[email protected]> wrote: > What Information in the log indicates that broker 1 is falling behind the > leader? > > What about the error: "ERROR Conditional update of path” in there. Is > that OK? > > I just want to be label to read the log better. > > I assume below two properties determines the broker behavior of removing > the follower(s). > > replica.lag.time.max.ms 10000 If a follower hasn't sent any fetch > requests for this window of time, the leader will remove the follower from > ISR (in-sync replicas) and treat it as dead. > replica.lag.max.messages 4000 If a replica falls more than this > many messages behind the leader, the leader will remove the follower from > ISR and treat it as dead. > > Thanks, > Maung > > On Mar 14, 2014, at 7:45 AM, Neha Narkhede <[email protected]> > wrote: > > > No. This indicates the follower broker 1 is falling behind the leader and > > hence the leader removes it from the ISR. We need to improve logging. > This > > does not indicate a problem with the broker. > > > > On Thursday, March 13, 2014, 陈小军 <[email protected]> wrote: > > > >> Hi all > >> i use 0.8.1 release to do test, and get following error logs > >> [hadoop@nelo76 libs]$ [2014-03-14 12:11:44,310] INFO Partition > >> [nelo2-normal-logs,0] on broker 0: Shrinking ISR for partition > >> [nelo2-normal-logs,0] from 0,1 to 0 (kafka.cluster.Partition) > >> [2014-03-14 12:11:44,313] ERROR Conditional update of path > >> /brokers/topics/nelo2-normal-logs/partitions/0/state with data > >> {"controller_epoch":4,"leader":0,"version":1,"leader_epoch":5,"isr":[0]} > >> and expected version 7 failed due to > >> org.apache.zookeeper.KeeperException$BadVersionException: > KeeperErrorCode = > >> BadVersion for /brokers/topics/nelo2-normal-logs/partitions/0/state > >> (kafka.utils.ZkUtils$) > >> [2014-03-14 12:11:44,313] INFO Partition [nelo2-normal-logs,0] on broker > >> 0: Cached zkVersion [7] not equal to that in zookeeper, skip updating > ISR > >> (kafka.cluster.Partition) > >> [2014-03-14 12:11:44,313] INFO Partition [nelo2-symbolicated-logs,1] on > >> broker 0: Shrinking ISR for partition [nelo2-symbolicated-logs,1] from > 0,2 > >> to 0 (kafka.cluster.Partition) > >> [2014-03-14 12:11:44,315] ERROR Conditional update of path > >> /brokers/topics/nelo2-symbolicated-logs/partitions/1/state with data > >> {"controller_epoch":4,"leader":0,"version":1,"leader_epoch":6,"isr":[0]} > >> and expected version 8 failed due to > >> org.apache.zookeeper.KeeperException$BadVersionException: > KeeperErrorCode = > >> BadVersion for > /brokers/topics/nelo2-symbolicated-logs/partitions/1/state > >> (kafka.utils.ZkUtils$) > >> [2014-03-14 12:11:44,315] INFO Partition [nelo2-symbolicated-logs,1] on > >> broker 0: Cached zkVersion [8] not equal to that in zookeeper, skip > >> updating ISR (kafka.cluster.Partition) > >> [2014-03-14 12:11:44,316] INFO Partition [nelo2-crash-logs,1] on broker > 0: > >> Shrinking ISR for partition [nelo2-crash-logs,1] from 0,1 to 0 > >> (kafka.cluster.Partition) > >> [2014-03-14 12:11:44,318] ERROR Conditional update of path > >> /brokers/topics/nelo2-crash-logs/partitions/1/state with data > >> {"controller_epoch":4,"leader":0,"version":1,"leader_epoch":5,"isr":[0]} > >> and expected version 7 failed due to > >> org.apache.zookeeper.KeeperException$BadVersionException: > KeeperErrorCode = > >> BadVersion for /brokers/topics/nelo2-crash-logs/partitions/1/state > >> (kafka.utils.ZkUtils$) > >> [2014-03-14 12:11:44,318] INFO Partition [nelo2-crash-logs,1] on broker > 0: > >> Cached zkVersion [7] not equal to that in zookeeper, skip updating ISR > >> (kafka.cluster.Partition) > >> > >> this issue is related withe zookeeper version? > >> my zookeeper version is 3.4.3 > >> > >> Best Regards > >> Jerry > >> > >
