[ https://issues.apache.org/jira/browse/KAFKA-1451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14346728#comment-14346728 ]
Aldrin Seychell commented on KAFKA-1451: ---------------------------------------- I just encountered this issue on version 0.8.2.0 after a period of slow PC performance and perhaps zookeeper and kafka were slow to communicate between each other (possibly the same issue highlighted by [~sinewy]. This resulted in an infinite loop in attempting to created the ephemeral node. The logs that were continously being written are as follows: [2015-03-03 13:48:19,831] INFO conflict in /brokers/ids/0 data: {"jmx_port":-1,"timestamp":"1425386833617","host":"MTDKP119.ix.com","version":1,"port":9092} stored data: {"jmx_port":-1,"timestamp":"1425380575230","host":"MTDKP119.ix.com","version":1,"port":9092} (kafka.utils.ZkUtils$) [2015-03-03 13:48:19,832] INFO I wrote this conflicted ephemeral node [{"jmx_port":-1,"timestamp":"1425386833617","host":"MTDKP119.ix.com","version":1,"port":9092}] at /brokers/ids/0 a while back in a different session, hence I will backoff for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$) [2015-03-03 13:48:25,844] INFO conflict in /brokers/ids/0 data: {"jmx_port":-1,"timestamp":"1425386833617","host":"MTDKP119.ix.com","version":1,"port":9092} stored data: {"jmx_port":-1,"timestamp":"1425380575230","host":"MTDKP119.ix.com","version":1,"port":9092} (kafka.utils.ZkUtils$) > Broker stuck due to leader election race > ----------------------------------------- > > Key: KAFKA-1451 > URL: https://issues.apache.org/jira/browse/KAFKA-1451 > Project: Kafka > Issue Type: Bug > Components: core > Affects Versions: 0.8.1.1 > Reporter: Maciek Makowski > Assignee: Manikumar Reddy > Priority: Minor > Labels: newbie > Fix For: 0.8.2.0 > > Attachments: KAFKA-1451.patch, KAFKA-1451_2014-07-28_20:27:32.patch, > KAFKA-1451_2014-07-29_10:13:23.patch > > > h3. Symptoms > The broker does not become available due to being stuck in an infinite loop > while electing leader. This can be recognised by the following line being > repeatedly written to server.log: > {code} > [2014-05-14 04:35:09,187] INFO I wrote this conflicted ephemeral node > [{"version":1,"brokerid":1,"timestamp":"1400060079108"}] at /controller a > while back in a different session, hence I will backoff for this node to be > deleted by Zookeeper and retry (kafka.utils.ZkUtils$) > {code} > h3. Steps to Reproduce > In a single kafka 0.8.1.1 node, single zookeeper 3.4.6 (but will likely > behave the same with the ZK version included in Kafka distribution) node > setup: > # start both zookeeper and kafka (in any order) > # stop zookeeper > # stop kafka > # start kafka > # start zookeeper > h3. Likely Cause > {{ZookeeperLeaderElector}} subscribes to data changes on startup, and then > triggers an election. if the deletion of ephemeral {{/controller}} node > associated with previous zookeeper session of the broker happens after > subscription to changes in new session, election will be invoked twice, once > from {{startup}} and once from {{handleDataDeleted}}: > * {{startup}}: acquire {{controllerLock}} > * {{startup}}: subscribe to data changes > * zookeeper: delete {{/controller}} since the session that created it timed > out > * {{handleDataDeleted}}: {{/controller}} was deleted > * {{handleDataDeleted}}: wait on {{controllerLock}} > * {{startup}}: elect -- writes {{/controller}} > * {{startup}}: release {{controllerLock}} > * {{handleDataDeleted}}: acquire {{controllerLock}} > * {{handleDataDeleted}}: elect -- attempts to write {{/controller}} and then > gets into infinite loop as a result of conflict > {{createEphemeralPathExpectConflictHandleZKBug}} assumes that the existing > znode was written from different session, which is not true in this case; it > was written from the same session. That adds to the confusion. > h3. Suggested Fix > In {{ZookeeperLeaderElector.startup}} first run {{elect}} and then subscribe > to data changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)