Hello, We seem to observe that after the zookeeper cluster had a leader reelection, kafka broker required a restart before the broker id showed up in /brokers/ids on ZK Is this a known issue and/or a manifestation of [KAFKA-2729] Cached zkVersion not equal to that in zookeeper, broker not recovering.? Thanks inter.broker.protocol.version = 0.10.0-IV1 ls /brokers/ids/1020Node does not exist: /brokers/ids/1020 From ZK017-03-21 04:00:21,794 [myid:4] - INFO [QuorumPeer[myid=4]/0.0.0.0:2181:Leader@493] - Shutting down2017-03-21 04:00:21,797 [myid:4] - INFO [QuorumPeer[myid=4]/0.0.0.0:2181:Leader@499] - Shutdown calledjava.lang.Exception: shutdown Leader! reason: Not sufficient followers synced, only synced with sids: [ 4 ] at org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.java:499) at org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:474) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:799) :2017-03-21 04:00:21,833 [myid:4] - INFO [QuorumPeer[myid=4]/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client / .7.229:40781 which had sessionid 0x5ab1fbe0500000:017-03-21 04:00:22,890 [myid:4] - INFO [QuorumPeer[myid=4]/0.0.0.0:2181:ZooKeeperServer@162] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /var/lib/zookeeper/version-2 snapdir /var/lib/zookeeper/version-22017-03-21 04:00:22,890 [myid:4] - INFO [QuorumPeer[myid=4]/0.0.0.0:2181:Follower@63] - FOLLOWING - LEADER ELECTION TOOK - 994 Here is the sequence of events.. on broker [2017-03-21 04:00:19,667] INFO Client session timed out, have not heard from server in 4002ms for sessionid 0x5ab1fbe0500000, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn) [2017-03-21 04:00:19,768] INFO Waiting for keeper state SyncConnected (org.I0Itec.zkclient.ZkClient) [2017-03-21 04:00:19,768] INFO zookeeper state changed (Disconnected) (org.I0Itec.zkclient.ZkClient) [2017-03-21 04:00:20,440] INFO Opening socket connection to server .7.243/.7.243:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn) [2017-03-21 04:00:20,441] INFO Socket connection established to .7.243/.7.243:2181, initiating session (org.apache.zookeeper.ClientCnxn) [2017-03-21 04:00:20,443] INFO Unable to read additional data from server sessionid 0x5ab1fbe0500000, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn) [2017-03-21 04:00:20,543] INFO Waiting for keeper state SyncConnected (org.I0Itec.zkclient.ZkClient) [2017-03-21 04:00:21,053] INFO Opening socket connection to server .7.244/7.244:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn) [2017-03-21 04:00:21,316] INFO Socket connection established to .7.244/1.7.244:2181, initiating session (org.apache.zookeeper.ClientCnxn) [2017-03-21 04:00:21,786] INFO Session establishment complete on server .7.244/1.7.244:2181, sessionid = 0x5ab1fbe0500000, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn) [2017-03-21 04:00:21,786] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient) [2017-03-21 04:00:21,844] INFO Unable to read additional data from server sessionid 0x5ab1fbe0500000, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn) [2017-03-21 04:00:21,945] INFO Waiting for keeper state SyncConnected (org.I0Itec.zkclient.ZkClient) [2017-03-21 04:00:21,945] INFO zookeeper state changed (Disconnected) (org.I0Itec.zkclient.ZkClient)::2017-03-21 04:00:22,225] ERROR [ReplicaFetcherThread-2-1012], Error for partition [b0ad7e5c-63e8-4ffd-8d53-9bf94277c5aa_topic,1] to broker 1012:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)[2017-03-21 04:00:22,869] INFO Opening socket connection to server .7.240/7.240:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)[2017-03-21 04:00:22,870] INFO Socket connection established to .7.240/7.240:2181, initiating session (org.apache.zookeeper.ClientCnxn)[2017-03-21 04:00:22,872] INFO Session establishment complete on server .7.240/7.240:2181, sessionid = 0x5ab1fbe0500000, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)[2017-03-21 04:00:22,873] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)[2017-03-21 04:00:22,882] INFO Partition [f61a1576-6913-47d1-b737-dc747f0cb550_topic,3] on broker 1020: Cached zkVersion [25] not equal to that in zookeeper, skip updating ISR (kafka.cluster.Partition)
