Hi all, I’m new to this list so please forgive me if I’m asking a question that has been discussed before or if I’m asking it in the wrong place.
I sometimes see 1 or 2 out of our 5 zookeepers going down. If it’s two going down, it appears to happen at almost the same millisecond. They stay down for ~16 minutes. This scares the hell out of me because it means that during this time we don’t have any redundancy left. Our 5 zookeepers are at version 3.5.8, running in libvirt VMs with Centos 8 and OpenJDK 1.8.0_322. They are all on a separate VM host. I checked for network issues (with ping, iperf3) but found nothing. It’s not always the same zookeeper VM(s) that suffers from this problem. It does not happen at a regular time so there does not seem to be a cron job affecting this behavior. I have some logging from the last time this happened; I’ll add it below. I know we’re running an old version of Zookeeper. I have looked through the release notes but I couldn’t find a fix for this. I’d welcome any ideas. How could we prevent this? Based on the SocketTimeoutException, I’d think there might be a network hiccup; but why is it so difficult for Zookeeper to recover from this? Is it a bug in Zookeeper? Should I report it on https://issues.apache.org? Kind regards, Onno [root@feszoo4 /var/log]# grep 'Oct 11 16:01:4.*zookeeper' messages-20221016 Oct 11 16:01:43 feszoo4 zookeeper[1199]: 2022-10-11 16:01:43,837 - WARN [QuorumPeer[myid=4](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):Follower@96] - Exception when following the leader Oct 11 16:01:43 feszoo4 zookeeper[1199]: java.net.SocketTimeoutException: Read timed out ... [root@feszoo3 /var/log]# grep 'Oct 11 16:01:4.*zookeeper' messages-20221016 Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,838 - WARN [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):Follower@96] - Exception when following the leader Oct 11 16:01:43 feszoo3 zookeeper[1187]: java.net.SocketTimeoutException: Read timed out Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at java.net.SocketInputStream.socketRead0(Native Method) Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at java.net.SocketInputStream.read(SocketInputStream.java:171) Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at java.net.SocketInputStream.read(SocketInputStream.java:141) Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at java.io.BufferedInputStream.read(BufferedInputStream.java:265) Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at java.io.DataInputStream.readInt(DataInputStream.java:387) Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:84) Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:86) Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:118) Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:158) Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:92) Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1253) Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,839 - INFO [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):Follower@201] - shutdown called Oct 11 16:01:43 feszoo3 zookeeper[1187]: java.lang.Exception: shutdown Follower Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:201) Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1257) Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,866 - INFO [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 2 (n.leader), 0x8a00010a28 (n.zxid), 0x52 (n.round), LOOKING (n.state), 2 (n.sid), 0x8a (n.peerEPoch), FOLLOWING (my state)0 (n.config version) Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,869 - INFO [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 1 (n.leader), 0x8a00010a28 (n.zxid), 0x52 (n.round), LOOKING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), FOLLOWING (my state)0 (n.config version) Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,869 - INFO [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 2 (n.leader), 0x8a00010a28 (n.zxid), 0x52 (n.round), LOOKING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), FOLLOWING (my state)0 (n.config version) Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,885 - INFO [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):LearnerZooKeeperServer@165] - Shutting down Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,885 - INFO [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):ZooKeeperServer@558] - shutting down Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,885 - INFO [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FollowerRequestProcessor@139] - Shutting down Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,885 - INFO [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):CommitProcessor@362] - Shutting down Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,887 - INFO [FollowerRequestProcessor:3:FollowerRequestProcessor@110] - FollowerRequestProcessor exited loop! Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,887 - INFO [CommitProcessor:3:CommitProcessor@195] - CommitProcessor exited loop! Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,888 - INFO [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FinalRequestProcessor@514] - shutdown of request processor complete Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,890 - INFO [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):SyncRequestProcessor@191] - Shutting down Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,890 - INFO [SyncThread:3:SyncRequestProcessor@169] - SyncRequestProcessor exited! Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,891 - WARN [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):QuorumPeer@1300] - PeerState set to LOOKING Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,891 - INFO [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):QuorumPeer@1175] - LOOKING Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,891 - INFO [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@903] - New election. My id = 3, proposed zxid=0x8a00010a28 Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,892 - INFO [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 3 (n.leader), 0x8a00010a28 (n.zxid), 0x52 (n.round), LOOKING (n.state), 3 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version) Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,893 - INFO [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 (n.round), FOLLOWING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version) Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,893 - INFO [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 (n.round), FOLLOWING (n.state), 2 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version) Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,938 - INFO [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52 (n.round), LOOKING (n.state), 4 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version) Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,939 - INFO [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52 (n.round), LOOKING (n.state), 3 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version) Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,940 - INFO [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 (n.round), FOLLOWING (n.state), 2 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version) Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,941 - INFO [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 (n.round), FOLLOWING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version) Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,004 - WARN [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,019 - WARN [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,046 - WARN [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,053 - WARN [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,057 - WARN [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,075 - WARN [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,086 - WARN [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,090 - WARN [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,095 - WARN [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,101 - WARN [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,113 - WARN [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,122 - WARN [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,142 - INFO [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@937] - Notification time out: 400 Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,142 - INFO [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52 (n.round), LOOKING (n.state), 3 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version) Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,143 - INFO [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52 (n.round), LOOKING (n.state), 4 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version) Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,144 - INFO [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 (n.round), FOLLOWING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version) Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,144 - INFO [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 (n.round), FOLLOWING (n.state), 2 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version) Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,149 - WARN [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,152 - WARN [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,158 - WARN [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,159 - WARN [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,162 - WARN [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,167 - WARN [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,179 - WARN [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,186 - WARN [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,198 - WARN [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,217 - WARN [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,228 - WARN [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,234 - WARN [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,235 - WARN [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,241 - WARN [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,242 - WARN [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,246 - WARN [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,250 - WARN [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,255 - WARN [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,255 - WARN [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,260 - WARN [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,265 - WARN [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,276 - WARN [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,280 - WARN [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,288 - WARN [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,300 - WARN [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,307 - WARN [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,312 - WARN [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,318 - WARN [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,322 - WARN [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,322 - WARN [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,325 - WARN [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,344 - WARN [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,351 - WARN [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,375 - WARN [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,377 - WARN [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,379 - WARN [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,379 - WARN [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,381 - WARN [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,394 - WARN [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,395 - WARN [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,408 - WARN [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,419 - WARN [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,419 - WARN [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,424 - WARN [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,432 - WARN [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,433 - WARN [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,439 - WARN [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,444 - WARN [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,476 - WARN [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,482 - WARN [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,486 - WARN [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,490 - WARN [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,495 - WARN [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,525 - WARN [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,526 - WARN [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,535 - WARN [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,537 - WARN [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,540 - WARN [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,541 - WARN [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,544 - WARN [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,545 - WARN [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,545 - INFO [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@937] - Notification time out: 800 Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,546 - INFO [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52 (n.round), LOOKING (n.state), 3 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version) Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,546 - INFO [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 (n.round), FOLLOWING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version) Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,547 - INFO [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52 (n.round), LOOKING (n.state), 4 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version) Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,547 - INFO [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 (n.round), FOLLOWING (n.state), 2 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version) Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,553 - WARN [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,556 - WARN [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,562 - WARN [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,580 - WARN [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,596 - WARN [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,597 - WARN [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,607 - WARN [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,625 - WARN [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,632 - WARN [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,638 - WARN [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,646 - WARN [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,648 - WARN [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,667 - WARN [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,673 - WARN [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,680 - WARN [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,690 - WARN [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,693 - WARN [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,697 - WARN [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,708 - WARN [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,712 - WARN [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,716 - WARN [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,723 - WARN [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,727 - WARN [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,734 - WARN [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,740 - WARN [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,744 - WARN [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,752 - WARN [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,765 - WARN [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,774 - WARN [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,778 - WARN [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,803 - WARN [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,804 - WARN [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,821 - WARN [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,832 - WARN [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,836 - WARN [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,837 - WARN [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,854 - WARN [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,862 - WARN [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,867 - WARN [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,872 - WARN [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,874 - WARN [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,883 - WARN [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,886 - WARN [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,888 - WARN [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,892 - WARN [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,895 - WARN [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,909 - WARN [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,913 - WARN [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,920 - WARN [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,920 - WARN [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,934 - WARN [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,938 - WARN [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,939 - WARN [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,939 - WARN [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,947 - WARN [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,951 - WARN [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,956 - WARN [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,964 - WARN [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,976 - WARN [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,980 - WARN [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,012 - WARN [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,027 - WARN [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,036 - WARN [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,070 - WARN [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,090 - WARN [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,127 - WARN [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,197 - WARN [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,348 - INFO [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@937] - Notification time out: 1600 Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,349 - INFO [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52 (n.round), LOOKING (n.state), 3 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version) Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,350 - INFO [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 (n.round), FOLLOWING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version) Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,350 - INFO [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52 (n.round), LOOKING (n.state), 4 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version) Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,351 - INFO [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 (n.round), FOLLOWING (n.state), 2 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version) Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,356 - WARN [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,385 - WARN [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,451 - WARN [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,561 - WARN [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,563 - WARN [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,576 - WARN [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,595 - WARN [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,639 - WARN [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,654 - WARN [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,685 - WARN [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,703 - WARN [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,744 - WARN [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,753 - WARN [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,014 - WARN [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,045 - WARN [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,062 - WARN [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,121 - WARN [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,141 - WARN [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,151 - WARN [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,224 - WARN [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,254 - WARN [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,322 - WARN [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,398 - WARN [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,435 - WARN [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,522 - WARN [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,541 - WARN [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,817 - WARN [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,873 - WARN [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,897 - WARN [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,914 - WARN [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,951 - INFO [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@937] - Notification time out: 3200 Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,952 - INFO [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52 (n.round), LOOKING (n.state), 3 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version) Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,954 - INFO [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52 (n.round), LOOKING (n.state), 4 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version) Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,954 - INFO [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 (n.round), FOLLOWING (n.state), 2 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version) Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,955 - INFO [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 (n.round), FOLLOWING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version) Oct 11 16:01:47 feszoo3 zookeeper[1187]: 2022-10-11 16:01:47,582 - WARN [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running [root@feszoo3 ~]# cat /etc/zookeeper/zoo.cfg # The number of milliseconds of each tick tickTime=2000 # The number of ticks that the initial # synchronization phase can take initLimit=10 # The number of ticks that can pass between # sending a request and getting an acknowledgement syncLimit=5 # The directory where the snapshot is stored. dataDir=/var/lib/zookeeper/data # The port at which the clients will connect clientPort=2181 skipACL=1 # 4-letter words: which commands are allowed 4lw.commands.whitelist=* # Cluster topo server.1=feszoo1:2888:3888 server.2=feszoo2:2888:3888 server.3=feszoo3:2888:3888 server.4=feszoo4:2888:3888 server.5=feszoo5:2888:3888
