Hello! once I saw something similar, when the problem was that the clocks were not in sync on the ZooKeeper VMs. And when the ntp server updated the clock, it screwed up the kerberos token renewal threads in the current leader, causing SASL related unhandled exceptions during authentication on leader election ports which basically made it impossible for Peers to connect to the Leader for some time.
Although I'm not sure if you use quorum authentication. (I see you don't really use ACLs, at least "skipACL=1" suggests that in your zoo.cfg, so maybe you have a different problem) Anyway, maybe you can check the logs from the leader during this time, maybe you see something interesting there. Also you might look for error messages that would indicate that some threads died in the leader / in other servers. E.g. "ERROR org.apache.zookeeper.server.NIOServerCnxnFactory: Thread Thread[QuorumConnectionThread-[myid=3]-43,5,main] died" In the leader, there are separate threads responsible to handle leader election of other quorum messages. For each peer, there is a separate thread. And it is possible some times that these threads die, and the Leader is still leading but can not accept new quorum members. Several bugs were fixed regarding this in the past. (but AFAIK all of them are present in 3.5.8) Kind regards, Máté On Thu, Oct 20, 2022 at 3:20 PM Onno Zweers <[email protected]> wrote: > Hi all, > > I’m new to this list so please forgive me if I’m asking a question that > has been discussed before or if I’m asking it in the wrong place. > > I sometimes see 1 or 2 out of our 5 zookeepers going down. If it’s two > going down, it appears to happen at almost the same millisecond. They stay > down for ~16 minutes. This scares the hell out of me because it means that > during this time we don’t have any redundancy left. > > Our 5 zookeepers are at version 3.5.8, running in libvirt VMs with Centos > 8 and OpenJDK 1.8.0_322. They are all on a separate VM host. I checked for > network issues (with ping, iperf3) but found nothing. It’s not always the > same zookeeper VM(s) that suffers from this problem. It does not happen at > a regular time so there does not seem to be a cron job affecting this > behavior. > > I have some logging from the last time this happened; I’ll add it below. > > I know we’re running an old version of Zookeeper. I have looked through > the release notes but I couldn’t find a fix for this. > > I’d welcome any ideas. How could we prevent this? Based on the > SocketTimeoutException, I’d think there might be a network hiccup; but why > is it so difficult for Zookeeper to recover from this? Is it a bug in > Zookeeper? Should I report it on https://issues.apache.org? > > Kind regards, > Onno > > [root@feszoo4 /var/log]# grep 'Oct 11 16:01:4.*zookeeper' > messages-20221016 > Oct 11 16:01:43 feszoo4 zookeeper[1199]: 2022-10-11 16:01:43,837 - WARN > [QuorumPeer[myid=4](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):Follower@96] > - Exception when following the leader > Oct 11 16:01:43 feszoo4 zookeeper[1199]: java.net.SocketTimeoutException: > Read timed out > ... > > > [root@feszoo3 /var/log]# grep 'Oct 11 16:01:4.*zookeeper' > messages-20221016 > Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,838 - WARN > [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):Follower@96] > - Exception when following the leader > Oct 11 16:01:43 feszoo3 zookeeper[1187]: java.net.SocketTimeoutException: > Read timed out > Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at > java.net.SocketInputStream.socketRead0(Native Method) > Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at > java.net.SocketInputStream.socketRead(SocketInputStream.java:116) > Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at > java.net.SocketInputStream.read(SocketInputStream.java:171) > Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at > java.net.SocketInputStream.read(SocketInputStream.java:141) > Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at > java.io.BufferedInputStream.fill(BufferedInputStream.java:246) > Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at > java.io.BufferedInputStream.read(BufferedInputStream.java:265) > Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at > java.io.DataInputStream.readInt(DataInputStream.java:387) > Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at > org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:84) > Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at > org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:86) > Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at > org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:118) > Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at > org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:158) > Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at > org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:92) > Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at > org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1253) > Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,839 - INFO > [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):Follower@201] > - shutdown called > Oct 11 16:01:43 feszoo3 zookeeper[1187]: java.lang.Exception: shutdown > Follower > Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at > org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:201) > Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at > org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1257) > Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,866 - INFO > [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 > (message format version), 2 (n.leader), 0x8a00010a28 (n.zxid), 0x52 > (n.round), LOOKING (n.state), 2 (n.sid), 0x8a (n.peerEPoch), FOLLOWING (my > state)0 (n.config version) > Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,869 - INFO > [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 > (message format version), 1 (n.leader), 0x8a00010a28 (n.zxid), 0x52 > (n.round), LOOKING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), FOLLOWING (my > state)0 (n.config version) > Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,869 - INFO > [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 > (message format version), 2 (n.leader), 0x8a00010a28 (n.zxid), 0x52 > (n.round), LOOKING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), FOLLOWING (my > state)0 (n.config version) > Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,885 - INFO > [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):LearnerZooKeeperServer@165] > - Shutting down > Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,885 - INFO > [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):ZooKeeperServer@558] > - shutting down > Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,885 - INFO > [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FollowerRequestProcessor@139] > - Shutting down > Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,885 - INFO > [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):CommitProcessor@362] > - Shutting down > Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,887 - INFO > [FollowerRequestProcessor:3:FollowerRequestProcessor@110] - > FollowerRequestProcessor exited loop! > Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,887 - INFO > [CommitProcessor:3:CommitProcessor@195] - CommitProcessor exited loop! > Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,888 - INFO > [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FinalRequestProcessor@514] > - shutdown of request processor complete > Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,890 - INFO > [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):SyncRequestProcessor@191] > - Shutting down > Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,890 - INFO > [SyncThread:3:SyncRequestProcessor@169] - SyncRequestProcessor exited! > Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,891 - WARN > [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):QuorumPeer@1300] > - PeerState set to LOOKING > Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,891 - INFO > [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):QuorumPeer@1175] > - LOOKING > Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,891 - INFO > [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@903] > - New election. My id = 3, proposed zxid=0x8a00010a28 > Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,892 - INFO > [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 > (message format version), 3 (n.leader), 0x8a00010a28 (n.zxid), 0x52 > (n.round), LOOKING (n.state), 3 (n.sid), 0x8a (n.peerEPoch), LOOKING (my > state)0 (n.config version) > Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,893 - INFO > [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 > (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 > (n.round), FOLLOWING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), LOOKING (my > state)0 (n.config version) > Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,893 - INFO > [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 > (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 > (n.round), FOLLOWING (n.state), 2 (n.sid), 0x8a (n.peerEPoch), LOOKING (my > state)0 (n.config version) > Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,938 - INFO > [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 > (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52 > (n.round), LOOKING (n.state), 4 (n.sid), 0x8a (n.peerEPoch), LOOKING (my > state)0 (n.config version) > Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,939 - INFO > [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 > (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52 > (n.round), LOOKING (n.state), 3 (n.sid), 0x8a (n.peerEPoch), LOOKING (my > state)0 (n.config version) > Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,940 - INFO > [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 > (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 > (n.round), FOLLOWING (n.state), 2 (n.sid), 0x8a (n.peerEPoch), LOOKING (my > state)0 (n.config version) > Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,941 - INFO > [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 > (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 > (n.round), FOLLOWING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), LOOKING (my > state)0 (n.config version) > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,004 - WARN > [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,019 - WARN > [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,046 - WARN > [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,053 - WARN > [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,057 - WARN > [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,075 - WARN > [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,086 - WARN > [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,090 - WARN > [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,095 - WARN > [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,101 - WARN > [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,113 - WARN > [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,122 - WARN > [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,142 - INFO > [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@937] > - Notification time out: 400 > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,142 - INFO > [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 > (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52 > (n.round), LOOKING (n.state), 3 (n.sid), 0x8a (n.peerEPoch), LOOKING (my > state)0 (n.config version) > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,143 - INFO > [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 > (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52 > (n.round), LOOKING (n.state), 4 (n.sid), 0x8a (n.peerEPoch), LOOKING (my > state)0 (n.config version) > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,144 - INFO > [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 > (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 > (n.round), FOLLOWING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), LOOKING (my > state)0 (n.config version) > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,144 - INFO > [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 > (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 > (n.round), FOLLOWING (n.state), 2 (n.sid), 0x8a (n.peerEPoch), LOOKING (my > state)0 (n.config version) > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,149 - WARN > [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,152 - WARN > [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,158 - WARN > [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,159 - WARN > [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,162 - WARN > [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,167 - WARN > [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,179 - WARN > [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,186 - WARN > [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,198 - WARN > [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,217 - WARN > [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,228 - WARN > [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,234 - WARN > [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,235 - WARN > [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,241 - WARN > [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,242 - WARN > [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,246 - WARN > [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,250 - WARN > [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,255 - WARN > [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,255 - WARN > [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,260 - WARN > [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,265 - WARN > [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,276 - WARN > [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,280 - WARN > [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,288 - WARN > [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,300 - WARN > [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,307 - WARN > [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,312 - WARN > [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,318 - WARN > [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,322 - WARN > [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,322 - WARN > [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,325 - WARN > [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,344 - WARN > [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,351 - WARN > [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,375 - WARN > [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,377 - WARN > [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,379 - WARN > [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,379 - WARN > [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,381 - WARN > [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,394 - WARN > [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,395 - WARN > [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,408 - WARN > [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,419 - WARN > [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,419 - WARN > [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,424 - WARN > [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,432 - WARN > [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,433 - WARN > [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,439 - WARN > [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,444 - WARN > [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,476 - WARN > [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,482 - WARN > [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,486 - WARN > [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,490 - WARN > [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,495 - WARN > [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,525 - WARN > [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,526 - WARN > [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,535 - WARN > [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,537 - WARN > [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,540 - WARN > [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,541 - WARN > [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,544 - WARN > [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,545 - WARN > [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,545 - INFO > [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@937] > - Notification time out: 800 > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,546 - INFO > [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 > (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52 > (n.round), LOOKING (n.state), 3 (n.sid), 0x8a (n.peerEPoch), LOOKING (my > state)0 (n.config version) > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,546 - INFO > [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 > (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 > (n.round), FOLLOWING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), LOOKING (my > state)0 (n.config version) > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,547 - INFO > [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 > (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52 > (n.round), LOOKING (n.state), 4 (n.sid), 0x8a (n.peerEPoch), LOOKING (my > state)0 (n.config version) > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,547 - INFO > [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 > (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 > (n.round), FOLLOWING (n.state), 2 (n.sid), 0x8a (n.peerEPoch), LOOKING (my > state)0 (n.config version) > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,553 - WARN > [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,556 - WARN > [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,562 - WARN > [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,580 - WARN > [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,596 - WARN > [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,597 - WARN > [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,607 - WARN > [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,625 - WARN > [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,632 - WARN > [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,638 - WARN > [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,646 - WARN > [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,648 - WARN > [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,667 - WARN > [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,673 - WARN > [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,680 - WARN > [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,690 - WARN > [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,693 - WARN > [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,697 - WARN > [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,708 - WARN > [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,712 - WARN > [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,716 - WARN > [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,723 - WARN > [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,727 - WARN > [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,734 - WARN > [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,740 - WARN > [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,744 - WARN > [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,752 - WARN > [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,765 - WARN > [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,774 - WARN > [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,778 - WARN > [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,803 - WARN > [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,804 - WARN > [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,821 - WARN > [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,832 - WARN > [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,836 - WARN > [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,837 - WARN > [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,854 - WARN > [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,862 - WARN > [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,867 - WARN > [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,872 - WARN > [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,874 - WARN > [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,883 - WARN > [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,886 - WARN > [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,888 - WARN > [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,892 - WARN > [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,895 - WARN > [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,909 - WARN > [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,913 - WARN > [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,920 - WARN > [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,920 - WARN > [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,934 - WARN > [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,938 - WARN > [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,939 - WARN > [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,939 - WARN > [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,947 - WARN > [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,951 - WARN > [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,956 - WARN > [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,964 - WARN > [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,976 - WARN > [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,980 - WARN > [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,012 - WARN > [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,027 - WARN > [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,036 - WARN > [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,070 - WARN > [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,090 - WARN > [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,127 - WARN > [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,197 - WARN > [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,348 - INFO > [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@937] > - Notification time out: 1600 > Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,349 - INFO > [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 > (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52 > (n.round), LOOKING (n.state), 3 (n.sid), 0x8a (n.peerEPoch), LOOKING (my > state)0 (n.config version) > Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,350 - INFO > [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 > (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 > (n.round), FOLLOWING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), LOOKING (my > state)0 (n.config version) > Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,350 - INFO > [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 > (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52 > (n.round), LOOKING (n.state), 4 (n.sid), 0x8a (n.peerEPoch), LOOKING (my > state)0 (n.config version) > Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,351 - INFO > [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 > (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 > (n.round), FOLLOWING (n.state), 2 (n.sid), 0x8a (n.peerEPoch), LOOKING (my > state)0 (n.config version) > Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,356 - WARN > [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,385 - WARN > [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,451 - WARN > [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,561 - WARN > [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,563 - WARN > [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,576 - WARN > [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,595 - WARN > [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,639 - WARN > [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,654 - WARN > [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,685 - WARN > [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,703 - WARN > [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,744 - WARN > [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,753 - WARN > [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,014 - WARN > [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,045 - WARN > [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,062 - WARN > [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,121 - WARN > [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,141 - WARN > [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,151 - WARN > [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,224 - WARN > [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,254 - WARN > [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,322 - WARN > [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,398 - WARN > [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,435 - WARN > [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,522 - WARN > [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,541 - WARN > [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,817 - WARN > [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,873 - WARN > [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,897 - WARN > [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,914 - WARN > [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,951 - INFO > [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@937] > - Notification time out: 3200 > Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,952 - INFO > [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 > (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52 > (n.round), LOOKING (n.state), 3 (n.sid), 0x8a (n.peerEPoch), LOOKING (my > state)0 (n.config version) > Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,954 - INFO > [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 > (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52 > (n.round), LOOKING (n.state), 4 (n.sid), 0x8a (n.peerEPoch), LOOKING (my > state)0 (n.config version) > Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,954 - INFO > [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 > (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 > (n.round), FOLLOWING (n.state), 2 (n.sid), 0x8a (n.peerEPoch), LOOKING (my > state)0 (n.config version) > Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,955 - INFO > [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 > (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 > (n.round), FOLLOWING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), LOOKING (my > state)0 (n.config version) > Oct 11 16:01:47 feszoo3 zookeeper[1187]: 2022-10-11 16:01:47,582 - WARN > [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of > session 0x0: ZooKeeperServer not running > > > [root@feszoo3 ~]# cat /etc/zookeeper/zoo.cfg > # The number of milliseconds of each tick > tickTime=2000 > # The number of ticks that the initial > # synchronization phase can take > initLimit=10 > # The number of ticks that can pass between > # sending a request and getting an acknowledgement > syncLimit=5 > # The directory where the snapshot is stored. > dataDir=/var/lib/zookeeper/data > # The port at which the clients will connect > clientPort=2181 > skipACL=1 > # 4-letter words: which commands are allowed > 4lw.commands.whitelist=* > # Cluster topo > server.1=feszoo1:2888:3888 > server.2=feszoo2:2888:3888 > server.3=feszoo3:2888:3888 > server.4=feszoo4:2888:3888 > server.5=feszoo5:2888:3888 > >
