Hi JS, Looks like there was a Leader election, and during sync phase (syncWithLeader), the follower tried to deserialize the snapshot, but it is an incomplete file, hence the EOF exception. How big is your snapshot? Did you run out of disc space? Also worth checking for fsync warnings / errors in the log.
Hope this helps. Regards, Norbert On Wed, Feb 27, 2019 at 8:05 AM zoo_js <[email protected]> wrote: > Hi all, > > We have a 3 node zookeeper cluster used for Vault as HA. Starting a few > days ago, the entire cluster crashes a few times per day, all nodes at the > exact same time. We are running some load test using vault for Data > encryption. Per minute 1000 keys unique keys will be generated, Once the > issue started around 270,000 keys. > > The following exception is got from the syslog, not sure what's causing > this > crash. Please help to proceed.. > > 2019-02-26 22:35:18,831 [myid:1] - WARN > [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:Follower@90] - Exception when > following the leader > java.io.EOFException > at > java.base/java.io.DataInputStream.readFully(DataInputStream.java:202) > at > java.base/java.io.DataInputStream.readFully(DataInputStream.java:170) > at > org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:94) > at > org.apache.zookeeper.server.DataNode.deserialize(DataNode.java:165) > at > org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99) > at > org.apache.zookeeper.server.DataTree.deserialize(DataTree.java:1076) > at > > org.apache.zookeeper.server.util.SerializeUtils.deserializeSnapshot(SerializeUtils.java:130) > at > > org.apache.zookeeper.server.ZKDatabase.deserializeSnapshot(ZKDatabase.java:452) > at > org.apache.zookeeper.server.quorum.Learner.syncWithLeader(Learner.java:340) > at > org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:83) > at > org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:981) > 2019-02-26 22:35:19,349 [myid:1] - INFO > [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:Follower@169] - shutdown called > java.lang.Exception: shutdown Follower > at > org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:169) > > thanks > JS > > > > > > > -- > Sent from: http://zookeeper-user.578899.n2.nabble.com/ >
