Hi all,
We have a 3 node zookeeper cluster used for Vault as HA. Starting a few
days ago, the entire cluster crashes a few times per day, all nodes at the
exact same time. We are running some load test using vault for Data
encryption. Per minute 1000 keys unique keys will be generated, Once the
issue started around 270,000 keys.
The following exception is got from the syslog, not sure what's causing this
crash. Please help to proceed..
2019-02-26 22:35:18,831 [myid:1] - WARN
[QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:Follower@90] - Exception when
following the leader
java.io.EOFException
at
java.base/java.io.DataInputStream.readFully(DataInputStream.java:202)
at
java.base/java.io.DataInputStream.readFully(DataInputStream.java:170)
at
org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:94)
at
org.apache.zookeeper.server.DataNode.deserialize(DataNode.java:165)
at
org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)
at
org.apache.zookeeper.server.DataTree.deserialize(DataTree.java:1076)
at
org.apache.zookeeper.server.util.SerializeUtils.deserializeSnapshot(SerializeUtils.java:130)
at
org.apache.zookeeper.server.ZKDatabase.deserializeSnapshot(ZKDatabase.java:452)
at
org.apache.zookeeper.server.quorum.Learner.syncWithLeader(Learner.java:340)
at
org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:83)
at
org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:981)
2019-02-26 22:35:19,349 [myid:1] - INFO
[QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:Follower@169] - shutdown called
java.lang.Exception: shutdown Follower
at
org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:169)
thanks
JS
--
Sent from: http://zookeeper-user.578899.n2.nabble.com/
