It's also likely you have a fair bit of packet loss between your datacenters, unless you know you have a solid network between them. If your observers are falling offline "randomly", packet loss is a pretty likely culprit.
On Thu, Mar 9, 2017 at 9:54 AM, Michael Han <[email protected]> wrote: > The log indicates that your server socket on observer timed out after > syncing with leader. It could simply because that the latency between your > DCs exceeds the socket timeout configuration ZK uses. The timeout is > calculated as tickTime * syncLimit so you might want tweak these values to > fit the latency between your DCs. > > On Thu, Mar 9, 2017 at 9:00 AM, rammohan ganapavarapu < > [email protected]> wrote: > > > Hi, > > > > We have a multi data-center zk cluster with all the followers are in one > > data-center and observers in other data-centers, for some reason > observers > > are going down with the following exception and i am not sure what could > be > > the reason and how to avoid this issue, any thoughts? > > > > Ram > > > > > > > > 2017-03-09 09:00:18,305 - WARN > > [QuorumPeer[myid=41]/0:0:0:0:0:0:0:0:2181:Observer@79] - Exception when > > observing the leader > > java.net.SocketTimeoutException: Read timed out > > at java.net.SocketInputStream.socketRead0(Native Method) > > at java.net.SocketInputStream.read(SocketInputStream.java:152) > > at java.net.SocketInputStream.read(SocketInputStream.java:122) > > at java.io.BufferedInputStream.fill(BufferedInputStream.java: > 235) > > at java.io.BufferedInputStream.read(BufferedInputStream.java: > 254) > > at java.io.DataInputStream.readInt(DataInputStream.java:387) > > at > > org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63) > > at > > org.apache.zookeeper.server.quorum.QuorumPacket. > > deserialize(QuorumPacket.java:83) > > at > > org.apache.jute.BinaryInputArchive.readRecord( > BinaryInputArchive.java:108) > > at > > org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:152) > > at > > org.apache.zookeeper.server.quorum.Observer.observeLeader( > > Observer.java:75) > > at > > org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:727) > > 2017-03-09 09:00:18,306 - INFO > > [QuorumPeer[myid=41]/0:0:0:0:0:0:0:0:2181:Observer@137] - shutdown > called > > java.lang.Exception: shutdown Observer > > at > > org.apache.zookeeper.server.quorum.Observer.shutdown(Observer.java:137) > > > > > > -- > Cheers > Michael. >
