[ https://issues.apache.org/jira/browse/ZOOKEEPER-4003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17843106#comment-17843106 ]
suhani gupta commented on ZOOKEEPER-4003: ----------------------------------------- [~ztzg] I am seeing this issue in 3.5.9. At this point, I don't know how to move forward without the fix. > Zookeeper server breakdown Frequently > ------------------------------------- > > Key: ZOOKEEPER-4003 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-4003 > Project: ZooKeeper > Issue Type: Bug > Components: server > Affects Versions: 3.5.1 > Environment: zookeeper verison 3.5.1 > > Reporter: xiaotong.wang > Priority: Critical > Attachments: image-2020-11-13-16-51-11-960.png, > image-2020-11-14-18-08-18-126.png, image-2020-11-14-18-10-14-384.png, > jmap.PNG, screenshot-1.png > > > *error log* > WARN [New I/O worker #16:NettyServerCnxn@400] - Closing connection to > /x.x.x.x:43766 > java.io.IOException: ZK down > at > org.apache.zookeeper.server.NettyServerCnxn.receiveMessage(NettyServerCnxn.java:337) > at > org.apache.zookeeper.server.NettyServerCnxnFactory$CnxnChannelHandler.processMessage(NettyServerCnxnFactory.java:243) > at > org.apache.zookeeper.server.NettyServerCnxnFactory$CnxnChannelHandler.messageReceived(NettyServerCnxnFactory.java:165) > at > org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) > at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) > at > org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) > at > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) > at > org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) > at > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > at > org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:748) > > and > > > [myid:2] - WARN [New I/O worker > #15:NettyServerCnxnFactory$CnxnChannelHandler@141] - Exception caught [id: > 0x9ba504cb, /x.x.x.x:39780 :> /x.x.x.x:2181] EXCEPTION: > java.nio.channels.ClosedChannelException > java.nio.channels.ClosedChannelException > at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:270) > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:461) > at > org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) > at > org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201) > at > org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151) > at > org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:292) > at > org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:391) > at > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:315) > at > org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) > at > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > at > org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:748) > > netstat -an|grep 2181|grep CLOSE_WAIT|wc -l > *28441* > > sample: > !image-2020-11-13-16-51-11-960.png! > -- This message was sent by Atlassian Jira (v8.20.10#820010)