[ https://issues.apache.org/jira/browse/ZOOKEEPER-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13097289#comment-13097289 ]
Thomas Koch commented on ZOOKEEPER-939: --------------------------------------- Since ZOOKEEPER-880 has been fixed, this one should also be closed since it was a duplicate. > the threads number of a zookeeper is increased all the time > ----------------------------------------------------------- > > Key: ZOOKEEPER-939 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-939 > Project: ZooKeeper > Issue Type: Bug > Components: server > Affects Versions: 3.3.0 > Environment: Linux 2.6.9-52bs #2 SMP Fri Jan 26 13:34:38 CST 2007 > x86_64 x86_64 x86_64 GNU/Linux > Reporter: Qian Ye > > I have a group of zookeeper servers, there are three servers in this group. > server.0=10.81.4.11:2888:3888 > server.1=10.23.240.93:2888:3888 > server.2=10.23.244.224:2888:3888 > At first, the cluster ran well. About several days ago, I shut down the > zookeeper process on one of servers(server.2)., and today, I find that the > other two servers run in wired status(the network is fine). The zookeeper > process take pretty much resource on the two servers: > on server.1 (it's the leader) > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > > 26836 work 18 0 12.8g 803m 8724 S 3.7 10.1 195:56.56 java > $ ll /proc/26836/fd/ | wc -l > 3586 > [w...@tc-test-aos03.tc.baidu.com conf]$ ll /proc/26836/task/ | wc -l > 10510 > some warning log: > 2010-11-24 15:37:48,705 - WARN > [Thread-37409:QuorumCnxManager$SendWorker@589] - Send worker leaving thread > 2010-11-24 15:39:48,626 - WARN > [Thread-37414:QuorumCnxManager$RecvWorker@658] - Connection broken: > java.nio.channels.AsynchronousCloseException > at > java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185) > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263) > at > org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629) > 2010-11-24 15:39:48,656 - WARN > [Thread-37413:QuorumCnxManager$SendWorker@581] - Interrupted while waiting > for message on queue > java.lang.InterruptedException > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976) > at > java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342) > at > org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570) > 2010-11-24 15:39:48,657 - WARN > [Thread-37413:QuorumCnxManager$SendWorker@589] - Send worker leaving thread > 2010-11-24 15:41:48,614 - WARN > [Thread-37417:QuorumCnxManager$SendWorker@581] - Interrupted while waiting > for message on queue > java.lang.InterruptedException > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976) > at > java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342) > at > org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570) > 2010-11-24 15:41:48,643 - WARN > [Thread-37418:QuorumCnxManager$RecvWorker@658] - Connection broken: > java.nio.channels.AsynchronousCloseException > at > java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185) > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263) > at > org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629) > 2010-11-24 15:41:48,662 - WARN > [Thread-37417:QuorumCnxManager$SendWorker@589] - Send worker leaving thread > 2010-11-24 15:43:48,627 - WARN > [Thread-37421:QuorumCnxManager$SendWorker@581] - Interrupted while waiting > for message on queue > java.lang.InterruptedException > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976) > at > java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342) > at > org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570) > 2010-11-24 15:43:48,627 - WARN > [Thread-37422:QuorumCnxManager$RecvWorker@658] - Connection broken: > java.nio.channels.AsynchronousCloseException > at > java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185) > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263) > at > org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629) > 2010-11-24 15:43:48,654 - WARN > [Thread-37421:QuorumCnxManager$SendWorker@589] - Send worker leaving thread > 2010-11-24 15:44:48,622 - WARN > [Thread-37424:QuorumCnxManager$RecvWorker@658] - Connection broken: > java.nio.channels.AsynchronousCloseException > at > java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185) > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263) > at > org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629) > 2010-11-24 15:44:48,652 - WARN > [Thread-37423:QuorumCnxManager$SendWorker@581] - Interrupted while waiting > for message on queue > java.lang.InterruptedException > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976) > at > java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342) > at > org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570) > 2010-11-24 15:44:48,653 - WARN > [Thread-37423:QuorumCnxManager$SendWorker@589] - Send worker leaving thread > 2010-11-24 15:45:48,668 - WARN > [Thread-37426:QuorumCnxManager$RecvWorker@658] - Connection broken: > java.io.IOException: Channel eof > at > org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630) > 2010-11-24 15:46:48,647 - WARN > [Thread-37427:QuorumCnxManager$SendWorker@581] - Interrupted while waiting > for message on queue > java.lang.InterruptedException > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976) > at > java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342) > at > org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570) > 2010-11-24 15:46:48,722 - WARN > [Thread-37428:QuorumCnxManager$RecvWorker@658] - Connection broken: > java.nio.channels.AsynchronousCloseException > at > java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185) > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263) > at > org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629) > 2010-11-24 15:46:48,736 - WARN > [Thread-37427:QuorumCnxManager$SendWorker@589] - Send worker leaving thread > 2010-11-24 15:47:48,687 - WARN > [Thread-37430:QuorumCnxManager$RecvWorker@658] - Connection broken: > java.io.IOException: Channel eof > at > org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630) > on server.0 > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > > 27322 work 19 0 15.2g 943m 9140 S 38.6 11.8 1396:51 java > $ ll /proc/27322/fd/ | wc -l > 3587 > $ ll /proc/27322/task/ | wc -l > 12938 > 2010-11-24 15:37:49,269 - WARN > [Thread-37407:QuorumCnxManager$SendWorker@589] - Send worker leaving thread > 2010-11-24 15:39:49,235 - WARN > [Thread-37412:QuorumCnxManager$RecvWorker@658] - Connection broken: > java.io.IOException: Channel eof > at > org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630) > 2010-11-24 15:39:49,410 - WARN > [Thread-37411:QuorumCnxManager$SendWorker@581] - Interrupted while waiting > for message on queue > java.lang.InterruptedException > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976) > at > java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342) > at > org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570) > 2010-11-24 15:39:49,411 - WARN > [Thread-37411:QuorumCnxManager$SendWorker@589] - Send worker leaving thread > 2010-11-24 15:41:49,314 - WARN > [Thread-37416:QuorumCnxManager$RecvWorker@658] - Connection broken: > java.io.IOException: Channel eof > at > org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630) > 2010-11-24 15:41:49,383 - WARN > [Thread-37415:QuorumCnxManager$SendWorker@581] - Interrupted while waiting > for message on queue > java.lang.InterruptedException > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976) > at > java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342) > at > org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570) > 2010-11-24 15:41:49,405 - WARN > [Thread-37415:QuorumCnxManager$SendWorker@589] - Send worker leaving thread > 2010-11-24 15:43:49,372 - WARN > [Thread-37420:QuorumCnxManager$RecvWorker@658] - Connection broken: > java.io.IOException: Channel eof > at > org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630) > 2010-11-24 15:43:49,512 - WARN > [Thread-37419:QuorumCnxManager$SendWorker@581] - Interrupted while waiting > for message on queue > java.lang.InterruptedException > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976) > at > java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342) > at > org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570) > 2010-11-24 15:43:49,513 - WARN > [Thread-37419:QuorumCnxManager$SendWorker@589] - Send worker leaving thread > 2010-11-24 15:44:49,407 - WARN > [Thread-37422:QuorumCnxManager$RecvWorker@658] - Connection broken: > java.io.IOException: Channel eof > at > org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630) > 2010-11-24 15:45:49,645 - WARN > [Thread-37424:QuorumCnxManager$RecvWorker@658] - Connection broken: > java.nio.channels.AsynchronousCloseException > at > java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185) > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263) > at > org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629) > 2010-11-24 15:45:49,781 - WARN > [Thread-37423:QuorumCnxManager$SendWorker@581] - Interrupted while waiting > for message on queue > java.lang.InterruptedException > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976) > at > java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342) > at > org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570) > 2010-11-24 15:45:49,799 - WARN > [Thread-37423:QuorumCnxManager$SendWorker@589] - Send worker leaving thread > 2010-11-24 15:46:49,495 - WARN > [Thread-37427:QuorumCnxManager$RecvWorker@658] - Connection broken: > java.io.IOException: Channel eof > at > org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630) > 2010-11-24 15:47:49,541 - WARN > [Thread-37429:QuorumCnxManager$RecvWorker@658] - Connection broken: > java.nio.channels.AsynchronousCloseException > at > java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185) > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263) > at > org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629) > 2010-11-24 15:47:49,622 - WARN > [Thread-37428:QuorumCnxManager$SendWorker@581] - Interrupted while waiting > for message on queue > java.lang.InterruptedException > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976) > at > java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342) > at > org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570) > 2010-11-24 15:47:49,622 - WARN > [Thread-37428:QuorumCnxManager$SendWorker@589] - Send worker leaving thread > 2010-11-24 15:48:48,827 - WARN > [Thread-37431:QuorumCnxManager$RecvWorker@658] - Connection broken: > java.io.IOException: Channel eof > at > org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630) > What's more, the number of threads under the zookeeper process is still > increasing time by time. It seems that , something is wrong in communication > of the two servers. Have anyone met such problem before? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira