[ https://issues.apache.org/jira/browse/SSHD-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16929033#comment-16929033 ]
Anudeep commented on SSHD-939: ------------------------------ Hi [~lgoldstein], I am sorry but i need to ask for a clear picture. Cipher here you mean the client one or the server one? What do you mean by decode bug ? The exceptions invalid packet length is seen when we receive a packet from client. Regards, Anudeep > SSH threads get blocked in Object.wait() method forever resulting in > CLOSE_WAIT connections > ------------------------------------------------------------------------------------------- > > Key: SSHD-939 > URL: https://issues.apache.org/jira/browse/SSHD-939 > Project: MINA SSHD > Issue Type: Bug > Affects Versions: 0.13.0 > Reporter: Anudeep > Assignee: Goldstein Lyor > Priority: Major > Attachments: svc-1-bnsiserv_06_09_2019-server.7z, > svc-1-bnsiserv_thread.log > > > Ssh threads get blocked in object.wait() for ever and if the count of such > threads crosses 3 no more ssh requests are served. > We had an analysis of the logs and from the thread model of apache we see the > maximum number of I/O threads is number of cores +1 in our server the number > of cores is 2. > SO after three such blocked threads we are not able to open any new ssh > connection. > Can you please confirm if the above logic is correct . > > We too found a ticket raised in 0.14.0 > (https://issues.apache.org/jira/browse/SSHD-348)which says threads in blocked > state are fixed with 0.14.0 so we need a confirmation from you if 0.14.0 will > fix the threads to be released and do not hung the process even if there are > more than 3 of such a kind. > > Logs from thread dumps > {code:java} > From the thread dump it looks like we have a deadlock, e.g. > In one thread we're trying to send out data on an SSH connection, but we're > waiting for "space" > "pool-7-thread-2213" #2677 prio=5 os_prio=0 tid=0x00007f85554cc800 nid=0x61e3 > in Object.wait() [0x00007f84ea29f000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(Object.java:502) > at org.apache.sshd.common.channel.Window.waitForSpace(Window.java:148) > - locked <0x0000000764942ad8> (a > org.apache.sshd.common.channel.Window) > at > org.apache.sshd.common.channel.ChannelOutputStream.flush(ChannelOutputStream.java:116) > - eliminated <0x0000000764942c20> (a > org.apache.sshd.common.channel.ChannelOutputStream) > at > org.apache.sshd.common.channel.ChannelOutputStream.write(ChannelOutputStream.java:84) > - locked <0x0000000764942c20> (a > org.apache.sshd.common.channel.ChannelOutputStream) > at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221) > at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282) > at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125) > - locked <0x0000000764943500> (a java.io.OutputStreamWriter) > at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207) > at java.io.BufferedWriter.write(BufferedWriter.java:188) > - locked <0x0000000764943500> (a java.io.OutputStreamWriter) > at java.io.PrintWriter.write(PrintWriter.java:426) > - locked <0x00000007649434d8> (a java.io.BufferedWriter) > at java.io.PrintWriter.write(PrintWriter.java:443) > In another thread, we trying to close the same connection > "sshd-SshServer[1ae27e4e]-nio2-thread-3" #356 prio=5 os_prio=0 > tid=0x00000000015f6000 nid=0x30b8 waiting for monitor entry > [0x00007f851ab9c000] > java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.sshd.common.channel.ChannelOutputStream.close(ChannelOutputStream.java:146) > - waiting to lock <0x0000000764942c20> (a > org.apache.sshd.common.channel.ChannelOutputStream) > at > com.ericsson.oss.nbi.connection.ssh.SshCommandContext.shutdown(SshCommandContext.java:146) > at > com.ericsson.oss.nbi.connection.ssh.SshCommand$ContextStarter.shutdown(SshCommand.java:104) > at > com.ericsson.oss.nbi.connection.ssh.SshCommand.destroy(SshCommand.java:78) > at > org.apache.sshd.server.channel.ChannelSession.doCloseImmediately(ChannelSession.java:242) > at > org.apache.sshd.common.util.CloseableUtils$AbstractCloseable.close(CloseableUtils.java:276) > at > org.apache.sshd.common.util.CloseableUtils$ParallelCloseable.doClose(CloseableUtils.java:182) > at > org.apache.sshd.common.util.CloseableUtils$SimpleCloseable.close(CloseableUtils.java:151) > .... at > org.apache.sshd.common.io.nio2.Nio2Session$1.onFailed(Nio2Session.java:199) > at > org.apache.sshd.common.io.nio2.Nio2CompletionHandler$2.run(Nio2CompletionHandler.java:41) > at java.security.AccessController.doPrivileged(Native Method) > at > org.apache.sshd.common.io.nio2.Nio2CompletionHandler.failed(Nio2CompletionHandler.java:39) > at > org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:195) > at > org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:170) > at > org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:32) > at java.security.AccessController.doPrivileged(Native Method) > at > org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:30) > at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126) > at sun.nio.ch.Invoker$2.run(Invoker.java:218) > at > sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {code} > > Theory for the number of I/O process threads > {code:java} > https://cwiki.apache.org/confluence/display/MINA/Configuring+Thread+Model > I/O processor thread performs the actual read and write operation until the > connection is closed.Each SocketAcceptor or SocketConnector creates its own > I/O processor thread(s). You can configure the number of the I/O processor > threads. The default maximum number of the I/O processor threads is the > number of CPU cores + 1. {code} > > > > > -- This message was sent by Atlassian Jira (v8.3.2#803003) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@mina.apache.org For additional commands, e-mail: dev-h...@mina.apache.org