[jira] [Closed] (SSHD-939) SSH threads get blocked in Object.wait() method forever resulting in CLOSE_WAIT connections
[
https://issues.apache.org/jira/browse/SSHD-939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Goldstein Lyor closed SSHD-939.
---
Resolution: Won't Fix
> SSH threads get blocked in Object.wait() method forever resulting in
> CLOSE_WAIT connections
> ---
>
> Key: SSHD-939
> URL: https://issues.apache.org/jira/browse/SSHD-939
> Project: MINA SSHD
> Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Anudeep
>Assignee: Goldstein Lyor
>Priority: Major
> Attachments: svc-1-bnsiserv_06_09_2019-server.7z,
> svc-1-bnsiserv_thread.log
>
>
> Ssh threads get blocked in object.wait() for ever and if the count of such
> threads crosses 3 no more ssh requests are served.
> We had an analysis of the logs and from the thread model of apache we see the
> maximum number of I/O threads is number of cores +1 in our server the number
> of cores is 2.
> SO after three such blocked threads we are not able to open any new ssh
> connection.
> Can you please confirm if the above logic is correct .
>
> We too found a ticket raised in 0.14.0
> (https://issues.apache.org/jira/browse/SSHD-348)which says threads in blocked
> state are fixed with 0.14.0 so we need a confirmation from you if 0.14.0 will
> fix the threads to be released and do not hung the process even if there are
> more than 3 of such a kind.
>
> Logs from thread dumps
> {code:java}
> From the thread dump it looks like we have a deadlock, e.g.
> In one thread we're trying to send out data on an SSH connection, but we're
> waiting for "space"
> "pool-7-thread-2213" #2677 prio=5 os_prio=0 tid=0x7f85554cc800 nid=0x61e3
> in Object.wait() [0x7f84ea29f000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.sshd.common.channel.Window.waitForSpace(Window.java:148)
> - locked <0x000764942ad8> (a
> org.apache.sshd.common.channel.Window)
> at
> org.apache.sshd.common.channel.ChannelOutputStream.flush(ChannelOutputStream.java:116)
> - eliminated <0x000764942c20> (a
> org.apache.sshd.common.channel.ChannelOutputStream)
> at
> org.apache.sshd.common.channel.ChannelOutputStream.write(ChannelOutputStream.java:84)
> - locked <0x000764942c20> (a
> org.apache.sshd.common.channel.ChannelOutputStream)
> at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
> at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
> at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
> - locked <0x000764943500> (a java.io.OutputStreamWriter)
> at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207)
> at java.io.BufferedWriter.write(BufferedWriter.java:188)
> - locked <0x000764943500> (a java.io.OutputStreamWriter)
> at java.io.PrintWriter.write(PrintWriter.java:426)
> - locked <0x0007649434d8> (a java.io.BufferedWriter)
> at java.io.PrintWriter.write(PrintWriter.java:443)
> In another thread, we trying to close the same connection
> "sshd-SshServer[1ae27e4e]-nio2-thread-3" #356 prio=5 os_prio=0
> tid=0x015f6000 nid=0x30b8 waiting for monitor entry
> [0x7f851ab9c000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.sshd.common.channel.ChannelOutputStream.close(ChannelOutputStream.java:146)
> - waiting to lock <0x000764942c20> (a
> org.apache.sshd.common.channel.ChannelOutputStream)
> at
> com.ericsson.oss.nbi.connection.ssh.SshCommandContext.shutdown(SshCommandContext.java:146)
> at
> com.ericsson.oss.nbi.connection.ssh.SshCommand$ContextStarter.shutdown(SshCommand.java:104)
> at
> com.ericsson.oss.nbi.connection.ssh.SshCommand.destroy(SshCommand.java:78)
> at
> org.apache.sshd.server.channel.ChannelSession.doCloseImmediately(ChannelSession.java:242)
> at
> org.apache.sshd.common.util.CloseableUtils$AbstractCloseable.close(CloseableUtils.java:276)
> at
> org.apache.sshd.common.util.CloseableUtils$ParallelCloseable.doClose(CloseableUtils.java:182)
> at
> org.apache.sshd.common.util.CloseableUtils$SimpleCloseable.close(CloseableUtils.java:151)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$1.onFailed(Nio2Session.java:199)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$2.run(Nio2CompletionHandler.java:41)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.failed(Nio2CompletionHandler.java:39)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:1
[jira] [Closed] (SSHD-939) SSH threads get blocked in Object.wait() method forever resulting in CLOSE_WAIT connections
[
https://issues.apache.org/jira/browse/SSHD-939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Goldstein Lyor closed SSHD-939.
---
Assignee: Goldstein Lyor
Resolution: Won't Fix
> SSH threads get blocked in Object.wait() method forever resulting in
> CLOSE_WAIT connections
> ---
>
> Key: SSHD-939
> URL: https://issues.apache.org/jira/browse/SSHD-939
> Project: MINA SSHD
> Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Anudeep
>Assignee: Goldstein Lyor
>Priority: Major
> Attachments: svc-1-bnsiserv_06_09_2019-server.7z,
> svc-1-bnsiserv_thread.log
>
>
> Ssh threads get blocked in object.wait() for ever and if the count of such
> threads crosses 3 no more ssh requests are served.
> We had an analysis of the logs and from the thread model of apache we see the
> maximum number of I/O threads is number of cores +1 in our server the number
> of cores is 2.
> SO after three such blocked threads we are not able to open any new ssh
> connection.
> Can you please confirm if the above logic is correct .
>
> We too found a ticket raised in 0.14.0
> (https://issues.apache.org/jira/browse/SSHD-348)which says threads in blocked
> state are fixed with 0.14.0 so we need a confirmation from you if 0.14.0 will
> fix the threads to be released and do not hung the process even if there are
> more than 3 of such a kind.
>
> Logs from thread dumps
> {code:java}
> From the thread dump it looks like we have a deadlock, e.g.
> In one thread we're trying to send out data on an SSH connection, but we're
> waiting for "space"
> "pool-7-thread-2213" #2677 prio=5 os_prio=0 tid=0x7f85554cc800 nid=0x61e3
> in Object.wait() [0x7f84ea29f000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.sshd.common.channel.Window.waitForSpace(Window.java:148)
> - locked <0x000764942ad8> (a
> org.apache.sshd.common.channel.Window)
> at
> org.apache.sshd.common.channel.ChannelOutputStream.flush(ChannelOutputStream.java:116)
> - eliminated <0x000764942c20> (a
> org.apache.sshd.common.channel.ChannelOutputStream)
> at
> org.apache.sshd.common.channel.ChannelOutputStream.write(ChannelOutputStream.java:84)
> - locked <0x000764942c20> (a
> org.apache.sshd.common.channel.ChannelOutputStream)
> at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
> at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
> at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
> - locked <0x000764943500> (a java.io.OutputStreamWriter)
> at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207)
> at java.io.BufferedWriter.write(BufferedWriter.java:188)
> - locked <0x000764943500> (a java.io.OutputStreamWriter)
> at java.io.PrintWriter.write(PrintWriter.java:426)
> - locked <0x0007649434d8> (a java.io.BufferedWriter)
> at java.io.PrintWriter.write(PrintWriter.java:443)
> In another thread, we trying to close the same connection
> "sshd-SshServer[1ae27e4e]-nio2-thread-3" #356 prio=5 os_prio=0
> tid=0x015f6000 nid=0x30b8 waiting for monitor entry
> [0x7f851ab9c000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.sshd.common.channel.ChannelOutputStream.close(ChannelOutputStream.java:146)
> - waiting to lock <0x000764942c20> (a
> org.apache.sshd.common.channel.ChannelOutputStream)
> at
> com.ericsson.oss.nbi.connection.ssh.SshCommandContext.shutdown(SshCommandContext.java:146)
> at
> com.ericsson.oss.nbi.connection.ssh.SshCommand$ContextStarter.shutdown(SshCommand.java:104)
> at
> com.ericsson.oss.nbi.connection.ssh.SshCommand.destroy(SshCommand.java:78)
> at
> org.apache.sshd.server.channel.ChannelSession.doCloseImmediately(ChannelSession.java:242)
> at
> org.apache.sshd.common.util.CloseableUtils$AbstractCloseable.close(CloseableUtils.java:276)
> at
> org.apache.sshd.common.util.CloseableUtils$ParallelCloseable.doClose(CloseableUtils.java:182)
> at
> org.apache.sshd.common.util.CloseableUtils$SimpleCloseable.close(CloseableUtils.java:151)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$1.onFailed(Nio2Session.java:199)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$2.run(Nio2CompletionHandler.java:41)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.failed(Nio2CompletionHandler.java:39)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$1
