[
https://issues.apache.org/jira/browse/SSHD-939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anudeep updated SSHD-939:
-------------------------
Description:
Ssh threads get blocked in object.wait() for ever and if the count of such
threads crosses 3 no more ssh requests are served.
We had an analysis of the logs and from the thread model of apache we see the
maximum number of I/O threads is number of cores +1 in our server the number of
cores is 2.
SO after three such blocked threads we are not able to open any new ssh
connection.
Can you please confirm if the above logic is correct .
We too found a ticket raised in 0.14.0
(https://issues.apache.org/jira/browse/SSHD-348)which says threads in blocked
state are fixed with 0.14.0 so we need a confirmation from you if 0.14.0 will
fix the threads to be released and do not hung the process even if there are
more than 3 of such a kind.
Logs from thread dumps
{code:java}
>From the thread dump it looks like we have a deadlock, e.g.
In one thread we're trying to send out data on an SSH connection, but we're
waiting for "space"
"pool-7-thread-2213" #2677 prio=5 os_prio=0 tid=0x00007f85554cc800 nid=0x61e3
in Object.wait() [0x00007f84ea29f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.apache.sshd.common.channel.Window.waitForSpace(Window.java:148)
- locked <0x0000000764942ad8> (a org.apache.sshd.common.channel.Window)
at
org.apache.sshd.common.channel.ChannelOutputStream.flush(ChannelOutputStream.java:116)
- eliminated <0x0000000764942c20> (a
org.apache.sshd.common.channel.ChannelOutputStream)
at
org.apache.sshd.common.channel.ChannelOutputStream.write(ChannelOutputStream.java:84)
- locked <0x0000000764942c20> (a
org.apache.sshd.common.channel.ChannelOutputStream)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
- locked <0x0000000764943500> (a java.io.OutputStreamWriter)
at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207)
at java.io.BufferedWriter.write(BufferedWriter.java:188)
- locked <0x0000000764943500> (a java.io.OutputStreamWriter)
at java.io.PrintWriter.write(PrintWriter.java:426)
- locked <0x00000007649434d8> (a java.io.BufferedWriter)
at java.io.PrintWriter.write(PrintWriter.java:443)
In another thread, we trying to close the same connection
"sshd-SshServer[1ae27e4e]-nio2-thread-3" #356 prio=5 os_prio=0
tid=0x00000000015f6000 nid=0x30b8 waiting for monitor entry [0x00007f851ab9c000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.sshd.common.channel.ChannelOutputStream.close(ChannelOutputStream.java:146)
- waiting to lock <0x0000000764942c20> (a
org.apache.sshd.common.channel.ChannelOutputStream)
at
com.ericsson.oss.nbi.connection.ssh.SshCommandContext.shutdown(SshCommandContext.java:146)
at
com.ericsson.oss.nbi.connection.ssh.SshCommand$ContextStarter.shutdown(SshCommand.java:104)
at
com.ericsson.oss.nbi.connection.ssh.SshCommand.destroy(SshCommand.java:78)
at
org.apache.sshd.server.channel.ChannelSession.doCloseImmediately(ChannelSession.java:242)
at
org.apache.sshd.common.util.CloseableUtils$AbstractCloseable.close(CloseableUtils.java:276)
at
org.apache.sshd.common.util.CloseableUtils$ParallelCloseable.doClose(CloseableUtils.java:182)
at
org.apache.sshd.common.util.CloseableUtils$SimpleCloseable.close(CloseableUtils.java:151)
.... at
org.apache.sshd.common.io.nio2.Nio2Session$1.onFailed(Nio2Session.java:199)
at
org.apache.sshd.common.io.nio2.Nio2CompletionHandler$2.run(Nio2CompletionHandler.java:41)
at java.security.AccessController.doPrivileged(Native Method)
at
org.apache.sshd.common.io.nio2.Nio2CompletionHandler.failed(Nio2CompletionHandler.java:39)
at
org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:195)
at
org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:170)
at
org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:32)
at java.security.AccessController.doPrivileged(Native Method)
at
org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:30)
at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
at sun.nio.ch.Invoker$2.run(Invoker.java:218)
at
sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
Theory for the number of I/O process threads
{code:java}
https://cwiki.apache.org/confluence/display/MINA/Configuring+Thread+Model
I/O processor thread performs the actual read and write operation until the
connection is closed.Each SocketAcceptor or SocketConnector creates its own I/O
processor thread(s). You can configure the number of the I/O processor threads.
The default maximum number of the I/O processor threads is the number of CPU
cores + 1. {code}
was:
Ssh threads get blocked in object.wait() for ever and if the count of such
threads crosses 3 no more ssh requests are served.
We had an analysis of the logs and from the thread model of apache we see the
maximum number of I/O threads is number of cores +1 in our server the number of
cores is 2.
SO after three such blocked threads we are not able to open any new ssh
connection.
Can you please confirm if the above logic is correct .
We too found a ticket raised in 0.14.0 which says threads in blocked state are
fixed with 0.14.0 so we need a confirmation from you if 0.14.0 will fix the
threads to release and do not hung the process even if there are more than 3 of
such a kind of threads in blocked state.
Logs from thread dumps
{code:java}
>From the thread dump it looks like we have a deadlock, e.g.
In one thread we're trying to send out data on an SSH connection, but we're
waiting for "space"
"pool-7-thread-2213" #2677 prio=5 os_prio=0 tid=0x00007f85554cc800 nid=0x61e3
in Object.wait() [0x00007f84ea29f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.apache.sshd.common.channel.Window.waitForSpace(Window.java:148)
- locked <0x0000000764942ad8> (a org.apache.sshd.common.channel.Window)
at
org.apache.sshd.common.channel.ChannelOutputStream.flush(ChannelOutputStream.java:116)
- eliminated <0x0000000764942c20> (a
org.apache.sshd.common.channel.ChannelOutputStream)
at
org.apache.sshd.common.channel.ChannelOutputStream.write(ChannelOutputStream.java:84)
- locked <0x0000000764942c20> (a
org.apache.sshd.common.channel.ChannelOutputStream)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
- locked <0x0000000764943500> (a java.io.OutputStreamWriter)
at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207)
at java.io.BufferedWriter.write(BufferedWriter.java:188)
- locked <0x0000000764943500> (a java.io.OutputStreamWriter)
at java.io.PrintWriter.write(PrintWriter.java:426)
- locked <0x00000007649434d8> (a java.io.BufferedWriter)
at java.io.PrintWriter.write(PrintWriter.java:443)
In another thread, we trying to close the same connection
"sshd-SshServer[1ae27e4e]-nio2-thread-3" #356 prio=5 os_prio=0
tid=0x00000000015f6000 nid=0x30b8 waiting for monitor entry [0x00007f851ab9c000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.sshd.common.channel.ChannelOutputStream.close(ChannelOutputStream.java:146)
- waiting to lock <0x0000000764942c20> (a
org.apache.sshd.common.channel.ChannelOutputStream)
at
com.ericsson.oss.nbi.connection.ssh.SshCommandContext.shutdown(SshCommandContext.java:146)
at
com.ericsson.oss.nbi.connection.ssh.SshCommand$ContextStarter.shutdown(SshCommand.java:104)
at
com.ericsson.oss.nbi.connection.ssh.SshCommand.destroy(SshCommand.java:78)
at
org.apache.sshd.server.channel.ChannelSession.doCloseImmediately(ChannelSession.java:242)
at
org.apache.sshd.common.util.CloseableUtils$AbstractCloseable.close(CloseableUtils.java:276)
at
org.apache.sshd.common.util.CloseableUtils$ParallelCloseable.doClose(CloseableUtils.java:182)
at
org.apache.sshd.common.util.CloseableUtils$SimpleCloseable.close(CloseableUtils.java:151)
.... at
org.apache.sshd.common.io.nio2.Nio2Session$1.onFailed(Nio2Session.java:199)
at
org.apache.sshd.common.io.nio2.Nio2CompletionHandler$2.run(Nio2CompletionHandler.java:41)
at java.security.AccessController.doPrivileged(Native Method)
at
org.apache.sshd.common.io.nio2.Nio2CompletionHandler.failed(Nio2CompletionHandler.java:39)
at
org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:195)
at
org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:170)
at
org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:32)
at java.security.AccessController.doPrivileged(Native Method)
at
org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:30)
at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
at sun.nio.ch.Invoker$2.run(Invoker.java:218)
at
sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
Theory for the number of I/O process threads
{code:java}
https://cwiki.apache.org/confluence/display/MINA/Configuring+Thread+Model
I/O processor thread performs the actual read and write operation until the
connection is closed.Each SocketAcceptor or SocketConnector creates its own I/O
processor thread(s). You can configure the number of the I/O processor threads.
The default maximum number of the I/O processor threads is the number of CPU
cores + 1. {code}
> SSH threads get blocked in Object.wait() method forever resulting in
> CLOSE_WAIT connections
> -------------------------------------------------------------------------------------------
>
> Key: SSHD-939
> URL: https://issues.apache.org/jira/browse/SSHD-939
> Project: MINA SSHD
> Issue Type: Bug
> Affects Versions: 0.13.0
> Reporter: Anudeep
> Priority: Major
>
> Ssh threads get blocked in object.wait() for ever and if the count of such
> threads crosses 3 no more ssh requests are served.
> We had an analysis of the logs and from the thread model of apache we see the
> maximum number of I/O threads is number of cores +1 in our server the number
> of cores is 2.
> SO after three such blocked threads we are not able to open any new ssh
> connection.
> Can you please confirm if the above logic is correct .
>
> We too found a ticket raised in 0.14.0
> (https://issues.apache.org/jira/browse/SSHD-348)which says threads in blocked
> state are fixed with 0.14.0 so we need a confirmation from you if 0.14.0 will
> fix the threads to be released and do not hung the process even if there are
> more than 3 of such a kind.
>
> Logs from thread dumps
> {code:java}
> From the thread dump it looks like we have a deadlock, e.g.
> In one thread we're trying to send out data on an SSH connection, but we're
> waiting for "space"
> "pool-7-thread-2213" #2677 prio=5 os_prio=0 tid=0x00007f85554cc800 nid=0x61e3
> in Object.wait() [0x00007f84ea29f000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.sshd.common.channel.Window.waitForSpace(Window.java:148)
> - locked <0x0000000764942ad8> (a
> org.apache.sshd.common.channel.Window)
> at
> org.apache.sshd.common.channel.ChannelOutputStream.flush(ChannelOutputStream.java:116)
> - eliminated <0x0000000764942c20> (a
> org.apache.sshd.common.channel.ChannelOutputStream)
> at
> org.apache.sshd.common.channel.ChannelOutputStream.write(ChannelOutputStream.java:84)
> - locked <0x0000000764942c20> (a
> org.apache.sshd.common.channel.ChannelOutputStream)
> at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
> at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
> at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
> - locked <0x0000000764943500> (a java.io.OutputStreamWriter)
> at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207)
> at java.io.BufferedWriter.write(BufferedWriter.java:188)
> - locked <0x0000000764943500> (a java.io.OutputStreamWriter)
> at java.io.PrintWriter.write(PrintWriter.java:426)
> - locked <0x00000007649434d8> (a java.io.BufferedWriter)
> at java.io.PrintWriter.write(PrintWriter.java:443)
> In another thread, we trying to close the same connection
> "sshd-SshServer[1ae27e4e]-nio2-thread-3" #356 prio=5 os_prio=0
> tid=0x00000000015f6000 nid=0x30b8 waiting for monitor entry
> [0x00007f851ab9c000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.sshd.common.channel.ChannelOutputStream.close(ChannelOutputStream.java:146)
> - waiting to lock <0x0000000764942c20> (a
> org.apache.sshd.common.channel.ChannelOutputStream)
> at
> com.ericsson.oss.nbi.connection.ssh.SshCommandContext.shutdown(SshCommandContext.java:146)
> at
> com.ericsson.oss.nbi.connection.ssh.SshCommand$ContextStarter.shutdown(SshCommand.java:104)
> at
> com.ericsson.oss.nbi.connection.ssh.SshCommand.destroy(SshCommand.java:78)
> at
> org.apache.sshd.server.channel.ChannelSession.doCloseImmediately(ChannelSession.java:242)
> at
> org.apache.sshd.common.util.CloseableUtils$AbstractCloseable.close(CloseableUtils.java:276)
> at
> org.apache.sshd.common.util.CloseableUtils$ParallelCloseable.doClose(CloseableUtils.java:182)
> at
> org.apache.sshd.common.util.CloseableUtils$SimpleCloseable.close(CloseableUtils.java:151)
> .... at
> org.apache.sshd.common.io.nio2.Nio2Session$1.onFailed(Nio2Session.java:199)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$2.run(Nio2CompletionHandler.java:41)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.failed(Nio2CompletionHandler.java:39)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:195)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:170)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:32)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:30)
> at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
> at sun.nio.ch.Invoker$2.run(Invoker.java:218)
> at
> sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
>
> Theory for the number of I/O process threads
> {code:java}
> https://cwiki.apache.org/confluence/display/MINA/Configuring+Thread+Model
> I/O processor thread performs the actual read and write operation until the
> connection is closed.Each SocketAcceptor or SocketConnector creates its own
> I/O processor thread(s). You can configure the number of the I/O processor
> threads. The default maximum number of the I/O processor threads is the
> number of CPU cores + 1. {code}
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
