[
https://issues.apache.org/jira/browse/SSHD-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17191424#comment-17191424
]
Feng Jiajie commented on SSHD-1070:
-----------------------------------
[~technobcn] Thanks for helping reproduce the issue.
[~wolft] Thanks for helping to fix the problem.
I think there maybe some problems with the commit: [PR
#163|https://github.com/apache/mina-sshd/pull/163/commits/37efbff75fed6eae374c7012277c433f002cd6d7]
BlockingQueue is a blocking queue, whereas SSHD is based on an asynchronous
framework, maybe this modification will cause blocking of asynchronous
processes?
a. Could this cause a deadlock? For example: if all the working threads are
blocking on BlockingQueue.put, the process would fail to provide service,
because there would be no thread (all already blocking on BlockingQueue.put) to
retrieve elements from BlockingQueues.
b. The performance of asynchronous frameworks will be degraded, and fast
connections may be slower in scenarios where both fast and slow connections
exist.
I don't know much about asynchronous frameworks, but I guess it's possible here
to
[offer|https://docs.oracle.com/javase/8/docs/api/java/util/Queue.html#offer-E-]
elements to the queue (unlike the blocking put operation), and if the queue is
full then immediately return the failure and put that "offer operation" into an
operation queue and wait until the next time there's a free working thread to
try to perform the operation?
> OutOfMemoryError when use port forwarding
> -----------------------------------------
>
> Key: SSHD-1070
> URL: https://issues.apache.org/jira/browse/SSHD-1070
> Project: MINA SSHD
> Issue Type: Bug
> Affects Versions: 2.5.1
> Reporter: Feng Jiajie
> Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Hi [~gnodet], I found this commit implemented asynchronous port forwarding:
> [https://github.com/apache/mina-sshd/commit/45f84aab59b2e11d72942cffe9d810e37ab64959#diff-33823b8546f71d77bb1d653358ecde70]
> However, when a large amount of data is returned from upstream application,
> it is possible to cause an OOM in SSHD.
> Step1. SSHD server:
> {code:java}
> import org.apache.sshd.common.FactoryManager;
> import org.apache.sshd.common.PropertyResolverUtils;
> import org.apache.sshd.common.util.security.SecurityUtils;
> import org.apache.sshd.server.SshServer;
> import org.apache.sshd.server.forward.AcceptAllForwardingFilter;
> import org.apache.sshd.server.keyprovider.SimpleGeneratorHostKeyProvider;
> import java.io.IOException;
> import java.nio.file.Paths;
> public class TestSshd2 {
> public static void main(String[] args) throws IOException,
> InterruptedException {
> SecurityUtils.setAPrioriDisabledProvider("BC", true);
> SshServer sshd = SshServer.setUpDefaultServer();
> sshd.setPort(12133);
> sshd.setKeyPairProvider(new
> SimpleGeneratorHostKeyProvider(Paths.get("/tmp/aa.key")));
> sshd.setPasswordAuthenticator((username, password, session) -> true);
> sshd.setForwardingFilter(AcceptAllForwardingFilter.INSTANCE);
> sshd.start();
> Thread.sleep(100000000);
> }
> }
> {code}
> Step2. start ssh client and iperf3 server:
> {code:java}
> ssh -o 'ExitOnForwardFailure yes' -p 12133 -f -x -N -T -L
> 0.0.0.0:15678:127.0.0.1:12345 test5@127.0.0.1
> iperf3 -s -p 12345
> {code}
> Step3. run iperf3 client:
> {code:java}
> iperf3 -c 127.0.0.1 -i 1 -t 120 -p 15678 -P 8 --reverse
> {code}
> *-R, --reverse run in reverse mode (server sends, client receives)*
> SSHD will receive a lot of data but will not be able to forward it in time.
> log when OOM:
> {code:java}
> 17:52:25.195 [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-1] WARN
> org.apache.sshd.common.io.nio2.Nio2Session -
> exceptionCaught(Nio2Session[local=/127.0.0.1:33524, remote=/127.0.0.1:12345])
> Exception handler threw OutOfMemoryError, closing the session: GC overhead
> limit exceeded17:52:25.195
> [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-1] WARN
> org.apache.sshd.common.io.nio2.Nio2Session -
> exceptionCaught(Nio2Session[local=/127.0.0.1:33524, remote=/127.0.0.1:12345])
> Exception handler threw OutOfMemoryError, closing the session: GC overhead
> limit exceeded at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
> at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126) at
> sun.nio.ch.Invoker.invokeDirect(Invoker.java:157) at
> sun.nio.ch.UnixAsynchronousSocketChannelImpl.implWrite(UnixAsynchronousSocketChannelImpl.java:736)
> at
> sun.nio.ch.AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:382)
> at
> sun.nio.ch.AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:399)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.doWriteCycle(Nio2Session.java:420)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.startWriting(Nio2Session.java:404)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.finishWrite(Nio2Session.java:495)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.handleCompletedWriteCycle(Nio2Session.java:465)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:429)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:426)17:52:25.639
> [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-6] WARN
> org.apache.sshd.common.io.nio2.Nio2Session -
> exceptionCaught(Nio2Session[local=/127.0.0.1:33530, remote=/127.0.0.1:12345])
> Exception handler threw OutOfMemoryError, closing the session: GC overhead
> limit exceeded at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
> at java.security.AccessController.doPrivileged(Native Method) at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
> at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126) at
> sun.nio.ch.Invoker.invokeDirect(Invoker.java:157) at
> sun.nio.ch.UnixAsynchronousSocketChannelImpl.implWrite(UnixAsynchronousSocketChannelImpl.java:736)17:52:26.045
> [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-2] DEBUG
> org.apache.sshd.common.io.nio2.Nio2Session -
> exceptionCaught(Nio2Session[local=/127.0.0.1:33522, remote=/127.0.0.1:12345])
> caught OutOfMemoryError[GC overhead limit exceeded] - calling
> handler17:52:26.045 [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-2]
> DEBUG org.apache.sshd.server.forward.TcpipServerChannel -
> exceptionCaught(TcpipServerChannel[id=3,
> recipient=4]-ServerSessionImpl[test5@/127.0.0.1:53702]) signal close
> immediately=false due to OutOfMemoryError[GC overhead limit
> exceeded]17:52:26.045 [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-2]
> DEBUG org.apache.sshd.server.forward.TcpipServerChannel -
> close(TcpipServerChannel[id=3,
> recipient=4]-ServerSessionImpl[test5@/127.0.0.1:53702])[Graceful] state
> already Graceful17:52:26.794
> [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-3] WARN
> org.apache.sshd.common.io.nio2.Nio2Session -
> exceptionCaught(Nio2Session[local=/127.0.0.1:33528, remote=/127.0.0.1:12345])
> Exception handler threw OutOfMemoryError, closing the session: GC overhead
> limit exceeded at
> sun.nio.ch.AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:382)
> at
> sun.nio.ch.AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:399)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.doWriteCycle(Nio2Session.java:420)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.startWriting(Nio2Session.java:404)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.finishWrite(Nio2Session.java:495)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.handleCompletedWriteCycle(Nio2Session.java:465)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:429)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:426)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)17:52:26.045
> [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-2] DEBUG
> org.apache.sshd.common.io.nio2.Nio2Session -
> close(Nio2Session[local=/127.0.0.1:33522, remote=/127.0.0.1:12345]) Closing
> immediately at java.security.AccessController.doPrivileged(Native
> Method)17:52:26.869 [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-2]
> DEBUG org.apache.sshd.common.io.nio2.Nio2Session -
> doCloseImmediately(Nio2Session[local=/127.0.0.1:33522,
> remote=/127.0.0.1:12345]) closing
> socket=sun.nio.ch.UnixAsynchronousSocketChannelImpl[connected
> local=/127.0.0.1:33522 remote=/127.0.0.1:12345] at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
> at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126) at
> sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)17:52:27.568
> [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-6] DEBUG
> org.apache.sshd.common.io.nio2.Nio2Session -
> exceptionCaught(Nio2Session[local=/127.0.0.1:33530, remote=/127.0.0.1:12345])
> caught OutOfMemoryError[GC overhead limit exceeded] - calling
> handler17:52:27.568 [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-6]
> DEBUG org.apache.sshd.server.forward.TcpipServerChannel -
> exceptionCaught(TcpipServerChannel[id=7,
> recipient=8]-ServerSessionImpl[test5@/127.0.0.1:53702]) signal close
> immediately=false due to OutOfMemoryError[GC overhead limit
> exceeded]17:52:27.568 [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-6]
> DEBUG org.apache.sshd.server.forward.TcpipServerChannel -
> close(TcpipServerChannel[id=7,
> recipient=8]-ServerSessionImpl[test5@/127.0.0.1:53702])[Graceful] state
> already Graceful at
> sun.nio.ch.UnixAsynchronousSocketChannelImpl.implWrite(UnixAsynchronousSocketChannelImpl.java:736)
> at
> sun.nio.ch.AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:382)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@mina.apache.org
For additional commands, e-mail: dev-h...@mina.apache.org
