[jira] [Comment Edited] (SSHD-1070) OutOfMemoryError when use port forwarding
[
https://issues.apache.org/jira/browse/SSHD-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195275#comment-17195275
]
Guillaume Nodet edited comment on SSHD-1070 at 9/14/20, 2:06 PM:
-
I agree that a bounded queue is complicated, as what is important is the total
size of messages, not their number, so the idea
[https://github.com/apache/mina-sshd/pull/163] is a good one imho.
I've tried to come up with something more esy to integrate:
[https://github.com/apache/mina-sshd/pull/166|https://github.com/apache/mina-sshd/pull/166],
but it would need a reproducible test to actually verify the behavior.
was (Author: gnt):
I agree that a bounded queue is complicated, as what is important is the total
size of messages, not their number, so the idea
[https://github.com/apache/mina-sshd/pull/163] is a good one imho.
I've tried to come up with something more esy to integrate:
[https://github.com/apache/mina-sshd/pull/166,] but it would need a
reproducible test to actually verify the behavior.
> OutOfMemoryError when use port forwarding
> -
>
> Key: SSHD-1070
> URL: https://issues.apache.org/jira/browse/SSHD-1070
> Project: MINA SSHD
> Issue Type: Bug
>Affects Versions: 2.5.1
>Reporter: Feng Jiajie
>Priority: Major
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Hi [~gnodet], I found this commit implemented asynchronous port forwarding:
> [https://github.com/apache/mina-sshd/commit/45f84aab59b2e11d72942cffe9d810e37ab64959#diff-33823b8546f71d77bb1d653358ecde70]
> However, when a large amount of data is returned from upstream application,
> it is possible to cause an OOM in SSHD.
> Step1. SSHD server:
> {code:java}
> import org.apache.sshd.common.FactoryManager;
> import org.apache.sshd.common.PropertyResolverUtils;
> import org.apache.sshd.common.util.security.SecurityUtils;
> import org.apache.sshd.server.SshServer;
> import org.apache.sshd.server.forward.AcceptAllForwardingFilter;
> import org.apache.sshd.server.keyprovider.SimpleGeneratorHostKeyProvider;
> import java.io.IOException;
> import java.nio.file.Paths;
> public class TestSshd2 {
> public static void main(String[] args) throws IOException,
> InterruptedException {
> SecurityUtils.setAPrioriDisabledProvider("BC", true);
> SshServer sshd = SshServer.setUpDefaultServer();
> sshd.setPort(12133);
> sshd.setKeyPairProvider(new
> SimpleGeneratorHostKeyProvider(Paths.get("/tmp/aa.key")));
> sshd.setPasswordAuthenticator((username, password, session) -> true);
> sshd.setForwardingFilter(AcceptAllForwardingFilter.INSTANCE);
> sshd.start();
> Thread.sleep(1);
> }
> }
> {code}
> Step2. start ssh client and iperf3 server:
> {code:java}
> ssh -o 'ExitOnForwardFailure yes' -p 12133 -f -x -N -T -L
> 0.0.0.0:15678:127.0.0.1:12345 test5@127.0.0.1
> iperf3 -s -p 12345
> {code}
> Step3. run iperf3 client:
> {code:java}
> iperf3 -c 127.0.0.1 -i 1 -t 120 -p 15678 -P 8 --reverse
> {code}
> *-R, --reverse run in reverse mode (server sends, client receives)*
> SSHD will receive a lot of data but will not be able to forward it in time.
> log when OOM:
> {code:java}
> 17:52:25.195 [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-1] WARN
> org.apache.sshd.common.io.nio2.Nio2Session -
> exceptionCaught(Nio2Session[local=/127.0.0.1:33524, remote=/127.0.0.1:12345])
> Exception handler threw OutOfMemoryError, closing the session: GC overhead
> limit exceeded17:52:25.195
> [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-1] WARN
> org.apache.sshd.common.io.nio2.Nio2Session -
> exceptionCaught(Nio2Session[local=/127.0.0.1:33524, remote=/127.0.0.1:12345])
> Exception handler threw OutOfMemoryError, closing the session: GC overhead
> limit exceeded at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
> at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126) at
> sun.nio.ch.Invoker.invokeDirect(Invoker.java:157) at
> sun.nio.ch.UnixAsynchronousSocketChannelImpl.implWrite(UnixAsynchronousSocketChannelImpl.java:736)
> at
> sun.nio.ch.AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:382)
> at
> sun.nio.ch.AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:399)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.doWriteCycle(Nio2Session.java:420)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.startWriting(Nio2Session.java:404)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.finishWrite(Nio2Session.java:495)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.handleCompletedWriteCycle(Nio2Session.java:465)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:429)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.jav
[jira] [Comment Edited] (SSHD-1070) OutOfMemoryError when use port forwarding
[
https://issues.apache.org/jira/browse/SSHD-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192639#comment-17192639
]
Thomas Wolf edited comment on SSHD-1070 at 9/9/20, 5:53 AM:
{quote}I think your implementation PR #163 has the minimal overall impact and
is more suitable for a quick fix to the problem.
{quote}
I'm not sure. I actually closed/abandoned that PR after having tried your
change. I did observe that with my change, I consistently did run into
SSHD-1069 with the iperf3 test you suggested here (with -P 8; my machine has 4
cores and thus 5 NIO threads), but with yours, it just worked fine even with -P
8, and the throughputs for -P 1 and -P 2 were identical for both approaches.
Also, if I understand your change right, it is specific for the forwarding
case, whereas mine has the risk of affecting other uses of the
{{BufferedIoOutputStream}}. But forwarding should be the only scenario where a
server would encounter such speed differences between the two ends. (Or are
there other cases?) So probably your approach is less risky.
was (Author: wolft):
{quote}
I think your implementation PR #163 has the minimal overall impact and is more
suitable for a quick fix to the problem.
{quote}
I'm not sure. I actually closed/abandoned that PR after having tried your
change. I did observe that with my change, I consistently did run into into
SSHD-1069 with the iperf3 test you suggested here (with -P 8; my machine has 4
cores and thus 5 NIO threads), but with yours, it just worked fine even with -P
8, and the throughputs for -P 1 and -P 2 were identical for both approaches.
Also, if I understand your change right, it is specific for the forwarding
case, whereas mine has the risk of affecting other uses of the
{{BufferedIoOutputstream}}. But forwarding should be the only scenario where a
server would encounter such speed differences between the two ends. (Or are
there other cases?) So probably your approach is less risky.
> OutOfMemoryError when use port forwarding
> -
>
> Key: SSHD-1070
> URL: https://issues.apache.org/jira/browse/SSHD-1070
> Project: MINA SSHD
> Issue Type: Bug
>Affects Versions: 2.5.1
>Reporter: Feng Jiajie
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Hi [~gnodet], I found this commit implemented asynchronous port forwarding:
> [https://github.com/apache/mina-sshd/commit/45f84aab59b2e11d72942cffe9d810e37ab64959#diff-33823b8546f71d77bb1d653358ecde70]
> However, when a large amount of data is returned from upstream application,
> it is possible to cause an OOM in SSHD.
> Step1. SSHD server:
> {code:java}
> import org.apache.sshd.common.FactoryManager;
> import org.apache.sshd.common.PropertyResolverUtils;
> import org.apache.sshd.common.util.security.SecurityUtils;
> import org.apache.sshd.server.SshServer;
> import org.apache.sshd.server.forward.AcceptAllForwardingFilter;
> import org.apache.sshd.server.keyprovider.SimpleGeneratorHostKeyProvider;
> import java.io.IOException;
> import java.nio.file.Paths;
> public class TestSshd2 {
> public static void main(String[] args) throws IOException,
> InterruptedException {
> SecurityUtils.setAPrioriDisabledProvider("BC", true);
> SshServer sshd = SshServer.setUpDefaultServer();
> sshd.setPort(12133);
> sshd.setKeyPairProvider(new
> SimpleGeneratorHostKeyProvider(Paths.get("/tmp/aa.key")));
> sshd.setPasswordAuthenticator((username, password, session) -> true);
> sshd.setForwardingFilter(AcceptAllForwardingFilter.INSTANCE);
> sshd.start();
> Thread.sleep(1);
> }
> }
> {code}
> Step2. start ssh client and iperf3 server:
> {code:java}
> ssh -o 'ExitOnForwardFailure yes' -p 12133 -f -x -N -T -L
> 0.0.0.0:15678:127.0.0.1:12345 test5@127.0.0.1
> iperf3 -s -p 12345
> {code}
> Step3. run iperf3 client:
> {code:java}
> iperf3 -c 127.0.0.1 -i 1 -t 120 -p 15678 -P 8 --reverse
> {code}
> *-R, --reverse run in reverse mode (server sends, client receives)*
> SSHD will receive a lot of data but will not be able to forward it in time.
> log when OOM:
> {code:java}
> 17:52:25.195 [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-1] WARN
> org.apache.sshd.common.io.nio2.Nio2Session -
> exceptionCaught(Nio2Session[local=/127.0.0.1:33524, remote=/127.0.0.1:12345])
> Exception handler threw OutOfMemoryError, closing the session: GC overhead
> limit exceeded17:52:25.195
> [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-1] WARN
> org.apache.sshd.common.io.nio2.Nio2Session -
> exceptionCaught(Nio2Session[local=/127.0.0.1:33524, remote=/127.0.0.1:12345])
> Exception handler threw OutOfMemoryError, closing the session: GC overhead
> limit exceeded at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2Compl
[jira] [Comment Edited] (SSHD-1070) OutOfMemoryError when use port forwarding
[
https://issues.apache.org/jira/browse/SSHD-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192396#comment-17192396
]
Thomas Wolf edited comment on SSHD-1070 at 9/8/20, 6:44 PM:
That's rather nice. Indeed it works well. (y) It also seems to solve SSHD-1069?
was (Author: wolft):
That's rather nice. Indeed it works well. (y)
> OutOfMemoryError when use port forwarding
> -
>
> Key: SSHD-1070
> URL: https://issues.apache.org/jira/browse/SSHD-1070
> Project: MINA SSHD
> Issue Type: Bug
>Affects Versions: 2.5.1
>Reporter: Feng Jiajie
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Hi [~gnodet], I found this commit implemented asynchronous port forwarding:
> [https://github.com/apache/mina-sshd/commit/45f84aab59b2e11d72942cffe9d810e37ab64959#diff-33823b8546f71d77bb1d653358ecde70]
> However, when a large amount of data is returned from upstream application,
> it is possible to cause an OOM in SSHD.
> Step1. SSHD server:
> {code:java}
> import org.apache.sshd.common.FactoryManager;
> import org.apache.sshd.common.PropertyResolverUtils;
> import org.apache.sshd.common.util.security.SecurityUtils;
> import org.apache.sshd.server.SshServer;
> import org.apache.sshd.server.forward.AcceptAllForwardingFilter;
> import org.apache.sshd.server.keyprovider.SimpleGeneratorHostKeyProvider;
> import java.io.IOException;
> import java.nio.file.Paths;
> public class TestSshd2 {
> public static void main(String[] args) throws IOException,
> InterruptedException {
> SecurityUtils.setAPrioriDisabledProvider("BC", true);
> SshServer sshd = SshServer.setUpDefaultServer();
> sshd.setPort(12133);
> sshd.setKeyPairProvider(new
> SimpleGeneratorHostKeyProvider(Paths.get("/tmp/aa.key")));
> sshd.setPasswordAuthenticator((username, password, session) -> true);
> sshd.setForwardingFilter(AcceptAllForwardingFilter.INSTANCE);
> sshd.start();
> Thread.sleep(1);
> }
> }
> {code}
> Step2. start ssh client and iperf3 server:
> {code:java}
> ssh -o 'ExitOnForwardFailure yes' -p 12133 -f -x -N -T -L
> 0.0.0.0:15678:127.0.0.1:12345 test5@127.0.0.1
> iperf3 -s -p 12345
> {code}
> Step3. run iperf3 client:
> {code:java}
> iperf3 -c 127.0.0.1 -i 1 -t 120 -p 15678 -P 8 --reverse
> {code}
> *-R, --reverse run in reverse mode (server sends, client receives)*
> SSHD will receive a lot of data but will not be able to forward it in time.
> log when OOM:
> {code:java}
> 17:52:25.195 [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-1] WARN
> org.apache.sshd.common.io.nio2.Nio2Session -
> exceptionCaught(Nio2Session[local=/127.0.0.1:33524, remote=/127.0.0.1:12345])
> Exception handler threw OutOfMemoryError, closing the session: GC overhead
> limit exceeded17:52:25.195
> [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-1] WARN
> org.apache.sshd.common.io.nio2.Nio2Session -
> exceptionCaught(Nio2Session[local=/127.0.0.1:33524, remote=/127.0.0.1:12345])
> Exception handler threw OutOfMemoryError, closing the session: GC overhead
> limit exceeded at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
> at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126) at
> sun.nio.ch.Invoker.invokeDirect(Invoker.java:157) at
> sun.nio.ch.UnixAsynchronousSocketChannelImpl.implWrite(UnixAsynchronousSocketChannelImpl.java:736)
> at
> sun.nio.ch.AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:382)
> at
> sun.nio.ch.AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:399)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.doWriteCycle(Nio2Session.java:420)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.startWriting(Nio2Session.java:404)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.finishWrite(Nio2Session.java:495)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.handleCompletedWriteCycle(Nio2Session.java:465)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:429)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:426)17:52:25.639
> [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-6] WARN
> org.apache.sshd.common.io.nio2.Nio2Session -
> exceptionCaught(Nio2Session[local=/127.0.0.1:33530, remote=/127.0.0.1:12345])
> Exception handler threw OutOfMemoryError, closing the session: GC overhead
> limit exceeded at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
> at java.security.AccessController.doPrivileged(Native Method) at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
> at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126) at
> sun.nio.ch.Invoker.invokeDirect(Invoker.ja
[jira] [Comment Edited] (SSHD-1070) OutOfMemoryError when use port forwarding
[
https://issues.apache.org/jira/browse/SSHD-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17191424#comment-17191424
]
Feng Jiajie edited comment on SSHD-1070 at 9/7/20, 3:31 AM:
[~technobcn] Thanks for helping reproduce the issue.
[~wolft] Thanks for helping to fix the problem.
I think there maybe some problems with the commit: [PR
#163|https://github.com/apache/mina-sshd/pull/163/commits/37efbff75fed6eae374c7012277c433f002cd6d7]
BlockingQueue is a blocking queue, whereas SSHD is based on an asynchronous
framework, maybe this modification will cause blocking of asynchronous
processes?
# Could this cause a deadlock? For example: if all the working threads are
blocking on BlockingQueue.put, the process would fail to provide service,
because there would be no thread (all already blocking on BlockingQueue.put) to
retrieve elements from BlockingQueues.
# The performance of asynchronous frameworks will be degraded, and fast
connections may be slower in scenarios where both fast and slow connections
exist.
I don't know much about asynchronous frameworks, but I guess it's possible here
to
[offer|https://docs.oracle.com/javase/8/docs/api/java/util/Queue.html#offer-E-]
elements to the queue (unlike the blocking put operation), and if the queue is
full then immediately return the failure and put that "offer operation" into an
operation queue and wait until the next time there's a free working thread to
try to perform the operation?
was (Author: fengjiajie):
[~technobcn] Thanks for helping reproduce the issue.
[~wolft] Thanks for helping to fix the problem.
I think there maybe some problems with the commit: [PR
#163|https://github.com/apache/mina-sshd/pull/163/commits/37efbff75fed6eae374c7012277c433f002cd6d7]
BlockingQueue is a blocking queue, whereas SSHD is based on an asynchronous
framework, maybe this modification will cause blocking of asynchronous
processes?
a. Could this cause a deadlock? For example: if all the working threads are
blocking on BlockingQueue.put, the process would fail to provide service,
because there would be no thread (all already blocking on BlockingQueue.put) to
retrieve elements from BlockingQueues.
b. The performance of asynchronous frameworks will be degraded, and fast
connections may be slower in scenarios where both fast and slow connections
exist.
I don't know much about asynchronous frameworks, but I guess it's possible here
to
[offer|https://docs.oracle.com/javase/8/docs/api/java/util/Queue.html#offer-E-]
elements to the queue (unlike the blocking put operation), and if the queue is
full then immediately return the failure and put that "offer operation" into an
operation queue and wait until the next time there's a free working thread to
try to perform the operation?
> OutOfMemoryError when use port forwarding
> -
>
> Key: SSHD-1070
> URL: https://issues.apache.org/jira/browse/SSHD-1070
> Project: MINA SSHD
> Issue Type: Bug
>Affects Versions: 2.5.1
>Reporter: Feng Jiajie
>Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Hi [~gnodet], I found this commit implemented asynchronous port forwarding:
> [https://github.com/apache/mina-sshd/commit/45f84aab59b2e11d72942cffe9d810e37ab64959#diff-33823b8546f71d77bb1d653358ecde70]
> However, when a large amount of data is returned from upstream application,
> it is possible to cause an OOM in SSHD.
> Step1. SSHD server:
> {code:java}
> import org.apache.sshd.common.FactoryManager;
> import org.apache.sshd.common.PropertyResolverUtils;
> import org.apache.sshd.common.util.security.SecurityUtils;
> import org.apache.sshd.server.SshServer;
> import org.apache.sshd.server.forward.AcceptAllForwardingFilter;
> import org.apache.sshd.server.keyprovider.SimpleGeneratorHostKeyProvider;
> import java.io.IOException;
> import java.nio.file.Paths;
> public class TestSshd2 {
> public static void main(String[] args) throws IOException,
> InterruptedException {
> SecurityUtils.setAPrioriDisabledProvider("BC", true);
> SshServer sshd = SshServer.setUpDefaultServer();
> sshd.setPort(12133);
> sshd.setKeyPairProvider(new
> SimpleGeneratorHostKeyProvider(Paths.get("/tmp/aa.key")));
> sshd.setPasswordAuthenticator((username, password, session) -> true);
> sshd.setForwardingFilter(AcceptAllForwardingFilter.INSTANCE);
> sshd.start();
> Thread.sleep(1);
> }
> }
> {code}
> Step2. start ssh client and iperf3 server:
> {code:java}
> ssh -o 'ExitOnForwardFailure yes' -p 12133 -f -x -N -T -L
> 0.0.0.0:15678:127.0.0.1:12345 test5@127.0.0.1
> iperf3 -s -p 12345
> {code}
> Step3. run iperf3 client:
> {code:java}
> iperf3 -c 127.0.0.1 -i 1 -t 120 -p 15678 -P 8 --reverse
> {code}
> *-R, --reverse run in reverse mode (server sends,
[jira] [Comment Edited] (SSHD-1070) OutOfMemoryError when use port forwarding
[
https://issues.apache.org/jira/browse/SSHD-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17191375#comment-17191375
]
Thomas Wolf edited comment on SSHD-1070 at 9/6/20, 9:54 PM:
[PR #163|https://github.com/apache/mina-sshd/pull/163] implements this
throttling. Seems to work fine in my tests using Feng's iperf3 test with -P 1
or -P 2; with -P 8 I run into SSHD-1069. Also works fine with -P 8 if I apply
the work-around from 1069 and increase the number of NIO2 workers up front to
be >= 8.
was (Author: wolft):
[PR #163|https://github.com/apache/mina-sshd/pull/163] implements this
throttling. Seems to work fine in my tests using Feng's iperf3 test (with -P 1
or -P 2; with -P 8 I run into SSHD-1069).
> OutOfMemoryError when use port forwarding
> -
>
> Key: SSHD-1070
> URL: https://issues.apache.org/jira/browse/SSHD-1070
> Project: MINA SSHD
> Issue Type: Bug
>Affects Versions: 2.5.1
>Reporter: Feng Jiajie
>Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Hi [~gnodet], I found this commit implemented asynchronous port forwarding:
> [https://github.com/apache/mina-sshd/commit/45f84aab59b2e11d72942cffe9d810e37ab64959#diff-33823b8546f71d77bb1d653358ecde70]
> However, when a large amount of data is returned from upstream application,
> it is possible to cause an OOM in SSHD.
> Step1. SSHD server:
> {code:java}
> import org.apache.sshd.common.FactoryManager;
> import org.apache.sshd.common.PropertyResolverUtils;
> import org.apache.sshd.common.util.security.SecurityUtils;
> import org.apache.sshd.server.SshServer;
> import org.apache.sshd.server.forward.AcceptAllForwardingFilter;
> import org.apache.sshd.server.keyprovider.SimpleGeneratorHostKeyProvider;
> import java.io.IOException;
> import java.nio.file.Paths;
> public class TestSshd2 {
> public static void main(String[] args) throws IOException,
> InterruptedException {
> SecurityUtils.setAPrioriDisabledProvider("BC", true);
> SshServer sshd = SshServer.setUpDefaultServer();
> sshd.setPort(12133);
> sshd.setKeyPairProvider(new
> SimpleGeneratorHostKeyProvider(Paths.get("/tmp/aa.key")));
> sshd.setPasswordAuthenticator((username, password, session) -> true);
> sshd.setForwardingFilter(AcceptAllForwardingFilter.INSTANCE);
> sshd.start();
> Thread.sleep(1);
> }
> }
> {code}
> Step2. start ssh client and iperf3 server:
> {code:java}
> ssh -o 'ExitOnForwardFailure yes' -p 12133 -f -x -N -T -L
> 0.0.0.0:15678:127.0.0.1:12345 test5@127.0.0.1
> iperf3 -s -p 12345
> {code}
> Step3. run iperf3 client:
> {code:java}
> iperf3 -c 127.0.0.1 -i 1 -t 120 -p 15678 -P 8 --reverse
> {code}
> *-R, --reverse run in reverse mode (server sends, client receives)*
> SSHD will receive a lot of data but will not be able to forward it in time.
> log when OOM:
> {code:java}
> 17:52:25.195 [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-1] WARN
> org.apache.sshd.common.io.nio2.Nio2Session -
> exceptionCaught(Nio2Session[local=/127.0.0.1:33524, remote=/127.0.0.1:12345])
> Exception handler threw OutOfMemoryError, closing the session: GC overhead
> limit exceeded17:52:25.195
> [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-1] WARN
> org.apache.sshd.common.io.nio2.Nio2Session -
> exceptionCaught(Nio2Session[local=/127.0.0.1:33524, remote=/127.0.0.1:12345])
> Exception handler threw OutOfMemoryError, closing the session: GC overhead
> limit exceeded at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
> at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126) at
> sun.nio.ch.Invoker.invokeDirect(Invoker.java:157) at
> sun.nio.ch.UnixAsynchronousSocketChannelImpl.implWrite(UnixAsynchronousSocketChannelImpl.java:736)
> at
> sun.nio.ch.AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:382)
> at
> sun.nio.ch.AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:399)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.doWriteCycle(Nio2Session.java:420)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.startWriting(Nio2Session.java:404)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.finishWrite(Nio2Session.java:495)
> at
> org.apache.sshd.common.io.nio2.Nio2Session.handleCompletedWriteCycle(Nio2Session.java:465)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:429)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$2.onCompleted(Nio2Session.java:426)17:52:25.639
> [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-6] WARN
> org.apache.sshd.common.io.nio2.Nio2Session -
> exceptionCaught(Nio2Session[local=/127.0.0.1:33530, remote=/127.0.0.1:12345])
> Exception handler threw OutOfMemoryError, closing the session: GC overhead
> limit exce
[jira] [Comment Edited] (SSHD-1070) OutOfMemoryError when use port forwarding
[
https://issues.apache.org/jira/browse/SSHD-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17188957#comment-17188957
]
Feng Jiajie edited comment on SSHD-1070 at 9/2/20, 3:46 AM:
Hi [~lgoldstein].
{quote}How large are we talking ?
{quote}
greater than JVM heap memory.
Here's another example for reproducing the problem.
Step1. start a nginx server, and put a large(4GB+) file in the html directory,
then the file can be download using wget:
{code:java}
[root@debugboxcreate2204 nginx]# ll html/
total 4372592
-rw-r--r-- 1 work work537 May 22 19:31 50x.html
-rw-r--r-- 1 work work 4477522014 Sep 2 10:37 b
-rw-r--r-- 1 work work612 May 22 19:31 index.html
[root@debugboxcreate2204 nginx]#
[root@debugboxcreate2204 nginx]# wget 127.0.0.1:8080/b
{code}
Step2. start SSHD and ssh client:
{code:java}
java -Xmx2G TestSshd2
ssh -o 'ExitOnForwardFailure yes' -p 12133 -f -x -N -T -L
0.0.0.0:15678:127.0.0.1:8080 test5@127.0.0.1
{code}
Step3. wget file through forwarding port:
{code:java}
wget 127.0.0.1:15678/b
{code}
Then OOM will occur in a few seconds:
{code:java}
11:34:33.631 [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-2] WARN
org.apache.sshd.common.channel.IoWriteFutureImpl -
notifyListener(IoWriteFutureImpl[id=TcpipServerChannel[id=0,
recipient=1]-ServerSessionImpl[test5@/127.0.0.1:54820][SSH_MSG_CHANNEL_DATA]][value=true])
failed (OutOfMemoryError) to invoke
org.apache.sshd.common.channel.BufferedIoOutputStream$1@4eb77084: Java heap
space11:34:33.631 [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-2] WARN
org.apache.sshd.common.channel.IoWriteFutureImpl -
notifyListener(IoWriteFutureImpl[id=TcpipServerChannel[id=0,
recipient=1]-ServerSessionImpl[test5@/127.0.0.1:54820][SSH_MSG_CHANNEL_DATA]][value=true])
failed (OutOfMemoryError) to invoke
org.apache.sshd.common.channel.BufferedIoOutputStream$1@4eb77084: Java heap
space11:34:33.772 [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-2] WARN
org.apache.sshd.common.io.nio2.Nio2DefaultIoWriteFuture -
notifyListener(Nio2DefaultIoWriteFuture[id=/127.0.0.1:54820][value=true])
failed (OutOfMemoryError) to invoke
org.apache.sshd.common.channel.ChannelAsyncOutputStream$$Lambda$86/673218654@3d3a7dd4:
GC overhead limit exceeded11:34:34.207
[sshd-SshServer[5bcea91b](port=12133)-nio2-thread-3] WARN
org.apache.sshd.common.io.nio2.Nio2Session -
exceptionCaught(Nio2Session[local=/127.0.0.1:26744, remote=/127.0.0.1:8080])
Exception handler threw OutOfMemoryError, closing the session: GC overhead
limit exceededException in thread
"sshd-SshServer[5bcea91b](port=12133)-nio2-thread-2"
org.apache.sshd.common.RuntimeSshException11:34:35.525
[sshd-SshServer[5bcea91b](port=12133)-nio2-thread-3] WARN
org.apache.sshd.common.io.nio2.Nio2Session -
exceptionCaught(Nio2Session[local=/127.0.0.1:26744, remote=/127.0.0.1:8080])
Exception handler threw OutOfMemoryError, closing the session: GC overhead
limit exceeded at
org.apache.sshd.common.io.nio2.Nio2Session.startWriting(Nio2Session.java:411)
at org.apache.sshd.common.io.nio2.Nio2Session.finishWrite(Nio2Session.java:495)
{code}
I think the reason is:
nginx -> SSHD -> ssh -> wget
"nginx -> SSHD" speed about 1GB/s (Asynchronous)
"SSHD -> ssh" speed about 60MB/s (Data encryption and decryption)
after few second, SSHD would buffer GBs data in memory.
was (Author: fengjiajie):
Hi [~lgoldstein].
{quote}How large are we talking ?
{quote}
greater than JVM heap memory.
Here's another example for reproducing the problem.
Step1. start a nginx server, and put a large(4GB+) file in the html directory,
then the file can be download using wget:
{code:java}
[root@debugboxcreate2204 nginx]# ll html/
total 4372592
-rw-r--r-- 1 work work537 May 22 19:31 50x.html
-rw-r--r-- 1 work work 4477522014 Sep 2 10:37 b
-rw-r--r-- 1 work work612 May 22 19:31 index.html
[root@debugboxcreate2204 nginx]#
[root@debugboxcreate2204 nginx]# wget 127.0.0.1:8080/b
{code}
Step2. start SSHD and ssh client:
{code:java}
java -Xmx2G TestSshd2
ssh -o 'ExitOnForwardFailure yes' -p 12133 -f -x -N -T -L
0.0.0.0:15678:127.0.0.1:8080 test5@127.0.0.1
{code}
Step3. wget file through forwarding port:
{code:java}
wget 127.0.0.1:15678/b
{code}
Then OOM will occur in a few seconds:
{code:java}
11:34:33.631 [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-2] WARN
org.apache.sshd.common.channel.IoWriteFutureImpl -
notifyListener(IoWriteFutureImpl[id=TcpipServerChannel[id=0,
recipient=1]-ServerSessionImpl[test5@/127.0.0.1:54820][SSH_MSG_CHANNEL_DATA]][value=true])
failed (OutOfMemoryError) to invoke
org.apache.sshd.common.channel.BufferedIoOutputStream$1@4eb77084: Java heap
space11:34:33.631 [sshd-SshServer[5bcea91b](port=12133)-nio2-thread-2] WARN
org.apache.sshd.common.channel.IoWriteFutureImpl -
notifyListener(IoWriteFutureImpl[id=TcpipServerChannel[id=0,
recipient=1]-ServerSessionImpl[test5
