I never investigated the issues deeply enough so I could write a proper bug report.We have been using the Nio2ServiceFactoryFactory for the last year and a half so I do not recal all the issues.
Since we switched back to MinaServiceFactoryFactory because of the deadlock, the only issue we keep having few times per day is the ssh connection is accepted from the client perspective but the server never process it so it hang until the timeout is reached at the reverse proxy and client get disconnected. I did not create a bug report yet because I need to gather more information otherwise it will be impossible to fix. On Fri, 2017-12-08 at 19:02 +0000, Jonathan Valliere wrote: > You said “less major issues”; are there bugs reported for > MinaServiceFactoryFactory? > > On Fri, Dec 8, 2017 at 1:22 PM Hugo Arès (JIRA) <j...@apache.org> > wrote: > > > > > > > [ > > https://issues.apache.org/jira/browse/SSHD-779?page=com.atlassian.j > > ira.plugin.system.issuetabpanels:comment- > > tabpanel&focusedCommentId=16283975#comment-16283975 > > ] > > > > Hugo Arès commented on SSHD-779: > > -------------------------------- > > > > By nio2, I mean if we use Nio2ServiceFactoryFactory as the > > IoServiceFactoryFactory. > > > > This deadlock is not happening with MinaServiceFactoryFactory as > > the > > IoServiceFactoryFactory. > > > > We have other less major issues with MinaServiceFactoryFactory so > > we > > currently use this one in production but we will go back to > > Nio2ServiceFactoryFactory when the deadlock is fixed. > > > > > > > > Deadlock in ChannelOutputStream > > > ------------------------------- > > > > > > Key: SSHD-779 > > > URL: https://issues.apache.org/jira/browse/SSHD-7 > > > 79 > > > Project: MINA SSHD > > > Issue Type: Bug > > > Affects Versions: 1.4.0, 1.6.0 > > > Reporter: Hugo Arès > > > > > > Since Gerrit upgraded to 1.4.0, the following deadlock is > > > happening with > > NI02: > > > > > > {noformat} > > > Found one Java-level deadlock: > > > ============================= > > > "SSH-Stream-Worker-20": > > > waiting to lock monitor 0x00007fac34021cf8 (object > > > 0x00007fb450addb38, > > a java.lang.Object), > > > > > > which is held by "sshd-SshServer[4c110f87]-nio2-thread-20" > > > "sshd-SshServer[4c110f87]-nio2-thread-20": > > > waiting to lock monitor 0x00007fad20048678 (object > > > 0x00007fb450ade608, > > a org.apache.sshd.common.channel.ChannelOutputStream), > > > > > > which is held by "SSH-Stream-Worker-20" > > > Java stack information for the threads listed above: > > > =================================================== > > > "SSH-Stream-Worker-20": > > > at > > org.apache.sshd.common.session.helpers.AbstractSession.doWritePacke > > t(AbstractSession.java:1056) > > > > > > - waiting to lock <0x00007fb450addb38> (a > > > java.lang.Object) > > > at > > org.apache.sshd.common.session.helpers.AbstractSession.writePacket( > > AbstractSession.java:1005) > > > > > > at > > org.apache.sshd.common.channel.AbstractChannel.writePacket(Abstract > > Channel.java:766) > > > > > > at > > org.apache.sshd.common.channel.ChannelOutputStream.flush(ChannelOut > > putStream.java:219) > > > > > > - locked <0x00007fb450ade608> (a > > org.apache.sshd.common.channel.ChannelOutputStream) > > > > > > at > > > sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:297) > > > at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141) > > > - locked <0x00007fb450ae2eb0> (a > > > java.io.OutputStreamWriter) > > > at > > > java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229) > > > at java.io.BufferedWriter.flush(BufferedWriter.java:254) > > > - locked <0x00007fb450ae2eb0> (a > > > java.io.OutputStreamWriter) > > > at java.io.PrintWriter.flush(PrintWriter.java:320) > > > - locked <0x00007fb450ae2e70> (a java.io.BufferedWriter) > > > at java.io.PrintWriter.checkError(PrintWriter.java:357) > > > at > > com.google.gerrit.sshd.commands.StreamEvents.writeEvents(StreamEven > > ts.java:234) > > > > > > at > > com.google.gerrit.sshd.commands.StreamEvents.access$000(StreamEvent > > s.java:53) > > > > > > at > > com.google.gerrit.sshd.commands.StreamEvents$1.run(StreamEvents.jav > > a:100) > > > > > > at > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java: > > 511) > > > > > > at > > > java.util.concurrent.FutureTask.run(FutureTask.java:266) > > > at > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTas > > k.access$201(ScheduledThreadPoolExecutor.java:180) > > > > > > at > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTas > > k.run(ScheduledThreadPoolExecutor.java:293) > > > > > > at > > com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:418) > > > > > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto > > r.java:1142) > > > > > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut > > or.java:617) > > > > > > at java.lang.Thread.run(Thread.java:748) > > > "sshd-SshServer[4c110f87]-nio2-thread-20": > > > at > > org.apache.sshd.common.channel.ChannelOutputStream.close(ChannelOut > > putStream.java:241) > > > > > > - waiting to lock <0x00007fb450ade608> (a > > org.apache.sshd.common.channel.ChannelOutputStream) > > > > > > at > > org.apache.sshd.common.util.io.IoUtils.closeQuietly(IoUtils.java:13 > > 7) > > > > > > at > > org.apache.sshd.server.channel.ChannelSession.doCloseImmediately(Ch > > annelSession.java:201) > > > > > > at > > org.apache.sshd.common.util.closeable.AbstractCloseable.close(Abstr > > actCloseable.java:81) > > > > > > at > > org.apache.sshd.common.channel.AbstractChannel.close(AbstractChanne > > l.java:548) > > > > > > at > > org.apache.sshd.common.util.closeable.ParallelCloseable.doClose(Par > > allelCloseable.java:61) > > > > > > at > > org.apache.sshd.common.util.closeable.SimpleCloseable.close(SimpleC > > loseable.java:63) > > > > > > at > > org.apache.sshd.common.util.closeable.AbstractInnerCloseable.doClos > > eImmediately(AbstractInnerCloseable.java:45) > > > > > > at > > org.apache.sshd.common.util.closeable.AbstractCloseable.close(Abstr > > actCloseable.java:81) > > > > > > at > > org.apache.sshd.common.util.closeable.ParallelCloseable.doClose(Par > > allelCloseable.java:61) > > > > > > at > > org.apache.sshd.common.util.closeable.SimpleCloseable.close(SimpleC > > loseable.java:63) > > > > > > at > > org.apache.sshd.common.util.closeable.SequentialCloseable$1.operati > > onComplete(SequentialCloseable.java:55) > > > > > > at > > org.apache.sshd.common.util.closeable.SequentialCloseable$1.operati > > onComplete(SequentialCloseable.java:45) > > > > > > at > > org.apache.sshd.common.util.closeable.SequentialCloseable.doClose(S > > equentialCloseable.java:68) > > > > > > at > > org.apache.sshd.common.util.closeable.SimpleCloseable.close(SimpleC > > loseable.java:63) > > > > > > at > > org.apache.sshd.common.util.closeable.AbstractInnerCloseable.doClos > > eImmediately(AbstractInnerCloseable.java:45) > > > > > > at > > org.apache.sshd.common.util.closeable.AbstractCloseable.close(Abstr > > actCloseable.java:81) > > > > > > at > > org.apache.sshd.common.session.helpers.AbstractSession.exceptionCau > > ght(AbstractSession.java:862) > > > > > > at > > org.apache.sshd.common.session.helpers.AbstractSessionIoHandler.exc > > eptionCaught(AbstractSessionIoHandler.java:56) > > > > > > at org.apache.sshd.common.io > > .nio2.Nio2Session.exceptionCaught(Nio2Session.java:164) > > > > > > at org.apache.sshd.common.io > > .nio2.Nio2Session.handleWriteCycleFailure(Nio2Session.java:386) > > > > > > at org.apache.sshd.common.io > > .nio2.Nio2Session$2.onFailed(Nio2Session.java:348) > > > > > > at org.apache.sshd.common.io > > .nio2.Nio2CompletionHandler.lambda$failed$1(Nio2CompletionHandler.j > > ava:46) > > > > > > at > > > org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$318/ > > > 1992024659.run(Unknown > > Source) > > > > > > at java.security.AccessController.doPrivileged(Native > > > Method) > > > at org.apache.sshd.common.io > > .nio2.Nio2CompletionHandler.failed(Nio2CompletionHandler.java:45) > > > > > > at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:128) > > > at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157) > > > at sun.nio.ch > > .UnixAsynchronousSocketChannelImpl.implWrite(UnixAsynchronousSocket > > ChannelImpl.java:736) > > > > > > at sun.nio.ch > > .AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl. > > java:382) > > > > > > at sun.nio.ch > > .AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl. > > java:399) > > > > > > at org.apache.sshd.common.io > > .nio2.Nio2Session.doWriteCycle(Nio2Session.java:334) > > > > > > at org.apache.sshd.common.io > > .nio2.Nio2Session.startWriting(Nio2Session.java:318) > > > > > > at org.apache.sshd.common.io > > .nio2.Nio2Session.write(Nio2Session.java:148) > > > > > > at > > org.apache.sshd.common.session.helpers.AbstractSession.doWritePacke > > t(AbstractSession.java:1063) > > > > > > - locked <0x00007fb450addb38> (a java.lang.Object) > > > at > > org.apache.sshd.common.session.helpers.AbstractSession.writePacket( > > AbstractSession.java:1005) > > > > > > at > > org.apache.sshd.common.session.helpers.AbstractConnectionService.se > > ndGlobalResponse(AbstractConnectionService.java:694) > > > > > > at > > org.apache.sshd.common.session.helpers.AbstractConnectionService.gl > > obalRequest(AbstractConnectionService.java:662) > > > > > > at > > org.apache.sshd.common.session.helpers.AbstractConnectionService.pr > > ocess(AbstractConnectionService.java:353) > > > > > > at > > org.apache.sshd.common.session.helpers.AbstractSession.doHandleMess > > age(AbstractSession.java:564) > > > > > > at > > org.apache.sshd.common.session.helpers.AbstractSession.handleMessag > > e(AbstractSession.java:497) > > > > > > - locked <0x00007fb450ad9c48> (a java.lang.Object) > > > at > > org.apache.sshd.common.session.helpers.AbstractSession.decode(Abstr > > actSession.java:1406) > > > > > > at > > org.apache.sshd.common.session.helpers.AbstractSession.messageRecei > > ved(AbstractSession.java:458) > > > > > > - locked <0x00007fb450addb48> (a java.lang.Object) > > > at > > org.apache.sshd.common.session.helpers.AbstractSessionIoHandler.mes > > sageReceived(AbstractSessionIoHandler.java:67) > > > > > > at org.apache.sshd.common.io > > .nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:277) > > > > > > at org.apache.sshd.common.io > > .nio2.Nio2Session$1.onCompleted(Nio2Session.java:257) > > > > > > at org.apache.sshd.common.io > > .nio2.Nio2Session$1.onCompleted(Nio2Session.java:254) > > > > > > at org.apache.sshd.common.io > > .nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandle > > r.java:38) > > > > > > at > > > org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$182/ > > > 2116066305.run(Unknown > > Source) > > > > > > at java.security.AccessController.doPrivileged(Native > > > Method) > > > at org.apache.sshd.common.io > > .nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37 > > ) > > > > > > at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126) > > > at sun.nio.ch.Invoker$2.run(Invoker.java:218) > > > at sun.nio.ch > > .AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.ja > > va:112) > > > > > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto > > r.java:1142) > > > > > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut > > or.java:617) > > > > > > at java.lang.Thread.run(Thread.java:748) > > > {noformat} > > > > > > -- > > This message was sent by Atlassian JIRA > > (v6.4.14#64029) > >
