Roberto Deandrea created SSHD-844:
-------------------------------------
Summary: Deadlock detected in Apache SSHD client code
Key: SSHD-844
URL: https://issues.apache.org/jira/browse/SSHD-844
Project: MINA SSHD
Issue Type: Bug
Affects Versions: 2.0.1
Environment: AIX 7.2
java version "1.8.0_151"
Java(TM) SE Runtime Environment (build 8.0.5.7 - pap6480sr5fp7-20171216_01(SR5
FP7))
IBM J9 VM (build 2.9, JRE 1.8.0 AIX ppc64-64 Compressed References
20171215_373586 (JIT enabled, AOT enabled)
OpenJ9 - 5aa401f
OMR - 101e793
IBM - b4a79bf)
JCL - 20171214_01 based on Oracle jdk8u151-b12
Reporter: Roberto Deandrea
Attachments: javacore.20180913.153332.14221746.0001.txt, jca457.jar
I found a deadlock in Apache SSHD client code running integration test for a
local port forwarding application. Attached the IBM javacore with information
about the deadlock.
Please run $java -jar jca457.jar to load the javacore file and display thread
stack traces.
[^javacore.20180913.153332.14221746.0001.txt]
[^jca457.jar]
There are 2 threads in DEADLOCK belonging to client pool's threads, that
blocked the other pool threads from working.
The locks involved are LOCK1 (object ChannelOutputStream) and LOCK2 Object
encodeLock of AbstractSession class.
The below thread is waiting on LOCK2 and owns LOCK1 (that is
ChannelOutputStream object)
The thread sshd-SshClient[61e9f863]-nio2-thread-6 has the following stack trace
:
at
org/apache/sshd/common/session/helpers/AbstractSession.doWritePacket(AbstractSession.java:1189(Compiled
Code))
at
org/apache/sshd/common/session/helpers/AbstractSession.writePacket(AbstractSession.java:1137(Compiled
Code))
at
org/apache/sshd/common/channel/AbstractChannel.writePacket(AbstractChannel.java:773(Compiled
Code))
at
org/apache/sshd/common/channel/ChannelOutputStream.flush(ChannelOutputStream.java:227(Compiled
Code))
at
org/apache/sshd/common/forward/DefaultForwardingFilter$StaticIoHandler.messageReceived(DefaultForwardingFilter.java:1063(Compiled
Code))
at
org/apache/sshd/common/io/nio2/Nio2Session.handleReadCycleCompletion(Nio2Session.java:339(Compiled
Code))
at
org/apache/sshd/common/io/nio2/Nio2Session$1.onCompleted(Nio2Session.java:318(Compiled
Code))
at
org/apache/sshd/common/io/nio2/Nio2Session$1.onCompleted(Nio2Session.java:315(Compiled
Code))
at
org/apache/sshd/common/io/nio2/Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38(Compiled
Code))
at
org/apache/sshd/common/io/nio2/Nio2CompletionHandler$$Lambda$254/0000000023F76250.run(Bytecode
PC:12(Compiled Code))
at
java/security/AccessController.doPrivileged(AccessController.java:638(Compiled
Code))
at
org/apache/sshd/common/io/nio2/Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37(Compiled
Code))
at sun/nio/ch/Invoker.invokeUnchecked(Invoker.java:138(Compiled Code))
at sun/nio/ch/Invoker$2.run(Invoker.java:230(Compiled Code))
at
sun/nio/ch/AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:124(Compiled
Code))
at
java/util/concurrent/ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160(Compiled
Code))
at
java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java/lang/Thread.run(Thread.java:811)
This thread is waiting for LOCK1 and owns LOCK2.
The thread sshd-SshClient[61e9f863]-nio2-thread-10 has the following stack
trace:
at
org/apache/sshd/common/channel/ChannelOutputStream.close(ChannelOutputStream.java:249(Compiled
Code))
at
org/apache/sshd/common/util/io/IoUtils.closeQuietly(IoUtils.java:151(Compiled
Code))
at
org/apache/sshd/client/channel/AbstractClientChannel.lambda$getInnerCloseable$1(AbstractClientChannel.java:203(Compiled
Code))
at
org/apache/sshd/client/channel/AbstractClientChannel$$Lambda$314/00000000252C5370.run(Bytecode
PC:4(Compiled Code))
at
org/apache/sshd/common/util/closeable/Builder$1.doClose(Builder.java:47(Compiled
Code))
at
org/apache/sshd/common/util/closeable/SimpleCloseable.close(SimpleCloseable.java:63(Compiled
Code))
at
org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:56(Compiled
Code))
at
org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:45(Compiled
Code))
at
org/apache/sshd/common/future/AbstractSshFuture.notifyListener(AbstractSshFuture.java:159(Compiled
Code))
at
org/apache/sshd/common/future/DefaultSshFuture.addListener(DefaultSshFuture.java:167(Compiled
Code))
at
org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:57(Compiled
Code))
at
org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:45(Compiled
Code))
at
org/apache/sshd/common/util/closeable/SequentialCloseable.doClose(SequentialCloseable.java:69(Compiled
Code))
at
org/apache/sshd/common/util/closeable/SimpleCloseable.close(SimpleCloseable.java:63(Compiled
Code))
at
org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:56(Compiled
Code))
at
org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:45(Compiled
Code))
at
org/apache/sshd/common/future/AbstractSshFuture.notifyListener(AbstractSshFuture.java:159(Compiled
Code))
at
org/apache/sshd/common/future/DefaultSshFuture.addListener(DefaultSshFuture.java:167(Compiled
Code))
at
org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:57(Compiled
Code))
at
org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:45(Compiled
Code))
at
org/apache/sshd/common/util/closeable/SequentialCloseable.doClose(SequentialCloseable.java:69(Compiled
Code))
at
org/apache/sshd/common/util/closeable/SimpleCloseable.close(SimpleCloseable.java:63(Compiled
Code))
at
org/apache/sshd/common/util/closeable/AbstractInnerCloseable.doCloseGracefully(AbstractInnerCloseable.java:40(Compiled
Code))
at
org/apache/sshd/common/util/closeable/AbstractCloseable.close(AbstractCloseable.java:98(Compiled
Code))
at
org/apache/sshd/common/forward/DefaultForwardingFilter$StaticIoHandler.sessionClosed(DefaultForwardingFilter.java:1043(Compiled
Code))
at
org/apache/sshd/common/io/nio2/Nio2Session.doCloseImmediately(Nio2Session.java:266(Compiled
Code))
at
org/apache/sshd/common/util/closeable/AbstractCloseable.close(AbstractCloseable.java:83(Compiled
Code))
at
org/apache/sshd/common/util/closeable/ParallelCloseable.doClose(ParallelCloseable.java:65(Compiled
Code))
at
org/apache/sshd/common/util/closeable/SimpleCloseable.close(SimpleCloseable.java:63(Compiled
Code))
at
org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:56(Compiled
Code))
at
org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:45(Compiled
Code))
at
org/apache/sshd/common/util/closeable/SequentialCloseable.doClose(SequentialCloseable.java:69(Compiled
Code))
at
org/apache/sshd/common/util/closeable/SimpleCloseable.close(SimpleCloseable.java:63(Compiled
Code))
at
org/apache/sshd/common/util/closeable/AbstractInnerCloseable.doCloseImmediately(AbstractInnerCloseable.java:46(Compiled
Code))
at
org/apache/sshd/common/util/closeable/AbstractCloseable.close(AbstractCloseable.java:83(Compiled
Code))
at
org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:56(Compiled
Code))
at
org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:45(Compiled
Code))
at
org/apache/sshd/common/future/AbstractSshFuture.notifyListener(AbstractSshFuture.java:159(Compiled
Code))
at
org/apache/sshd/common/future/DefaultSshFuture.addListener(DefaultSshFuture.java:167(Compiled
Code))
at
org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:57(Compiled
Code))
at
org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:45(Compiled
Code))
at
org/apache/sshd/common/util/closeable/SequentialCloseable.doClose(SequentialCloseable.java:69(Compiled
Code))
at
org/apache/sshd/common/util/closeable/SimpleCloseable.close(SimpleCloseable.java:63(Compiled
Code))
at
org/apache/sshd/common/util/closeable/AbstractInnerCloseable.doCloseImmediately(AbstractInnerCloseable.java:46(Compiled
Code))
at
org/apache/sshd/common/util/closeable/AbstractCloseable.close(AbstractCloseable.java:83(Compiled
Code))
at
org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:56(Compiled
Code))
at
org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:45(Compiled
Code))
at
org/apache/sshd/common/util/closeable/SequentialCloseable.doClose(SequentialCloseable.java:69(Compiled
Code))
at
org/apache/sshd/common/util/closeable/SimpleCloseable.close(SimpleCloseable.java:63(Compiled
Code))
at
org/apache/sshd/common/util/closeable/AbstractInnerCloseable.doCloseImmediately(AbstractInnerCloseable.java:46(Compiled
Code))
at
org/apache/sshd/common/util/closeable/AbstractCloseable.close(AbstractCloseable.java:83(Compiled
Code))
at
org/apache/sshd/common/util/closeable/ParallelCloseable.doClose(ParallelCloseable.java:65(Compiled
Code))
at
org/apache/sshd/common/util/closeable/SimpleCloseable.close(SimpleCloseable.java:63(Compiled
Code))
at
org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:56(Compiled
Code))
at
org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:45(Compiled
Code))
at
org/apache/sshd/common/util/closeable/SequentialCloseable.doClose(SequentialCloseable.java:69(Compiled
Code))
at
org/apache/sshd/common/util/closeable/SimpleCloseable.close(SimpleCloseable.java:63(Compiled
Code))
at
org/apache/sshd/common/util/closeable/AbstractInnerCloseable.doCloseImmediately(AbstractInnerCloseable.java:46(Compiled
Code))
at
org/apache/sshd/common/util/closeable/AbstractCloseable.close(AbstractCloseable.java:83(Compiled
Code))
at
org/apache/sshd/common/session/helpers/AbstractSession.exceptionCaught(AbstractSession.java:988)
at
org/apache/sshd/client/session/ClientSessionImpl.exceptionCaught(ClientSessionImpl.java:121)
at
org/apache/sshd/common/session/helpers/AbstractSessionIoHandler.exceptionCaught(AbstractSessionIoHandler.java:53)
at
org/apache/sshd/common/io/nio2/Nio2Session.exceptionCaught(Nio2Session.java:184(Compiled
Code))
at
org/apache/sshd/common/io/nio2/Nio2Session.handleWriteCycleFailure(Nio2Session.java:460)
at org/apache/sshd/common/io/nio2/Nio2Session$2.onFailed(Nio2Session.java:415)
at
org/apache/sshd/common/io/nio2/Nio2CompletionHandler.lambda$failed$1(Nio2CompletionHandler.java:46(Compiled
Code))
at
org/apache/sshd/common/io/nio2/Nio2CompletionHandler$$Lambda$324/00000000252CCA50.run(Bytecode
PC:12(Compiled Code))
at
java/security/AccessController.doPrivileged(AccessController.java:638(Compiled
Code))
at
org/apache/sshd/common/io/nio2/Nio2CompletionHandler.failed(Nio2CompletionHandler.java:45(Compiled
Code))
at sun/nio/ch/Invoker.invokeUnchecked(Invoker.java:140(Compiled Code))
at sun/nio/ch/Invoker.invokeDirect(Invoker.java:169(Compiled Code))
at
sun/nio/ch/UnixAsynchronousSocketChannelImpl.implWrite(UnixAsynchronousSocketChannelImpl.java:749(Compiled
Code))
at
sun/nio/ch/AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:394(Compiled
Code))
at
sun/nio/ch/AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:411(Compiled
Code))
at
org/apache/sshd/common/io/nio2/Nio2Session.doWriteCycle(Nio2Session.java:401(Compiled
Code))
at
org/apache/sshd/common/io/nio2/Nio2Session.startWriting(Nio2Session.java:386(Compiled
Code))
at
org/apache/sshd/common/io/nio2/Nio2Session.writePacket(Nio2Session.java:167(Compiled
Code))
at
org/apache/sshd/common/session/helpers/AbstractSession.doWritePacket(AbstractSession.java:1196(Compiled
Code))
at
org/apache/sshd/common/session/helpers/AbstractSession.writePacket(AbstractSession.java:1137(Compiled
Code))
at
org/apache/sshd/common/channel/AbstractChannel.writePacket(AbstractChannel.java:773(Compiled
Code))
at
org/apache/sshd/common/channel/ChannelOutputStream.flush(ChannelOutputStream.java:227(Compiled
Code))
at
org/apache/sshd/common/forward/DefaultForwardingFilter$StaticIoHandler.lambda$messageReceived$1(DefaultForwardingFilter.java:1069)
at
org/apache/sshd/common/forward/DefaultForwardingFilter$StaticIoHandler$$Lambda$327/00000000246D5DB0.operationComplete(Bytecode
PC:20)
at
org/apache/sshd/common/future/AbstractSshFuture.notifyListener(AbstractSshFuture.java:159(Compiled
Code))
at
org/apache/sshd/common/future/DefaultSshFuture.notifyListeners(DefaultSshFuture.java:217(Compiled
Code))
at
org/apache/sshd/common/future/DefaultSshFuture.setValue(DefaultSshFuture.java:112(Compiled
Code))
at
org/apache/sshd/client/future/DefaultOpenFuture.setOpened(DefaultOpenFuture.java:65(Compiled
Code))
at
org/apache/sshd/client/channel/AbstractClientChannel.handleOpenSuccess(AbstractClientChannel.java:352(Compiled
Code))
at
org/apache/sshd/common/session/helpers/AbstractConnectionService.channelOpenConfirmation(AbstractConnectionService.java:423(Compiled
Code))
at
org/apache/sshd/common/session/helpers/AbstractConnectionService.process(AbstractConnectionService.java:333(Compiled
Code))
at
org/apache/sshd/common/session/helpers/AbstractSession.doHandleMessage(AbstractSession.java:626(Compiled
Code))
at
org/apache/sshd/common/session/helpers/AbstractSession.handleMessage(AbstractSession.java:559(Compiled
Code))
at
org/apache/sshd/common/session/helpers/AbstractSession.decode(AbstractSession.java:1542(Compiled
Code))
at
org/apache/sshd/common/session/helpers/AbstractSession.messageReceived(AbstractSession.java:520(Compiled
Code))
at
org/apache/sshd/common/session/helpers/AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:63(Compiled
Code))
at
org/apache/sshd/common/io/nio2/Nio2Session.handleReadCycleCompletion(Nio2Session.java:339(Compiled
Code))
at
org/apache/sshd/common/io/nio2/Nio2Session$1.onCompleted(Nio2Session.java:318(Compiled
Code))
at
org/apache/sshd/common/io/nio2/Nio2Session$1.onCompleted(Nio2Session.java:315(Compiled
Code))
at
org/apache/sshd/common/io/nio2/Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38(Compiled
Code))
at
org/apache/sshd/common/io/nio2/Nio2CompletionHandler$$Lambda$254/0000000023F76250.run(Bytecode
PC:12(Compiled Code))
at
java/security/AccessController.doPrivileged(AccessController.java:638(Compiled
Code))
at
org/apache/sshd/common/io/nio2/Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37(Compiled
Code))
at sun/nio/ch/Invoker.invokeUnchecked(Invoker.java:138(Compiled Code))
at sun/nio/ch/Invoker$2.run(Invoker.java:230(Compiled Code))
at
sun/nio/ch/AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:124(Compiled
Code))
at
java/util/concurrent/ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160(Compiled
Code))
at
java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java/lang/Thread.run(Thread.java:811)
Can you find a solution to avoid this deadlock ?
Thanks in advance
Roberto
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
