*Hi Emmanuel,* *Deadlock is happening in the 3rd party jars,not within the code,which is out of the scope of the business logic. When large data is being transferred between two entities with parallel TCP ip connections **over SSL connection.** This is happening * *Could you please provide a work around this problem,we are kind of stuck now in the production.* *Currently we are using mina version 2.2.1.*
*StateMachine.java:138* *public* *void* handle(Event <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/event/Event.html#Event> event) {136 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L136> StateContext <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/context/StateContext.html#StateContext> context = event.getContext();137 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L137> 138 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L138> *synchronized (context) {*139 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L139> LinkedList<Event> eventQueue = eventQueueThreadLocal.get();140 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L140> eventQueue.addLast(event);141 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L141> 142 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L142> *if* (processingThreadLocal.get()) {143 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L143> */**144 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L144> * * This thread is already processing an event. Queue this *145 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L145> * * event.*146 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L146> * */*147 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L147> *if* (LOGGER.isDebugEnabled()) {148 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L148> LOGGER.debug("State machine called recursively. Queuing event k{} for later processing.", event);149 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L149> }150 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L150> } *else* {151 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L151> processingThreadLocal.set(*true*);152 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L152> 153 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L153> *try* {154 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L154> *if* (context.getCurrentState() == *null*) {155 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L155> context.setCurrentState(startState);156 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L156> }157 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L157> 158 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L158> processEvents(eventQueue);159 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L159> } *finally* {160 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L160> processingThreadLocal.set(false);161 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L161> }162 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L162> }163 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L163> }164 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L164> } * at org.apache.mina.filter.ssl.SSLHandlerG0.write(SSLHandlerG0.java:312)* *SSLHandlerG0.java:* synchronized public void write(NextFilter next, WriteRequest request) throws SSLException, WriteRejectedException { if (LOGGER.isDebugEnabled()) { LOGGER.debug("{} write() - source {}", toString(), request); } ........................ .......................... } ------------------------------------------ M.V.S.Kishore Lead Dev Engineer NetScout S/w Pvt. Ltd. 91-9886412814 On Fri, 22 Dec 2023 at 20:10, Kishore Mokkarala <kishore....@gmail.com> wrote: > > 1. > Is the below issue fixed ? > 1. DIRMINA-604 <https://issues.apache.org/jira/browse/DIRMINA-604> Deadlock > occurs when implementing two mina StateMachine > > ------------------------------------------ > M.V.S.Kishore > Lead Dev Engineer > NetScout S/w Pvt. Ltd. > 91-9886412814 > > > On Fri, 22 Dec 2023 at 13:04, Emmanuel Lécharny <elecha...@gmail.com> > wrote: > >> Hi, >> >> you have a deadlock on 0x00007f5c9f001408, which is in >> com.netscout.nsaapp.geo.minaG10Proto.server.G10StateContext. >> >> >> >> On 22/12/2023 08:10, Kishore Mokkarala wrote: >> > Hi Emmanuel, >> > >> > We are facing deadlock in production with mina 2.2.1 version,earlier it >> > was on 2.0.21, After upgrade only we are seeing this issue,kindly >> please >> > provide an update any similar kind issue is fixed in 2.2.3 or not ? >> > >> > >> > Thread pool-123-thread-3 is in deadlock with thread NioProcessor-37 >> > >> > >> > pool-123-thread-3 >> > >> > PRIORITY : 5 >> > >> > THREAD ID : 0X00007F5978002870 >> > >> > NATIVE ID : 0X7C15 >> > >> > NATIVE ID (DECIMAL) : 31765 >> > >> > STATE : BLOCKED >> > >> > >> > stackTrace: >> > java.lang.Thread.State: BLOCKED (on object monitor) >> > at org.apache.mina.filter.ssl.SSLHandlerG0.write(SSLHandlerG0.java:312) >> > - waiting to lock <0x00007f5c9f001168> (a >> > org.apache.mina.filter.ssl.SSLHandlerG0) >> > at org.apache.mina.filter.ssl.SslFilter.filterWrite(SslFilter.java:380) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1500(DefaultIoFilterChain.java:49) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.filterWrite(DefaultIoFilterChain.java:1146) >> > at >> > >> org.apache.mina.core.filterchain.IoFilterAdapter.filterWrite(IoFilterAdapter.java:138) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1500(DefaultIoFilterChain.java:49) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.filterWrite(DefaultIoFilterChain.java:1146) >> > at >> > >> org.apache.mina.filter.codec.ProtocolCodecFilter.filterWrite(ProtocolCodecFilter.java:332) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1500(DefaultIoFilterChain.java:49) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.filterWrite(DefaultIoFilterChain.java:1146) >> > at >> > >> org.apache.mina.filter.executor.ExecutorFilter.filterWrite(ExecutorFilter.java:595) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1500(DefaultIoFilterChain.java:49) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.filterWrite(DefaultIoFilterChain.java:1146) >> > at >> > >> com.netscout.nsaapp.geo.minaG10Proto.server.G10GPBMessageIoFilter.filterWrite(G10GPBMessageIoFilter.java:63) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1500(DefaultIoFilterChain.java:49) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.filterWrite(DefaultIoFilterChain.java:1146) >> > at >> > >> org.apache.mina.core.filterchain.IoFilterAdapter.filterWrite(IoFilterAdapter.java:138) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1500(DefaultIoFilterChain.java:49) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.filterWrite(DefaultIoFilterChain.java:1146) >> > at >> > >> org.apache.mina.core.filterchain.IoFilterAdapter.filterWrite(IoFilterAdapter.java:138) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain.fireFilterWrite(DefaultIoFilterChain.java:746) >> > at >> > >> org.apache.mina.core.session.AbstractIoSession.write(AbstractIoSession.java:575) >> > at >> > >> org.apache.mina.core.session.AbstractIoSession.write(AbstractIoSession.java:520) >> > at >> > >> com.netscout.nsaapp.geo.g10Plugin.g10.processor.G10PluginCaptureProcessor.verifyAndSendStartMsgs(G10PluginCaptureProcessor.java:2627) >> > at >> > >> com.netscout.nsaapp.geo.g10Plugin.g10.processor.G10PluginCaptureProcessor.sessionConnected(G10PluginCaptureProcessor.java:2552) >> > at >> > >> com.netscout.nsaapp.geo.minaG10Proto.server.G10MinaClient.connect(G10MinaClient.java:220) >> > at jdk.internal.reflect.GeneratedMethodAccessor144.invoke(Unknown >> Source) >> > at >> > >> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@17.0.7 >> /DelegatingMethodAccessorImpl.java:43) >> > at java.lang.reflect.Method.invoke(java.base@17.0.7/Method.java:568) >> > at >> > >> org.apache.mina.statemachine.transition.MethodTransition.invokeMethod(MethodTransition.java:281) >> > at >> > >> org.apache.mina.statemachine.transition.MethodTransition.doExecute(MethodTransition.java:232) >> > at >> > >> org.apache.mina.statemachine.transition.AbstractTransition.execute(AbstractTransition.java:100) >> > at >> org.apache.mina.statemachine.StateMachine.handle(StateMachine.java:183) >> > at >> > >> org.apache.mina.statemachine.StateMachine.processEvents(StateMachine.java:170) >> > at >> org.apache.mina.statemachine.StateMachine.handle(StateMachine.java:158) >> > - locked <0x00007f5c9f001408> (a >> > com.netscout.nsaapp.geo.minaG10Proto.server.G10StateContext) >> > at >> > >> org.apache.mina.statemachine.StateMachineProxyBuilder$MethodInvocationHandler.invoke(StateMachineProxyBuilder.java:261) >> > at jdk.proxy4.$Proxy87.sessionOpened(jdk.proxy4/Unknown Source) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.sessionOpened(DefaultIoFilterChain.java:940) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextSessionOpened(DefaultIoFilterChain.java:574) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$800(DefaultIoFilterChain.java:49) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.sessionOpened(DefaultIoFilterChain.java:1083) >> > at >> > >> org.apache.mina.core.filterchain.IoFilterAdapter.sessionOpened(IoFilterAdapter.java:90) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextSessionOpened(DefaultIoFilterChain.java:574) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$800(DefaultIoFilterChain.java:49) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.sessionOpened(DefaultIoFilterChain.java:1083) >> > at >> > >> org.apache.mina.core.filterchain.IoFilterAdapter.sessionOpened(IoFilterAdapter.java:90) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextSessionOpened(DefaultIoFilterChain.java:574) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$800(DefaultIoFilterChain.java:49) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.sessionOpened(DefaultIoFilterChain.java:1083) >> > at >> > >> org.apache.mina.core.filterchain.IoFilterEvent.fire(IoFilterEvent.java:127) >> > at org.apache.mina.core.session.IoEvent.run(IoEvent.java:89) >> > at >> > >> org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTask(OrderedThreadPoolExecutor.java:763) >> > at >> > >> org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTasks(OrderedThreadPoolExecutor.java:755) >> > at >> > >> org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.run(OrderedThreadPoolExecutor.java:695) >> > at java.lang.Thread.run(java.base@17.0.7/Thread.java:833) >> > >> > >> > NioProcessor-37 >> > >> > PRIORITY : 5 >> > >> > THREAD ID : 0X00007F5A040B5880 >> > >> > NATIVE ID : 0X6D19 >> > >> > NATIVE ID (DECIMAL) : 27929 >> > >> > STATE : BLOCKED >> > >> > >> > stackTrace: >> > java.lang.Thread.State: BLOCKED (on object monitor) >> > at >> org.apache.mina.statemachine.StateMachine.handle(StateMachine.java:138) >> > - waiting to lock <0x00007f5c9f001408> (a >> > com.netscout.nsaapp.geo.minaG10Proto.server.G10StateContext) >> > at >> > >> org.apache.mina.statemachine.StateMachineProxyBuilder$MethodInvocationHandler.invoke(StateMachineProxyBuilder.java:261) >> > at jdk.proxy4.$Proxy87.event(jdk.proxy4/Unknown Source) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.event(DefaultIoFilterChain.java:1039) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextFilterEvent(DefaultIoFilterChain.java:789) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1700(DefaultIoFilterChain.java:49) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.event(DefaultIoFilterChain.java:1164) >> > at >> > >> org.apache.mina.core.filterchain.IoFilterAdapter.event(IoFilterAdapter.java:162) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextFilterEvent(DefaultIoFilterChain.java:789) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1700(DefaultIoFilterChain.java:49) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.event(DefaultIoFilterChain.java:1164) >> > at >> > >> org.apache.mina.core.filterchain.IoFilterAdapter.event(IoFilterAdapter.java:162) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextFilterEvent(DefaultIoFilterChain.java:789) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1700(DefaultIoFilterChain.java:49) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.event(DefaultIoFilterChain.java:1164) >> > at >> > >> org.apache.mina.core.filterchain.IoFilterAdapter.event(IoFilterAdapter.java:162) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextFilterEvent(DefaultIoFilterChain.java:789) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1700(DefaultIoFilterChain.java:49) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.event(DefaultIoFilterChain.java:1164) >> > at >> > >> org.apache.mina.core.filterchain.IoFilterAdapter.event(IoFilterAdapter.java:162) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextFilterEvent(DefaultIoFilterChain.java:789) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1700(DefaultIoFilterChain.java:49) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.event(DefaultIoFilterChain.java:1164) >> > at >> > >> org.apache.mina.core.filterchain.IoFilterAdapter.event(IoFilterAdapter.java:162) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextFilterEvent(DefaultIoFilterChain.java:789) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1700(DefaultIoFilterChain.java:49) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.event(DefaultIoFilterChain.java:1164) >> > at >> > >> org.apache.mina.filter.ssl.SSLHandlerG0.finish_handshake(SSLHandlerG0.java:589) >> > - locked <0x00007f5c9f001168> (a >> org.apache.mina.filter.ssl.SSLHandlerG0) >> > at >> > >> org.apache.mina.filter.ssl.SSLHandlerG0.receive_loop(SSLHandlerG0.java:271) >> > at >> > >> org.apache.mina.filter.ssl.SSLHandlerG0.receive_loop(SSLHandlerG0.java:246) >> > at >> > >> org.apache.mina.filter.ssl.SSLHandlerG0.receive_loop(SSLHandlerG0.java:246) >> > at >> org.apache.mina.filter.ssl.SSLHandlerG0.receive(SSLHandlerG0.java:162) >> > - locked <0x00007f5c9f001168> (a >> org.apache.mina.filter.ssl.SSLHandlerG0) >> > at >> org.apache.mina.filter.ssl.SslFilter.messageReceived(SslFilter.java:342) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128) >> > at >> > >> org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:122) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650) >> > at >> > >> org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:643) >> > at >> > >> org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:539) >> > at >> > >> org.apache.mina.core.polling.AbstractPollingIoProcessor.access$1200(AbstractPollingIoProcessor.java:68) >> > at >> > >> org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.process(AbstractPollingIoProcessor.java:1224) >> > at >> > >> org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.process(AbstractPollingIoProcessor.java:1213) >> > at >> > >> org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:683) >> > at >> > >> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) >> > at >> > java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.7 >> /ThreadPoolExecutor.java:1136) >> > at >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.7 >> /ThreadPoolExecutor.java:635) >> > at java.lang.Thread.run(java.base@17.0.7/Thread.java:833) >> > >> > Regards, >> > ------------------------------------------ >> > M.V.S.Kishore >> > Lead Dev Engineer >> > NetScout S/w Pvt. Ltd. >> > 91-9886412814 >> >> -- >> *Emmanuel Lécharny* P. +33 (0)6 08 33 32 61 >> elecha...@apache.org >> >
