Hi,
the problem is most certainly in the StateMachine part. I have to see
what it does.
On 22/12/2023 17:13, Kishore Mokkarala wrote:
*Hi Emmanuel,*
*
*
*Deadlock is happening in the 3rd party jars,not within the code,which
is out of the scope of the business logic. When large data is being
transferred between two entities with parallel TCP ip connections **over
SSL connection.** This is happening *
*Could you please provide a work around this problem,we are kind of
stuck now in the production.*
*Currently we are using mina version 2.2.1.*
*
*
*StateMachine.java:138*
*public* *void* handle(Event
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/event/Event.html#Event>
event) {
136
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L136>
StateContext
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/context/StateContext.html#StateContext>
context = event.getContext();
137 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L137>
138 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L138> *synchronized (context) {*
139
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L139>
LinkedList<Event> eventQueue = eventQueueThreadLocal.get();
140
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L140>
eventQueue.addLast(event);
141 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L141>
142 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L142> *if* (processingThreadLocal.get()) {
143
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L143>
//*/
144
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L144>
/* This thread is already processing an event. Queue this /
145
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L145>
/* event./
146
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L146>
/*//
147
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L147>
*if* (LOGGER.isDebugEnabled()) {
148
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L148>
LOGGER.debug("State machine called recursively. Queuing event k{}
for later processing.", event);
149
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L149>
}
150
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L150>
}*else* {
151
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L151>
processingThreadLocal.set(*true*);
152 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L152>
153 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L153> *try* {
154
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L154>
*if* (context.getCurrentState() ==*null*) {
155
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L155>
context.setCurrentState(startState);
156
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L156>
}
157 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L157>
158 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L158> processEvents(eventQueue);
159
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L159>
}*finally* {
160
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L160>
processingThreadLocal.set(false);
161
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L161>
}
162
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L162>
}
163
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L163>
}
164
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L164>
}
*
*
* at org.apache.mina.filter.ssl.SSLHandlerG0.write(SSLHandlerG0.java:312)*
*
*
*SSLHandlerG0.java:*
synchronized public void write(NextFilter next, WriteRequest request)
throws SSLException, WriteRejectedException {
if (LOGGER.isDebugEnabled()) {
LOGGER.debug("{} write() - source {}", toString(), request);
}*
*
........................
..........................
}
------------------------------------------
M.V.S.Kishore
Lead Dev Engineer
NetScout S/w Pvt. Ltd.
91-9886412814
On Fri, 22 Dec 2023 at 20:10, Kishore Mokkarala <[email protected]
<mailto:[email protected]>> wrote:
1.
Is the below issue fixed ?
1. DIRMINA-604 <https://issues.apache.org/jira/browse/DIRMINA-604>
Deadlock occurs when implementing two mina StateMachine
------------------------------------------
M.V.S.Kishore
Lead Dev Engineer
NetScout S/w Pvt. Ltd.
91-9886412814
On Fri, 22 Dec 2023 at 13:04, Emmanuel Lécharny <[email protected]
<mailto:[email protected]>> wrote:
Hi,
you have a deadlock on 0x00007f5c9f001408, which is in
com.netscout.nsaapp.geo.minaG10Proto.server.G10StateContext.
On 22/12/2023 08:10, Kishore Mokkarala wrote:
> Hi Emmanuel,
>
> We are facing deadlock in production with mina 2.2.1
version,earlier it
> was on 2.0.21, After upgrade only we are seeing this
issue,kindly please
> provide an update any similar kind issue is fixed in 2.2.3 or
not ?
>
>
> Thread pool-123-thread-3 is in deadlock with thread
NioProcessor-37
>
>
> pool-123-thread-3
>
> PRIORITY : 5
>
> THREAD ID : 0X00007F5978002870
>
> NATIVE ID : 0X7C15
>
> NATIVE ID (DECIMAL) : 31765
>
> STATE : BLOCKED
>
>
> stackTrace:
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
org.apache.mina.filter.ssl.SSLHandlerG0.write(SSLHandlerG0.java:312)
> - waiting to lock <0x00007f5c9f001168> (a
> org.apache.mina.filter.ssl.SSLHandlerG0)
> at
org.apache.mina.filter.ssl.SslFilter.filterWrite(SslFilter.java:380)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1500(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.filterWrite(DefaultIoFilterChain.java:1146)
> at
>
org.apache.mina.core.filterchain.IoFilterAdapter.filterWrite(IoFilterAdapter.java:138)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1500(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.filterWrite(DefaultIoFilterChain.java:1146)
> at
>
org.apache.mina.filter.codec.ProtocolCodecFilter.filterWrite(ProtocolCodecFilter.java:332)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1500(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.filterWrite(DefaultIoFilterChain.java:1146)
> at
>
org.apache.mina.filter.executor.ExecutorFilter.filterWrite(ExecutorFilter.java:595)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1500(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.filterWrite(DefaultIoFilterChain.java:1146)
> at
>
com.netscout.nsaapp.geo.minaG10Proto.server.G10GPBMessageIoFilter.filterWrite(G10GPBMessageIoFilter.java:63)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1500(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.filterWrite(DefaultIoFilterChain.java:1146)
> at
>
org.apache.mina.core.filterchain.IoFilterAdapter.filterWrite(IoFilterAdapter.java:138)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1500(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.filterWrite(DefaultIoFilterChain.java:1146)
> at
>
org.apache.mina.core.filterchain.IoFilterAdapter.filterWrite(IoFilterAdapter.java:138)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.fireFilterWrite(DefaultIoFilterChain.java:746)
> at
>
org.apache.mina.core.session.AbstractIoSession.write(AbstractIoSession.java:575)
> at
>
org.apache.mina.core.session.AbstractIoSession.write(AbstractIoSession.java:520)
> at
>
com.netscout.nsaapp.geo.g10Plugin.g10.processor.G10PluginCaptureProcessor.verifyAndSendStartMsgs(G10PluginCaptureProcessor.java:2627)
> at
>
com.netscout.nsaapp.geo.g10Plugin.g10.processor.G10PluginCaptureProcessor.sessionConnected(G10PluginCaptureProcessor.java:2552)
> at
>
com.netscout.nsaapp.geo.minaG10Proto.server.G10MinaClient.connect(G10MinaClient.java:220)
> at
jdk.internal.reflect.GeneratedMethodAccessor144.invoke(Unknown
Source)
> at
>
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke([email protected]/DelegatingMethodAccessorImpl.java:43)
> at
java.lang.reflect.Method.invoke([email protected]/Method.java:568)
> at
>
org.apache.mina.statemachine.transition.MethodTransition.invokeMethod(MethodTransition.java:281)
> at
>
org.apache.mina.statemachine.transition.MethodTransition.doExecute(MethodTransition.java:232)
> at
>
org.apache.mina.statemachine.transition.AbstractTransition.execute(AbstractTransition.java:100)
> at
org.apache.mina.statemachine.StateMachine.handle(StateMachine.java:183)
> at
>
org.apache.mina.statemachine.StateMachine.processEvents(StateMachine.java:170)
> at
org.apache.mina.statemachine.StateMachine.handle(StateMachine.java:158)
> - locked <0x00007f5c9f001408> (a
> com.netscout.nsaapp.geo.minaG10Proto.server.G10StateContext)
> at
>
org.apache.mina.statemachine.StateMachineProxyBuilder$MethodInvocationHandler.invoke(StateMachineProxyBuilder.java:261)
> at jdk.proxy4.$Proxy87.sessionOpened(jdk.proxy4/Unknown Source)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.sessionOpened(DefaultIoFilterChain.java:940)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextSessionOpened(DefaultIoFilterChain.java:574)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$800(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.sessionOpened(DefaultIoFilterChain.java:1083)
> at
>
org.apache.mina.core.filterchain.IoFilterAdapter.sessionOpened(IoFilterAdapter.java:90)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextSessionOpened(DefaultIoFilterChain.java:574)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$800(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.sessionOpened(DefaultIoFilterChain.java:1083)
> at
>
org.apache.mina.core.filterchain.IoFilterAdapter.sessionOpened(IoFilterAdapter.java:90)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextSessionOpened(DefaultIoFilterChain.java:574)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$800(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.sessionOpened(DefaultIoFilterChain.java:1083)
> at
>
org.apache.mina.core.filterchain.IoFilterEvent.fire(IoFilterEvent.java:127)
> at org.apache.mina.core.session.IoEvent.run(IoEvent.java:89)
> at
>
org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTask(OrderedThreadPoolExecutor.java:763)
> at
>
org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTasks(OrderedThreadPoolExecutor.java:755)
> at
>
org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.run(OrderedThreadPoolExecutor.java:695)
> at java.lang.Thread.run([email protected]/Thread.java:833)
>
>
> NioProcessor-37
>
> PRIORITY : 5
>
> THREAD ID : 0X00007F5A040B5880
>
> NATIVE ID : 0X6D19
>
> NATIVE ID (DECIMAL) : 27929
>
> STATE : BLOCKED
>
>
> stackTrace:
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
org.apache.mina.statemachine.StateMachine.handle(StateMachine.java:138)
> - waiting to lock <0x00007f5c9f001408> (a
> com.netscout.nsaapp.geo.minaG10Proto.server.G10StateContext)
> at
>
org.apache.mina.statemachine.StateMachineProxyBuilder$MethodInvocationHandler.invoke(StateMachineProxyBuilder.java:261)
> at jdk.proxy4.$Proxy87.event(jdk.proxy4/Unknown Source)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.event(DefaultIoFilterChain.java:1039)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextFilterEvent(DefaultIoFilterChain.java:789)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1700(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.event(DefaultIoFilterChain.java:1164)
> at
>
org.apache.mina.core.filterchain.IoFilterAdapter.event(IoFilterAdapter.java:162)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextFilterEvent(DefaultIoFilterChain.java:789)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1700(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.event(DefaultIoFilterChain.java:1164)
> at
>
org.apache.mina.core.filterchain.IoFilterAdapter.event(IoFilterAdapter.java:162)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextFilterEvent(DefaultIoFilterChain.java:789)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1700(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.event(DefaultIoFilterChain.java:1164)
> at
>
org.apache.mina.core.filterchain.IoFilterAdapter.event(IoFilterAdapter.java:162)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextFilterEvent(DefaultIoFilterChain.java:789)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1700(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.event(DefaultIoFilterChain.java:1164)
> at
>
org.apache.mina.core.filterchain.IoFilterAdapter.event(IoFilterAdapter.java:162)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextFilterEvent(DefaultIoFilterChain.java:789)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1700(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.event(DefaultIoFilterChain.java:1164)
> at
>
org.apache.mina.core.filterchain.IoFilterAdapter.event(IoFilterAdapter.java:162)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextFilterEvent(DefaultIoFilterChain.java:789)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1700(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.event(DefaultIoFilterChain.java:1164)
> at
>
org.apache.mina.filter.ssl.SSLHandlerG0.finish_handshake(SSLHandlerG0.java:589)
> - locked <0x00007f5c9f001168> (a
org.apache.mina.filter.ssl.SSLHandlerG0)
> at
>
org.apache.mina.filter.ssl.SSLHandlerG0.receive_loop(SSLHandlerG0.java:271)
> at
>
org.apache.mina.filter.ssl.SSLHandlerG0.receive_loop(SSLHandlerG0.java:246)
> at
>
org.apache.mina.filter.ssl.SSLHandlerG0.receive_loop(SSLHandlerG0.java:246)
> at
org.apache.mina.filter.ssl.SSLHandlerG0.receive(SSLHandlerG0.java:162)
> - locked <0x00007f5c9f001168> (a
org.apache.mina.filter.ssl.SSLHandlerG0)
> at
org.apache.mina.filter.ssl.SslFilter.messageReceived(SslFilter.java:342)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128)
> at
>
org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:122)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:643)
> at
>
org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:539)
> at
>
org.apache.mina.core.polling.AbstractPollingIoProcessor.access$1200(AbstractPollingIoProcessor.java:68)
> at
>
org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.process(AbstractPollingIoProcessor.java:1224)
> at
>
org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.process(AbstractPollingIoProcessor.java:1213)
> at
>
org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:683)
> at
>
org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
> at
>
java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1136)
> at
>
java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:635)
> at java.lang.Thread.run([email protected]/Thread.java:833)
>
> Regards,
> ------------------------------------------
> M.V.S.Kishore
> Lead Dev Engineer
> NetScout S/w Pvt. Ltd.
> 91-9886412814
--
*Emmanuel Lécharny* P. +33 (0)6 08 33 32 61
[email protected] <mailto:[email protected]>
--
*Emmanuel Lécharny* P. +33 (0)6 08 33 32 61
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
