The thread dump is not readable from a HD analysys tool (bad HPROF format)
On 22/12/2023 17:21, Kishore Mokkarala wrote:
Attached thread dump too.
------------------------------------------
M.V.S.Kishore
Lead Dev Engineer
NetScout S/w Pvt. Ltd.
91-9886412814
---------- Forwarded message ---------
From: *Kishore Mokkarala* <[email protected]
<mailto:[email protected]>>
Date: Fri, 22 Dec 2023 at 21:43
Subject: Re: Deadlock in Apache mina version 2.2.1
To: Emmanuel Lécharny <[email protected]
<mailto:[email protected]>>, <[email protected]
<mailto:[email protected]>>, <[email protected]
<mailto:[email protected]>>
*Hi Emmanuel,*
*
*
*Deadlock is happening in the 3rd party jars,not within the code,which
is out of the scope of the business logic. When large data is being
transferred between two entities with parallel TCP ip connections **over
SSL connection.** This is happening *
*Could you please provide a work around this problem,we are kind of
stuck now in the production.*
*Currently we are using mina version 2.2.1.*
*
*
*StateMachine.java:138*
*public* *void* handle(Event
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/event/Event.html#Event>
event) {
136
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L136>
StateContext
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/context/StateContext.html#StateContext>
context = event.getContext();
137 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L137>
138 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L138> *synchronized (context) {*
139
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L139>
LinkedList<Event> eventQueue = eventQueueThreadLocal.get();
140
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L140>
eventQueue.addLast(event);
141 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L141>
142 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L142> *if* (processingThreadLocal.get()) {
143
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L143>
//*/
144
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L144>
/* This thread is already processing an event. Queue this /
145
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L145>
/* event./
146
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L146>
/*//
147
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L147>
*if* (LOGGER.isDebugEnabled()) {
148
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L148>
LOGGER.debug("State machine called recursively. Queuing event k{}
for later processing.", event);
149
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L149>
}
150
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L150>
}*else* {
151
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L151>
processingThreadLocal.set(*true*);
152 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L152>
153 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L153> *try* {
154
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L154>
*if* (context.getCurrentState() ==*null*) {
155
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L155>
context.setCurrentState(startState);
156
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L156>
}
157 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L157>
158 <https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L158> processEvents(eventQueue);
159
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L159>
}*finally* {
160
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L160>
processingThreadLocal.set(false);
161
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L161>
}
162
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L162>
}
163
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L163>
}
164
<https://nightlies.apache.org/mina/mina/2.0.24/xref/org/apache/mina/statemachine/StateMachine.html#L164>
}
*
*
* at org.apache.mina.filter.ssl.SSLHandlerG0.write(SSLHandlerG0.java:312)*
*
*
*SSLHandlerG0.java:*
synchronized public void write(NextFilter next, WriteRequest request)
throws SSLException, WriteRejectedException {
if (LOGGER.isDebugEnabled()) {
LOGGER.debug("{} write() - source {}", toString(), request);
}*
*
........................
..........................
}
------------------------------------------
M.V.S.Kishore
Lead Dev Engineer
NetScout S/w Pvt. Ltd.
91-9886412814
On Fri, 22 Dec 2023 at 20:10, Kishore Mokkarala <[email protected]
<mailto:[email protected]>> wrote:
1.
Is the below issue fixed ?
1. DIRMINA-604 <https://issues.apache.org/jira/browse/DIRMINA-604>
Deadlock occurs when implementing two mina StateMachine
------------------------------------------
M.V.S.Kishore
Lead Dev Engineer
NetScout S/w Pvt. Ltd.
91-9886412814
On Fri, 22 Dec 2023 at 13:04, Emmanuel Lécharny <[email protected]
<mailto:[email protected]>> wrote:
Hi,
you have a deadlock on 0x00007f5c9f001408, which is in
com.netscout.nsaapp.geo.minaG10Proto.server.G10StateContext.
On 22/12/2023 08:10, Kishore Mokkarala wrote:
> Hi Emmanuel,
>
> We are facing deadlock in production with mina 2.2.1
version,earlier it
> was on 2.0.21, After upgrade only we are seeing this
issue,kindly please
> provide an update any similar kind issue is fixed in 2.2.3 or
not ?
>
>
> Thread pool-123-thread-3 is in deadlock with thread
NioProcessor-37
>
>
> pool-123-thread-3
>
> PRIORITY : 5
>
> THREAD ID : 0X00007F5978002870
>
> NATIVE ID : 0X7C15
>
> NATIVE ID (DECIMAL) : 31765
>
> STATE : BLOCKED
>
>
> stackTrace:
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
org.apache.mina.filter.ssl.SSLHandlerG0.write(SSLHandlerG0.java:312)
> - waiting to lock <0x00007f5c9f001168> (a
> org.apache.mina.filter.ssl.SSLHandlerG0)
> at
org.apache.mina.filter.ssl.SslFilter.filterWrite(SslFilter.java:380)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1500(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.filterWrite(DefaultIoFilterChain.java:1146)
> at
>
org.apache.mina.core.filterchain.IoFilterAdapter.filterWrite(IoFilterAdapter.java:138)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1500(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.filterWrite(DefaultIoFilterChain.java:1146)
> at
>
org.apache.mina.filter.codec.ProtocolCodecFilter.filterWrite(ProtocolCodecFilter.java:332)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1500(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.filterWrite(DefaultIoFilterChain.java:1146)
> at
>
org.apache.mina.filter.executor.ExecutorFilter.filterWrite(ExecutorFilter.java:595)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1500(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.filterWrite(DefaultIoFilterChain.java:1146)
> at
>
com.netscout.nsaapp.geo.minaG10Proto.server.G10GPBMessageIoFilter.filterWrite(G10GPBMessageIoFilter.java:63)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1500(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.filterWrite(DefaultIoFilterChain.java:1146)
> at
>
org.apache.mina.core.filterchain.IoFilterAdapter.filterWrite(IoFilterAdapter.java:138)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1500(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.filterWrite(DefaultIoFilterChain.java:1146)
> at
>
org.apache.mina.core.filterchain.IoFilterAdapter.filterWrite(IoFilterAdapter.java:138)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.fireFilterWrite(DefaultIoFilterChain.java:746)
> at
>
org.apache.mina.core.session.AbstractIoSession.write(AbstractIoSession.java:575)
> at
>
org.apache.mina.core.session.AbstractIoSession.write(AbstractIoSession.java:520)
> at
>
com.netscout.nsaapp.geo.g10Plugin.g10.processor.G10PluginCaptureProcessor.verifyAndSendStartMsgs(G10PluginCaptureProcessor.java:2627)
> at
>
com.netscout.nsaapp.geo.g10Plugin.g10.processor.G10PluginCaptureProcessor.sessionConnected(G10PluginCaptureProcessor.java:2552)
> at
>
com.netscout.nsaapp.geo.minaG10Proto.server.G10MinaClient.connect(G10MinaClient.java:220)
> at
jdk.internal.reflect.GeneratedMethodAccessor144.invoke(Unknown
Source)
> at
>
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke([email protected]/DelegatingMethodAccessorImpl.java:43)
> at
java.lang.reflect.Method.invoke([email protected]/Method.java:568)
> at
>
org.apache.mina.statemachine.transition.MethodTransition.invokeMethod(MethodTransition.java:281)
> at
>
org.apache.mina.statemachine.transition.MethodTransition.doExecute(MethodTransition.java:232)
> at
>
org.apache.mina.statemachine.transition.AbstractTransition.execute(AbstractTransition.java:100)
> at
org.apache.mina.statemachine.StateMachine.handle(StateMachine.java:183)
> at
>
org.apache.mina.statemachine.StateMachine.processEvents(StateMachine.java:170)
> at
org.apache.mina.statemachine.StateMachine.handle(StateMachine.java:158)
> - locked <0x00007f5c9f001408> (a
> com.netscout.nsaapp.geo.minaG10Proto.server.G10StateContext)
> at
>
org.apache.mina.statemachine.StateMachineProxyBuilder$MethodInvocationHandler.invoke(StateMachineProxyBuilder.java:261)
> at jdk.proxy4.$Proxy87.sessionOpened(jdk.proxy4/Unknown Source)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.sessionOpened(DefaultIoFilterChain.java:940)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextSessionOpened(DefaultIoFilterChain.java:574)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$800(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.sessionOpened(DefaultIoFilterChain.java:1083)
> at
>
org.apache.mina.core.filterchain.IoFilterAdapter.sessionOpened(IoFilterAdapter.java:90)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextSessionOpened(DefaultIoFilterChain.java:574)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$800(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.sessionOpened(DefaultIoFilterChain.java:1083)
> at
>
org.apache.mina.core.filterchain.IoFilterAdapter.sessionOpened(IoFilterAdapter.java:90)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextSessionOpened(DefaultIoFilterChain.java:574)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$800(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.sessionOpened(DefaultIoFilterChain.java:1083)
> at
>
org.apache.mina.core.filterchain.IoFilterEvent.fire(IoFilterEvent.java:127)
> at org.apache.mina.core.session.IoEvent.run(IoEvent.java:89)
> at
>
org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTask(OrderedThreadPoolExecutor.java:763)
> at
>
org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTasks(OrderedThreadPoolExecutor.java:755)
> at
>
org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.run(OrderedThreadPoolExecutor.java:695)
> at java.lang.Thread.run([email protected]/Thread.java:833)
>
>
> NioProcessor-37
>
> PRIORITY : 5
>
> THREAD ID : 0X00007F5A040B5880
>
> NATIVE ID : 0X6D19
>
> NATIVE ID (DECIMAL) : 27929
>
> STATE : BLOCKED
>
>
> stackTrace:
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
org.apache.mina.statemachine.StateMachine.handle(StateMachine.java:138)
> - waiting to lock <0x00007f5c9f001408> (a
> com.netscout.nsaapp.geo.minaG10Proto.server.G10StateContext)
> at
>
org.apache.mina.statemachine.StateMachineProxyBuilder$MethodInvocationHandler.invoke(StateMachineProxyBuilder.java:261)
> at jdk.proxy4.$Proxy87.event(jdk.proxy4/Unknown Source)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.event(DefaultIoFilterChain.java:1039)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextFilterEvent(DefaultIoFilterChain.java:789)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1700(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.event(DefaultIoFilterChain.java:1164)
> at
>
org.apache.mina.core.filterchain.IoFilterAdapter.event(IoFilterAdapter.java:162)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextFilterEvent(DefaultIoFilterChain.java:789)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1700(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.event(DefaultIoFilterChain.java:1164)
> at
>
org.apache.mina.core.filterchain.IoFilterAdapter.event(IoFilterAdapter.java:162)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextFilterEvent(DefaultIoFilterChain.java:789)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1700(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.event(DefaultIoFilterChain.java:1164)
> at
>
org.apache.mina.core.filterchain.IoFilterAdapter.event(IoFilterAdapter.java:162)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextFilterEvent(DefaultIoFilterChain.java:789)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1700(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.event(DefaultIoFilterChain.java:1164)
> at
>
org.apache.mina.core.filterchain.IoFilterAdapter.event(IoFilterAdapter.java:162)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextFilterEvent(DefaultIoFilterChain.java:789)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1700(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.event(DefaultIoFilterChain.java:1164)
> at
>
org.apache.mina.core.filterchain.IoFilterAdapter.event(IoFilterAdapter.java:162)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextFilterEvent(DefaultIoFilterChain.java:789)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1700(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.event(DefaultIoFilterChain.java:1164)
> at
>
org.apache.mina.filter.ssl.SSLHandlerG0.finish_handshake(SSLHandlerG0.java:589)
> - locked <0x00007f5c9f001168> (a
org.apache.mina.filter.ssl.SSLHandlerG0)
> at
>
org.apache.mina.filter.ssl.SSLHandlerG0.receive_loop(SSLHandlerG0.java:271)
> at
>
org.apache.mina.filter.ssl.SSLHandlerG0.receive_loop(SSLHandlerG0.java:246)
> at
>
org.apache.mina.filter.ssl.SSLHandlerG0.receive_loop(SSLHandlerG0.java:246)
> at
org.apache.mina.filter.ssl.SSLHandlerG0.receive(SSLHandlerG0.java:162)
> - locked <0x00007f5c9f001168> (a
org.apache.mina.filter.ssl.SSLHandlerG0)
> at
org.apache.mina.filter.ssl.SslFilter.messageReceived(SslFilter.java:342)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128)
> at
>
org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:122)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
> at
>
org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:643)
> at
>
org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:539)
> at
>
org.apache.mina.core.polling.AbstractPollingIoProcessor.access$1200(AbstractPollingIoProcessor.java:68)
> at
>
org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.process(AbstractPollingIoProcessor.java:1224)
> at
>
org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.process(AbstractPollingIoProcessor.java:1213)
> at
>
org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:683)
> at
>
org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
> at
>
java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1136)
> at
>
java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:635)
> at java.lang.Thread.run([email protected]/Thread.java:833)
>
> Regards,
> ------------------------------------------
> M.V.S.Kishore
> Lead Dev Engineer
> NetScout S/w Pvt. Ltd.
> 91-9886412814
--
*Emmanuel Lécharny* P. +33 (0)6 08 33 32 61
[email protected] <mailto:[email protected]>
--
*Emmanuel Lécharny* P. +33 (0)6 08 33 32 61
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
