Hi Jonathan,
Kishore provided a thrzad dump a few weeks ago, which shows that there
is a lock:
NioProcessor-12
---------------
stackTrace:
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.mina.statemachine.StateMachine.handle(StateMachine.java:138)
- waiting to lock <0x00007fc1611faec8> (a
com.netscout.nsaapp.geo.minaG10Proto.server.G10StateContext)
at
org.apache.mina.statemachine.StateMachineProxyBuilder$MethodInvocationHandler.invoke(StateMachineProxyBuilder.java:261)
at jdk.proxy4.$Proxy83.event(jdk.proxy4/Unknown Source)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.event(DefaultIoFilterChain.java:1039)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextFilterEvent(DefaultIoFilterChain.java:789)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1700(DefaultIoFilterChain.java:49)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.event(DefaultIoFilterChain.java:1164)
at
org.apache.mina.core.filterchain.IoFilterAdapter.event(IoFilterAdapter.java:162)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextFilterEvent(DefaultIoFilterChain.java:789)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1700(DefaultIoFilterChain.java:49)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.event(DefaultIoFilterChain.java:1164)
at
org.apache.mina.core.filterchain.IoFilterAdapter.event(IoFilterAdapter.java:162)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextFilterEvent(DefaultIoFilterChain.java:789)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1700(DefaultIoFilterChain.java:49)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.event(DefaultIoFilterChain.java:1164)
at
org.apache.mina.core.filterchain.IoFilterAdapter.event(IoFilterAdapter.java:162)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextFilterEvent(DefaultIoFilterChain.java:789)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1700(DefaultIoFilterChain.java:49)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.event(DefaultIoFilterChain.java:1164)
at
org.apache.mina.core.filterchain.IoFilterAdapter.event(IoFilterAdapter.java:162)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextFilterEvent(DefaultIoFilterChain.java:789)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1700(DefaultIoFilterChain.java:49)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.event(DefaultIoFilterChain.java:1164)
at
org.apache.mina.core.filterchain.IoFilterAdapter.event(IoFilterAdapter.java:162)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextFilterEvent(DefaultIoFilterChain.java:789)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1700(DefaultIoFilterChain.java:49)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.event(DefaultIoFilterChain.java:1164)
at
org.apache.mina.filter.ssl.SSLHandlerG0.finish_handshake(SSLHandlerG0.java:589)
- locked <0x00007fc1611fb470> (a org.apache.mina.filter.ssl.SSLHandlerG0)
at
org.apache.mina.filter.ssl.SSLHandlerG0.receive_loop(SSLHandlerG0.java:271)
at
org.apache.mina.filter.ssl.SSLHandlerG0.receive_loop(SSLHandlerG0.java:246)
at
org.apache.mina.filter.ssl.SSLHandlerG0.receive_loop(SSLHandlerG0.java:246)
at org.apache.mina.filter.ssl.SSLHandlerG0.receive(SSLHandlerG0.java:162)
- locked <0x00007fc1611fb470> (a org.apache.mina.filter.ssl.SSLHandlerG0)
at org.apache.mina.filter.ssl.SslFilter.messageReceived(SslFilter.java:405)
...
and
pool-120-thread-37
------------------
stackTrace:
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.mina.filter.ssl.SSLHandlerG0.write(SSLHandlerG0.java:312)
- waiting to lock <0x00007fc1611fb470> (a
org.apache.mina.filter.ssl.SSLHandlerG0)
at org.apache.mina.filter.ssl.SslFilter.filterWrite(SslFilter.java:451)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1500(DefaultIoFilterChain.java:49)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.filterWrite(DefaultIoFilterChain.java:1146)
at
org.apache.mina.core.filterchain.IoFilterAdapter.filterWrite(IoFilterAdapter.java:138)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1500(DefaultIoFilterChain.java:49)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.filterWrite(DefaultIoFilterChain.java:1146)
at
org.apache.mina.filter.codec.ProtocolCodecFilter.filterWrite(ProtocolCodecFilter.java:332)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1500(DefaultIoFilterChain.java:49)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.filterWrite(DefaultIoFilterChain.java:1146)
at
org.apache.mina.filter.executor.ExecutorFilter.filterWrite(ExecutorFilter.java:595)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1500(DefaultIoFilterChain.java:49)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.filterWrite(DefaultIoFilterChain.java:1146)
at
com.netscout.nsaapp.geo.minaG10Proto.server.G10GPBMessageIoFilter.filterWrite(G10GPBMessageIoFilter.java:63)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1500(DefaultIoFilterChain.java:49)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.filterWrite(DefaultIoFilterChain.java:1146)
at
org.apache.mina.core.filterchain.IoFilterAdapter.filterWrite(IoFilterAdapter.java:138)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1500(DefaultIoFilterChain.java:49)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.filterWrite(DefaultIoFilterChain.java:1146)
at
org.apache.mina.core.filterchain.IoFilterAdapter.filterWrite(IoFilterAdapter.java:138)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:753)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.fireFilterWrite(DefaultIoFilterChain.java:746)
at
org.apache.mina.core.session.AbstractIoSession.write(AbstractIoSession.java:575)
at
org.apache.mina.core.session.AbstractIoSession.write(AbstractIoSession.java:520)
at
com.netscout.nsaapp.geo.g10Plugin.g10.processor.G10PluginCaptureProcessor.verifyAndSendStartMsgs(G10PluginCaptureProcessor.java:2627)
at
com.netscout.nsaapp.geo.g10Plugin.g10.processor.G10PluginCaptureProcessor.sessionConnected(G10PluginCaptureProcessor.java:2552)
at
com.netscout.nsaapp.geo.minaG10Proto.server.G10MinaClient.connect(G10MinaClient.java:220)
at jdk.internal.reflect.GeneratedMethodAccessor99.invoke(Unknown Source)
at
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@17.0.7/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(java.base@17.0.7/Method.java:568)
at
org.apache.mina.statemachine.transition.MethodTransition.invokeMethod(MethodTransition.java:281)
at
org.apache.mina.statemachine.transition.MethodTransition.doExecute(MethodTransition.java:232)
at
org.apache.mina.statemachine.transition.AbstractTransition.execute(AbstractTransition.java:100)
at org.apache.mina.statemachine.StateMachine.handle(StateMachine.java:183)
at
org.apache.mina.statemachine.StateMachine.processEvents(StateMachine.java:170)
at org.apache.mina.statemachine.StateMachine.handle(StateMachine.java:158)
- locked <0x00007fc1611faec8> (a
com.netscout.nsaapp.geo.minaG10Proto.server.G10StateContext)
at
org.apache.mina.statemachine.StateMachineProxyBuilder$MethodInvocationHandler.invoke(StateMachineProxyBuilder.java:261)
at jdk.proxy4.$Proxy83.sessionOpened(jdk.proxy4/Unknown Source)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.sessionOpened(DefaultIoFilterChain.java:940)
...
Basically, on the sessionOpened event, the StateMachine filter takes a
lock (0x00007fc1611faec8), then the IoHandler tries to write some data,
which ends in the SslFilter.filterWrite method, which takes a lock
(0x00007fc1611fb470)
Another thread is processing the messageReceived method, go through the
SslFilter.messageReceived method which takes a lock
(0x00007fc1611fb470), the same as in the upper thread, then goes through
the StateMachine.handle method, which takes a lock (0x00007fc1611faec8,
already seen upper).
The IoHandler should not try to write something on the sessionOpened
message IMO (until the session has been secured), but in any case, the
double lock is clearly the root cause of the problem.
On 17/02/2024 17:41, Emmanuel Lécharny wrote:
Hi Jonathan,
regarding the queing, we did that in MINA 2.1:
public void filterWrite(NextFilter nextFilter, IoSession session,
WriteRequest writeRequest) throws SSLException {
if (LOGGER.isDebugEnabled()) {
LOGGER.debug("{}: Writing Message : {}",
getSessionInfo(session), writeRequest);
}
boolean needsFlush = true;
SslHandler sslHandler = getSslSessionHandler(session);
try {
synchronized (sslHandler) {
if (!isSslStarted(session)) {
sslHandler.scheduleFilterWrite(nextFilter,
writeRequest);
}
...
and
/* no qualifier */void scheduleFilterWrite(NextFilter nextFilter,
WriteRequest writeRequest) {
filterWriteEventQueue.add(new IoFilterEvent(nextFilter,
IoEventType.WRITE, session, writeRequest));
}
Sobasically, of the session handshake has not been fully negociated,
then we enqueue the write request. Pretty much what you suggest to do
instead of using synchronized blocks around read and write.
Otherwise, comments inline:
On 17/02/2024 14:06, Jonathan Valliere wrote:
There is also some additional complexity supporting the scenario
where two
different threads are triggering “receive” events on the filter.
You mean on the same session?
We either have to process ALL messages out of the payload, then AFTER
send
them to the downstream filters or we have to perform a kind of
dual-locking
mechanism to ensure that multiple receiver threads to not operate in
parallel. Either way, we have to guarantee a serial processing of
incoming
messages.
Actually, this is not a problem unless someone puts an executor filter
*before the SslFilter in the chain. We process the incoming message in a
way a given session is associated with one single IoProcessor instance,
so all the messages for a given session should be processed sequentially
by this Io processor. If we don't have an excutor filter before the
Sslfilter, it should be safe.
The more I think about it the more concerned I am that we’re following a
rabbit down a hole.
@Kishore Mokkarala <kishore....@gmail.com> have you tried NOT using
the SSL
and instead creating a dummy filter which simply added synchronized to
the
receive and write method? I would be interested to know if that
causes the
problem also. Put this dummy filter in the same place as the SSL
would be.
Another way to deal with the lock would be to check for the
SessioSnecured event in the IoHandler side:
- if the event has not be received, then don't send writes, but enqueue
them
- if the session has been secured, then you can write at will
- when teh sessionSecured even is received, unqueue the write messages.
Somehow, that should replace what I think we should do in the SslFilter.
But I don't think the burden should be put on the MINA library user...
On Feb 17, 2024 at 2:00:30 AM, Emmanuel Lécharny <elecha...@gmail.com>
wrote:
Hi Jonathan,
there are two aspects to consider:
- first the establishemnt of the secured session
- second the standard exchange of data when the session has been
secured.
In the first case, we should *never* have any write done by the
IoHandler, or those writes should be enqueued until the session has been
secured. Here, the upstream filters should not be aware that the session
has been secured or not (all in all, filters are supposed to be
independenct)
In the second case, the SslFilter is responsible for handling the
encoding and decoding of the data, regardless of what the upper filters
are doing
So there should not be any assumption made on what wcould or should do
the upstream filters (that's because anyone can add a filter, and we
should not force the implementors to take care of unerlying filters
constraints).
So the best way to deal with writes while the Secured session
establishment is ongoing is to enqueue the writes until the session is
secured. It adds some complexity, but it's safe for the full stack
(reads are a different beast: we have to assume that we can only read
Handshake packets until the HS is completed)
I guess it fits with what you have in mind.
On 17/02/2024 04:12, Jonathan Valliere wrote:
I was thinking this over last weekend….
If I simply removed the synchronization then there COULD be several
incoming data corruption problems unless the upstream filters/
processors
MUST guarantee FIFO of the data stream.
What I would have todo is push OR copy the incoming buffer into a Queue
then perform processing off the head of that Queue. Essentially, do the
same as the mEncodeQueue but for Decodes. The lock would then only
be held
when processing data off of that Decode queue. Following this
pattern, I
could probably remove both the READ and WRITE synchronization blocks.
My understanding, for MINA, that the assumption is that it SHOULD be
safe
to hold on to any object passed into a Filter since object lifecycles
SHOULD be controlled by the VM and not the User’s code.
If we’re in agreement, I can make the change.
On Feb 10, 2024 at 2:14:01 PM, Kishore Mokkarala <kishore....@gmail.com>
wrote:
Any time line for removing synchronization block in SSLFilter?
On Sat, 10 Feb, 2024, 9:48 pm Emmanuel Lécharny, <elecha...@gmail.com>
wrote:
Netty is not Apache, but Eclipse.
We are discussing the error at the moment, trying to move away the
SSLFilter synchronized block.
On 10/02/2024 08:10, Kishore Mokkarala wrote:
We had to revert mina version to 2.0.25 from 2.2.1 to make it work in
production and trying for other alternatives like apache netty.
On Fri, 9 Feb, 2024, 1:59 pm Emmanuel Lécharny, <elecha...@gmail.com
<mailto:elecha...@gmail.com>> wrote:
Hi Jonathan,
in this very case, I think there is a race condition when
using the
SSLFilter in conjonction with the StateMachine filter.
On 09/02/2024 05:33, Jonathan Valliere wrote:
> No, you should not have to create multiple instances. The
necessary
> stateful data is saved to the Connection.
>
>
> On Feb 1, 2024 at 5:22:36 AM, Kishore Mokkarala
<kishore....@gmail.com <mailto:kishore....@gmail.com>>
> wrote:
>
>> Any response would be greatly appreciated.
>> ------------------------------------------
>> M.V.S.Kishore
>> 91-9886412814
>>
>>
>> On Wed, 31 Jan 2024 at 22:17, Kishore Mokkarala
<kishore....@gmail.com <mailto:kishore....@gmail.com>>
>> wrote:
>>
>> Hi Emmanuel,
>>
>>
>> Do we need to create a new instance of SSLFilter per tcp ip
connection or
>>
>> can we reuse it ?
>>
>>
>> For example, do we need to repeat the same code for each
tcp ip
connection
>>
>> ?
>>
>>
>> *Approach 1:*
>>
>> for(int i=0;i< 500;i++)
>>
>> {
>>
>> NioSocketConnector connector = new NioSocketConnector();
>>
>> connector.getFilterChain().addLast("LoggingFilter",
>>
>> G10CaptureService.loggingFilter);
>>
>> connector.getFilterChain().addLast("codecFilter",
>>
>> G10CaptureService.probeCodecFilter);
>>
>> connector.getFilterChain().addLast("executorFilter",
>>
>> G10CaptureService.executorFilter);
>>
>> connector.getFilterChain().addLast("gpbMessageFilter",
>>
>> G10CaptureService.gpbMessageFilter);
>>
>> connector.getFilterChain().addLast("keepAliveFilter",
>>
>> G10CaptureService.keepAliveFilter);
>>
>> SslFilter sslFilter;
>>
>> try {
>>
>> SSLContext sslContext = TLSUtil.getSSLContext();
>>
>> sslFilter = new CustomSslFilter(sslContext);
>>
>> connector.getFilterChain().addFirst("sslFilter", sslFilter);
>>
>> } catch (Exception e) {
>>
>> e.printStackTrace();
>>
>> LOG.error("Exception during creating SSL context..." +
>>
>> XError.getStackTrace(e));
>>
>> }
>>
>> //io handler creation
>>
>> StateMachine stateMachine =
>>
>>
StateMachineFactory.getInstance(IoHandlerTransition.class).create(
>>
>> G10MinaClient.CONNECTED, new
G10MinaClient(processor));
>>
>>
>> IoHandler ioHandler = new
>>
>> StateMachineProxyBuilder().setStateContextLookup(
>>
>> new IoSessionStateContextLookup(new
StateContextFactory() {
>>
>> @Override
>>
>> public StateContext create() {
>>
>> final G10StateContext
stateContext =
new
>>
>> G10StateContext();
>>
>> stateContext.setStartedTime(new
Date());
>>
>> LOG.info("G10StateContext
initialized
at:{}
>>
>> ",System.currentTimeMillis());
>>
>> return stateContext;
>>
>> }
>>
>> })).create(IoHandler.class, stateMachine);
>>
>> connector.setHandler(ioHandler);
>>
>> connector.connect(primaryAddress);
>>
>> connectFuture.awaitUninterruptibly();
>>
>> if (connectFuture.isConnected()) {
>>
>> IoSession session = connectFuture.getSession();
>>
>> // Do something with the session if needed
>>
>> } else {
>>
>> System.err.println("Connection failed for iteration: " + i);
>>
>> }
>>
>> }
>>
>>
>> *Approach 2:*
>>
>> Reuse the generic connector implemented above for opening
all
TCP/IP
>>
>> connections.
>>
>> //just do the below for getting connections for example
>>
>> NioSocketConnector connector = new NioSocketConnector();
>>
>> //filter chain creation
>>
>> //add SSLFilter to filer chain
>>
>>
>> for(int i=0;i< 500;i++)
>>
>> {
>>
>> ConnectFuture connectFuture =
connector.connect(serverAddress);
>>
>> connectFuture.awaitUninterruptibly();
>>
>> if (connectFuture.isConnected()) {
>>
>> IoSession session = connectFuture.getSession();
>>
>> // Do something with the session if needed
>>
>> } else {
>>
>> System.err.println("Connection failed for iteration: " + i);
>>
>> }
>>
>> }
>>
>>
>> Which approach is better ?
>>
>>
>> Regards,
>>
>> ------------------------------------------
>>
>> M.V.S.Kishore
>>
>> 91-9886412814
>>
>>
>>
>
--
*Emmanuel Lécharny* P. +33 (0)6 08 33 32 61
elecha...@apache.org <mailto:elecha...@apache.org>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@mina.apache.org
<mailto:users-unsubscr...@mina.apache.org>
For additional commands, e-mail: users-h...@mina.apache.org
<mailto:users-h...@mina.apache.org>
--
*Emmanuel Lécharny* P. +33 (0)6 08 33 32 61
elecha...@apache.org
--
*Emmanuel Lécharny* P. +33 (0)6 08 33 32 61
elecha...@apache.org
--
*Emmanuel Lécharny* P. +33 (0)6 08 33 32 61
elecha...@apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@mina.apache.org
For additional commands, e-mail: users-h...@mina.apache.org
