Re: MINA filter chains breaking under stress
I am aware of that the filterchain is not thread safe. As long I understood
Mina the encode/decode methods can be called by various threads, but it
Will always be the same thread for the same session.
For example, we have 3 session and 2 Threads, if session 1 is supplied
to the encode method it will always be called by Thread 1. If session 2
is supplied, it will be thread 2. But a thread can be used for multiple session,
e.g. session 3 is process by thread 1 again (in this example). Am I right?
On 14.01.19, 21:13, "Jonathan Valliere" wrote:
Upon a quick review of your email, I have the following notes.
The filterchain is not thread safe. You have to write threadsafe filters.
You must also make your encoded message system threadsafe. Just as with
SSL the actual encoding process must occur within a synchronized block.
As far as ProtocolCodecFilter goes, I will have to look into that.
On Mon, Jan 14, 2019 at 8:37 AM Max Larsson
wrote:
> I don't yet have a test case for the issues, but i have a description from
> a
> colleague in German under which circumstances this behavior occurs using
> Apache Mina 2.0.19:
>
> - There must be multiple Threads (at least two) calling nearly
> simultaneously session.write and there must be multiple
ProtocolCodecFilter
> in the FilterChain, whereas the first encoder doesn't produce messages of
> type IoBuffer
>
> - For example if thread A calls session.write and the first
> ProtocolCodecFilter has processed the write request, there will be an
> encoded message in the WriteRequestQueue of the session. Lets assume that
> it
> is an byte array.
>
> - Now if another thread B even called session.write in such circumstances
> that the call of thread B has passed all FilterChain and it is in progress
> to call AbstractPollingIoProcessor$Processor.flushNow(), the encoded
> message
> from thread A (which still is in process to complete FilterChain) result
in
> the known error "Don't know how to handle message of type 'xyz'. Are you
> missing a protocol encoder?'
>
> - all following calls to session.write (regardless which thread) will even
> produce the error, because the following snippet (somewhere in mina):
>
> // Check for pending writes.
> req = session.getCurrentWriteRequest();
> if (req == null) {
> req = writeRequestQueue.poll(session);
> if (req == null) {
> break;
> }
> session.setCurrentWriteRequest(req);
> }
>
> always return the same incomplete encoded message at line 'req =
> session.getCurrentWriteRequest();',
> because it isn't removed from initial first faulty processing.
>
>
> I hope someone understand whats going on, and if someone could tell me, if
> it is a bug in mina or a faulty
> usage of mina.
>
> best regards
>
> Max Larsson
>
>
>
> --
> Sent from:
> http://apache-mina.10907.n7.nabble.com/Apache-MINA-User-Forum-f31345.html
>
--
CONFIDENTIALITY NOTICE: The contents of this email message and any
attachments are intended solely for the addressee(s) and may contain
confidential and/or privileged information and may be legally protected
from disclosure.
RE: MINA filter chains breaking under stress
Thanks you all for your reply.
I managed to resolve it, actually mina functionality is that if it receive less
bytes to read it tries to shrink the read buffer size, and when next line is
longer then read buffer size at that time it reads data in two chunks
I just set the min read buffer size as max and that solve the issue
-Original Message-
From: Jonathan Valliere [mailto:jon.valli...@emoten.com]
Sent: Tuesday, January 15, 2019 1:43 AM
To: users@mina.apache.org
Subject: Re: MINA filter chains breaking under stress
Upon a quick review of your email, I have the following notes.
The filterchain is not thread safe. You have to write threadsafe filters.
You must also make your encoded message system threadsafe. Just as with SSL
the actual encoding process must occur within a synchronized block.
As far as ProtocolCodecFilter goes, I will have to look into that.
On Mon, Jan 14, 2019 at 8:37 AM Max Larsson
wrote:
> I don't yet have a test case for the issues, but i have a description
> from a colleague in German under which circumstances this behavior
> occurs using Apache Mina 2.0.19:
>
> - There must be multiple Threads (at least two) calling nearly
> simultaneously session.write and there must be multiple
> ProtocolCodecFilter in the FilterChain, whereas the first encoder
> doesn't produce messages of type IoBuffer
>
> - For example if thread A calls session.write and the first
> ProtocolCodecFilter has processed the write request, there will be an
> encoded message in the WriteRequestQueue of the session. Lets assume
> that it is an byte array.
>
> - Now if another thread B even called session.write in such
> circumstances that the call of thread B has passed all FilterChain and
> it is in progress to call
> AbstractPollingIoProcessor$Processor.flushNow(), the encoded message
> from thread A (which still is in process to complete FilterChain)
> result in the known error "Don't know how to handle message of type
> 'xyz'. Are you missing a protocol encoder?'
>
> - all following calls to session.write (regardless which thread) will
> even produce the error, because the following snippet (somewhere in mina):
>
> // Check for pending writes.
> req = session.getCurrentWriteRequest();
> if (req == null) {
> req = writeRequestQueue.poll(session);
> if (req == null) {
> break;
> }
> session.setCurrentWriteRequest(req);
> }
>
> always return the same incomplete encoded message at line 'req =
> session.getCurrentWriteRequest();',
> because it isn't removed from initial first faulty processing.
>
>
> I hope someone understand whats going on, and if someone could tell
> me, if it is a bug in mina or a faulty usage of mina.
>
> best regards
>
> Max Larsson
>
>
>
> --
> Sent from:
> http://apache-mina.10907.n7.nabble.com/Apache-MINA-User-Forum-f31345.h
> tml
>
--
CONFIDENTIALITY NOTICE: The contents of this email message and any attachments
are intended solely for the addressee(s) and may contain confidential and/or
privileged information and may be legally protected from disclosure.
Disclaimer: This message and the information contained herein is proprietary
and confidential and subject to the Tech Mahindra policy statement, you may
review the policy at http://www.techmahindra.com/Disclaimer.html
<http://www.techmahindra.com/Disclaimer.html> externally
http://tim.techmahindra.com/tim/disclaimer.html
<http://tim.techmahindra.com/tim/disclaimer.html> internally within
TechMahindra.
Re: MINA filter chains breaking under stress
Upon a quick review of your email, I have the following notes.
The filterchain is not thread safe. You have to write threadsafe filters.
You must also make your encoded message system threadsafe. Just as with
SSL the actual encoding process must occur within a synchronized block.
As far as ProtocolCodecFilter goes, I will have to look into that.
On Mon, Jan 14, 2019 at 8:37 AM Max Larsson
wrote:
> I don't yet have a test case for the issues, but i have a description from
> a
> colleague in German under which circumstances this behavior occurs using
> Apache Mina 2.0.19:
>
> - There must be multiple Threads (at least two) calling nearly
> simultaneously session.write and there must be multiple ProtocolCodecFilter
> in the FilterChain, whereas the first encoder doesn't produce messages of
> type IoBuffer
>
> - For example if thread A calls session.write and the first
> ProtocolCodecFilter has processed the write request, there will be an
> encoded message in the WriteRequestQueue of the session. Lets assume that
> it
> is an byte array.
>
> - Now if another thread B even called session.write in such circumstances
> that the call of thread B has passed all FilterChain and it is in progress
> to call AbstractPollingIoProcessor$Processor.flushNow(), the encoded
> message
> from thread A (which still is in process to complete FilterChain) result in
> the known error "Don't know how to handle message of type 'xyz'. Are you
> missing a protocol encoder?'
>
> - all following calls to session.write (regardless which thread) will even
> produce the error, because the following snippet (somewhere in mina):
>
> // Check for pending writes.
> req = session.getCurrentWriteRequest();
> if (req == null) {
> req = writeRequestQueue.poll(session);
> if (req == null) {
> break;
> }
> session.setCurrentWriteRequest(req);
> }
>
> always return the same incomplete encoded message at line 'req =
> session.getCurrentWriteRequest();',
> because it isn't removed from initial first faulty processing.
>
>
> I hope someone understand whats going on, and if someone could tell me, if
> it is a bug in mina or a faulty
> usage of mina.
>
> best regards
>
> Max Larsson
>
>
>
> --
> Sent from:
> http://apache-mina.10907.n7.nabble.com/Apache-MINA-User-Forum-f31345.html
>
--
CONFIDENTIALITY NOTICE: The contents of this email message and any
attachments are intended solely for the addressee(s) and may contain
confidential and/or privileged information and may be legally protected
from disclosure.
Re: MINA filter chains breaking under stress
I don't yet have a test case for the issues, but i have a description from a
colleague in German under which circumstances this behavior occurs using
Apache Mina 2.0.19:
- There must be multiple Threads (at least two) calling nearly
simultaneously session.write and there must be multiple ProtocolCodecFilter
in the FilterChain, whereas the first encoder doesn't produce messages of
type IoBuffer
- For example if thread A calls session.write and the first
ProtocolCodecFilter has processed the write request, there will be an
encoded message in the WriteRequestQueue of the session. Lets assume that it
is an byte array.
- Now if another thread B even called session.write in such circumstances
that the call of thread B has passed all FilterChain and it is in progress
to call AbstractPollingIoProcessor$Processor.flushNow(), the encoded message
from thread A (which still is in process to complete FilterChain) result in
the known error "Don't know how to handle message of type 'xyz'. Are you
missing a protocol encoder?'
- all following calls to session.write (regardless which thread) will even
produce the error, because the following snippet (somewhere in mina):
// Check for pending writes.
req = session.getCurrentWriteRequest();
if (req == null) {
req = writeRequestQueue.poll(session);
if (req == null) {
break;
}
session.setCurrentWriteRequest(req);
}
always return the same incomplete encoded message at line 'req =
session.getCurrentWriteRequest();',
because it isn't removed from initial first faulty processing.
I hope someone understand whats going on, and if someone could tell me, if
it is a bug in mina or a faulty
usage of mina.
best regards
Max Larsson
--
Sent from:
http://apache-mina.10907.n7.nabble.com/Apache-MINA-User-Forum-f31345.html
Re: MINA filter chains breaking under stress
Le 11/29/12 5:16 AM, Dhruv Chopra a écrit : So I tried acquiring the lock (added a new session attribute in sessionCreated as suggested) when the encoder A starts encoding and unlock after encoder B has written its output. However that didn't fix the problem. In fact I then also some exceptions where a thread was trying to unlock a resource it didn't own. I suppose it means that its not guaranteed that the same thread will encode a message through all the filters. Finally I synchronized the method that writes to a session in IoHandler itself and the problem appears to have gone. I was able to do its since I was already maintaining a wrapper around each IoSesion for some business logic - and everywhere I needed to write to a session, it was going through this wrapper. Ok, cool to see you have found a workaround. Altough it might be a contention point... It would be interesting for us to be able to have a reproductible example we can play with, if you can provide it (of course, with all the business part removed). Is that something possible ? -- Regards, Cordialement, Emmanuel Lécharny www.iktek.com
Re: MINA filter chains breaking under stress
Hi Emmanuel, It won't be possible for me to provide with a reproducible test setup *at the moment*.. Unfortunately our business logic is quite tightly integrated with MINA constructs (encoder/decoder/handler/session etc.) and it will not be easy to separate. However once I get some time off, I will look in to it. -Dhruv On Thu, Nov 29, 2012 at 2:19 PM, Emmanuel Lécharny elecha...@gmail.comwrote: Le 11/29/12 5:16 AM, Dhruv Chopra a écrit : So I tried acquiring the lock (added a new session attribute in sessionCreated as suggested) when the encoder A starts encoding and unlock after encoder B has written its output. However that didn't fix the problem. In fact I then also some exceptions where a thread was trying to unlock a resource it didn't own. I suppose it means that its not guaranteed that the same thread will encode a message through all the filters. Finally I synchronized the method that writes to a session in IoHandler itself and the problem appears to have gone. I was able to do its since I was already maintaining a wrapper around each IoSesion for some business logic - and everywhere I needed to write to a session, it was going through this wrapper. Ok, cool to see you have found a workaround. Altough it might be a contention point... It would be interesting for us to be able to have a reproductible example we can play with, if you can provide it (of course, with all the business part removed). Is that something possible ? -- Regards, Cordialement, Emmanuel Lécharny www.iktek.com
Re: MINA filter chains breaking under stress
Le 11/28/12 4:21 PM, Dhruv Chopra a écrit : Hi, Hi, My MINA server has a 2 filter chain (A and B - where output of encoder A is further encoded by B and then sent out on the wire). Now everything works fine under normal load (20 clients with incoming 20 msgs/sec and outgoing 100 msg/sec). But when I increase the load - I randomly start getting exceptions with cause *don't know how **to handle message of type 'X'**. Are you missing a protocol encoder?*. This message of type 'X' is the output of my filter A's encoder and input for filter B's encoder. I added a bunch of logs and see that in the failure case, my filter B never received the output of filter A and this exception followed. I also logged the threadids and see that when this occurred, thread 't' had encoded the message through filter A and then started encoding another message through filter A. Then the exception occurs and same Thread 't' continues encoding the other message through filter B. So its like it completely forgot to encode the earlier message through filter B. In the good cases, thread 't' would after encoding an object through encoder A always encode that through encoder B before doing anything else. Another thing is that when I only have one filter - even under higher stress, things work fine. Do you have an executor filter in your chain ? Otherwise, it seems you have enqueued a mssage which is not an IoBuffer, and this can't be sent. Like if encoder A directly pushes the message instead of going through the second encoder. It's difficult to tell what's going wrong without more information. Can you tell us more ? -- Regards, Cordialement, Emmanuel Lécharny www.iktek.com
Re: MINA filter chains breaking under stress
Do you have an executor filter in your chain ? *- No, both the filters extend ProtocolEncoderAdapter* Otherwise, it seems you have enqueued a mssage which is not an IoBuffer, and this can't be sent. Like if encoder A directly pushes the message instead of going through the second encoder. *- The encoder A always writes to the ProtocolEncoderOutput out (given as argument in the encode methodhttp://www.ostools.net/uploads/apidocs/mina/org/apache/mina/filter/codec/ProtocolEncoder.html#encode(org.apache.mina.core.session.IoSession, java.lang.Object, org.apache.mina.filter.codec.ProtocolEncoderOutput))* It's difficult to tell what's going wrong without more information. Can you tell us more ? *- The use case is that the IoHandler upon receipt of a message, broadcasts a message of type 'K' to a set of clients. It does so by looping through all the IoSessions belonging to that set of clients and calling write on them with this object of type 'K'. Encoder A accepts messages of type 'K' as input. Encoder A then encodes the message into type 'X' and writes it to the given ProtocolEncoderOutput for encoder B to consume (As described earlier). Anything specific you would like me to tell you? Or what else I could log that could help.* * * -Dhruv On Wed, Nov 28, 2012 at 9:00 PM, Emmanuel Lécharny elecha...@gmail.comwrote: Le 11/28/12 4:21 PM, Dhruv Chopra a écrit : Hi, Hi, My MINA server has a 2 filter chain (A and B - where output of encoder A is further encoded by B and then sent out on the wire). Now everything works fine under normal load (20 clients with incoming 20 msgs/sec and outgoing 100 msg/sec). But when I increase the load - I randomly start getting exceptions with cause *don't know how **to handle message of type 'X'**. Are you missing a protocol encoder?*. This message of type 'X' is the output of my filter A's encoder and input for filter B's encoder. I added a bunch of logs and see that in the failure case, my filter B never received the output of filter A and this exception followed. I also logged the threadids and see that when this occurred, thread 't' had encoded the message through filter A and then started encoding another message through filter A. Then the exception occurs and same Thread 't' continues encoding the other message through filter B. So its like it completely forgot to encode the earlier message through filter B. In the good cases, thread 't' would after encoding an object through encoder A always encode that through encoder B before doing anything else. Another thing is that when I only have one filter - even under higher stress, things work fine. Do you have an executor filter in your chain ? Otherwise, it seems you have enqueued a mssage which is not an IoBuffer, and this can't be sent. Like if encoder A directly pushes the message instead of going through the second encoder. It's difficult to tell what's going wrong without more information. Can you tell us more ? -- Regards, Cordialement, Emmanuel Lécharny www.iktek.com
Re: MINA filter chains breaking under stress
Le 11/28/12 5:10 PM, Dhruv Chopra a écrit : It's difficult to tell what's going wrong without more information. Can you tell us more ? *- The use case is that the IoHandler upon receipt of a message, broadcasts a message of type 'K' to a set of clients. It does so by looping through all the IoSessions belonging to that set of clients and calling write on them with this object of type 'K'. Encoder A accepts messages of type 'K' as input. Encoder A then encodes the message into type 'X' and writes it to the given ProtocolEncoderOutput for encoder B to consume (As described earlier). Anything specific you would like me to tell you? Or what else I could log that could help.* If you do that, under heavy load, it's likely that a client from another session sends a message, and tries to write. If those two operations (your own thread injecting some message in another session's write queue, and the session's handling an incoming message) are to happen at the same time, you may have some collision. May be adding a shared lock in the session's attributes which is hold by your thread when you write could help... -- Regards, Cordialement, Emmanuel Lécharny www.iktek.com
