Dear Jonathan,
Thanks for bringing more clarity on Processor Thread and it's
relation with IoSession. We are also suspecting the network connection and in a
crisis one should pull all the strings to check where you get the lead to
identify the problem and that's what we are doing.
Thanks a tone for all your replies and giving very useful information. We will
check further on network connection.
Thanks And Regards,
Nitin Phuria
Confidentiality Disclaimer: “The information contained in this electronic
message (email) and any attachments to this email are intended for the
exclusive use of the addressee(s) and access to this email by anyone else is
unauthorized. The email may contain proprietary, confidential or privileged
information or information relating to Integra Group. If you are not the
intended recipient, please notify the sender by telephone, fax, or return email
and delete this communication and any attachments thereto, immediately from
your computer. Any dissemination, distribution, or copying of this
communication and the attachments thereto (in whole or part), in any manner, is
strictly prohibited and actionable at law. The recipient acknowledges that
emails are susceptible to alteration and their integrity cannot be guaranteed
and that Company does not guarantee that any e-mail is virus-free and accept no
liability for any damage caused by any virus transmitted by this email.”
From: Jonathan Valliere [mailto:[email protected]]
Sent: 09 April 2020 20:03
To: Nitin Phuria
Cc: [email protected]
Subject: Re: MINA: ExecutorFilter on NioSocketConnector
Every IoSession is bound to one Processor Thread. So only by creating more
IoSessions can you take advantage of multiple Processor Threads.
ExecutorFilter only works for received messages. So it generally does nothing
on the client side however it will cause the received messages to be executed
in multiple threads.
The purpose of the ExecutorFilter is to do potentially blocking tasks in there
so as not to block the Processor Thread from handling tasks from other
IoSessions.
Since you’re seeing a huge jump from 15ms to several minutes and a very small
number of messages it is most likely a problem with your network connection.
On Thu, Apr 9, 2020 at 10:26 AM Nitin Phuria <[email protected]> wrote:
Dear Jonathan,
Our Server and System-B are on WAN network with leased line.
One more clarification required As per documentation
NioSocketConnector
<http://mina.apache.org/mina-project/gen-docs/2.0.10/apidocs/org/apache/mina/transport/socket/nio/NioSocketConnector.html#NioSocketConnector-->
() Constructor for
<http://mina.apache.org/mina-project/gen-docs/2.0.10/apidocs/org/apache/mina/transport/socket/nio/NioSocketConnector.html>
NioSocketConnector with default configuration (multiple thread model).
So when we create NioSocketConnector connector = new NioSocketConnector(); I
expect it to create n+1 processor
where n is number of core on the hardware. In our case it is 17 (16 core
machine)
When we took threaddump of our Server process I could see only one thread for
this connector and in the log (log4j) when we print the thread name it prints
NioProcessor-x where x is some number.
So does it means that n+1 processors use single thread to work.
Also you mentioned that The ExecutorFilter only works for the Server receiver
side.
http://apache-mina.10907.n7.nabble.com/OrderedThreadPoolExecutor-shutdown-problem-in-MINA-2-0-0-M4-td34417.html
But I could see from this above URL that we can add it to Connector also and it
says "My understanding is that the Executor filter just adds an executor in the
chain which will be used to spread the load on many threads. This is an
optimization, rather than something you need to use. If you don't use it, your
program will work fine."
Could you provide more insight and your thoughts.
Thanks And Regards,
Nitin Phuria
Confidentiality Disclaimer: “The information contained in this electronic
message (email) and any attachments to this email are intended for the
exclusive use of the addressee(s) and access to this email by anyone else is
unauthorized. The email may contain proprietary, confidential or privileged
information or information relating to Integra Group. If you are not the
intended recipient, please notify the sender by telephone, fax, or return email
and delete this communication and any attachments thereto, immediately from
your computer. Any dissemination, distribution, or copying of this
communication and the attachments thereto (in whole or part), in any manner, is
strictly prohibited and actionable at law. The recipient acknowledges that
emails are susceptible to alteration and their integrity cannot be guaranteed
and that Company does not guarantee that any e-mail is virus-free and accept no
liability for any damage caused by any virus transmitted by this email.”
From: Jonathan Valliere [mailto:[email protected]]
Sent: 09 April 2020 19:34
To: Nitin Phuria
Cc: [email protected]
Subject: Re: MINA: ExecutorFilter on NioSocketConnector
How far are the servers? TCP Packet loss could explain this.
You can get the Queue size from the IoSession object.
On Thu, Apr 9, 2020 at 10:02 AM Nitin Phuria <[email protected]> wrote:
Dear Jonathan,
Anytime we write message to System-B session with session.write method the
messageSent event has to be generated with it actually gets written on TCP/IP
org.apache.mina.core.session.IoSession.write(Object)
We see that there is time difference of 10-14 seconds between timestamp of
session.write for particular message and messageSent event getting generated
for the same message.
In non-peak load this time difference is in 10-15 milliseconds and in peak load
(60-70 messages per second) it goes till 10-14 seconds.
Is there anything that we can log to see what is happening or see the queue
size etc.. for further better understaning.
Thanks And Regards,
Nitin Phuria
Confidentiality Disclaimer: “The information contained in this electronic
message (email) and any attachments to this email are intended for the
exclusive use of the addressee(s) and access to this email by anyone else is
unauthorized. The email may contain proprietary, confidential or privileged
information or information relating to Integra Group. If you are not the
intended recipient, please notify the sender by telephone, fax, or return email
and delete this communication and any attachments thereto, immediately from
your computer. Any dissemination, distribution, or copying of this
communication and the attachments thereto (in whole or part), in any manner, is
strictly prohibited and actionable at law. The recipient acknowledges that
emails are susceptible to alteration and their integrity cannot be guaranteed
and that Company does not guarantee that any e-mail is virus-free and accept no
liability for any damage caused by any virus transmitted by this email.”
From: Jonathan Valliere [mailto:[email protected]]
Sent: 09 April 2020 18:31
To: Nitin Phuria
Cc: Nitin Phuria; [email protected]
Subject: Re: MINA: ExecutorFilter on NioSocketConnector
The ExecutorFilter only works for the Server receiver side.
When you say the responses are slow, what do you consider slow?
All writes are put into a queue dedicated for each session. The Session is
then enabled for writing by adding the write interest to the poll mechanism.
Reads and Writes IO happen in the same thread. After the Session finishes
reading it will then call the poll mechanism for new events to process.
Hopefully it should get the write event for your Session then begin to flush
the queue.
Processor Thread Loop —> Select Poll Events —> Process Events
(read/write) —> Loop
On Thu, Apr 9, 2020 at 8:53 AM Nitin Phuria <[email protected]> wrote:
Dear Jonathan,
Thanks for the reply, can you provide more details on the "All writes
are queued up and scheduled
later using the poll/kqueue system." How this poll & kqueue system works? Do we
have any control on this from MINA framework?
We are not facing any problem with the higher CPU consumption but we see that
when there is high load from System-B to our server the reads from System-B and
write back to System-B on the single persistence session is becoming slow. So
we were looking at i/o operations and how to make them fast.
With your explanation provided in below email about ExecutorFilter, If we add
it in our connector after codec filter will help us reduce the I/O overhead on
reactor/processor thread. I hope my understanding is right kindly confirm?
Thanks And Regards,
Nitin Phuria
Confidentiality Disclaimer: “The information contained in this electronic
message (email) and any attachments to this email are intended for the
exclusive use of the addressee(s) and access to this email by anyone else is
unauthorized. The email may contain proprietary, confidential or privileged
information or information relating to Integra Group. If you are not the
intended recipient, please notify the sender by telephone, fax, or return email
and delete this communication and any attachments thereto, immediately from
your computer. Any dissemination, distribution, or copying of this
communication and the attachments thereto (in whole or part), in any manner, is
strictly prohibited and actionable at law. The recipient acknowledges that
emails are susceptible to alteration and their integrity cannot be guaranteed
and that Company does not guarantee that any e-mail is virus-free and accept no
liability for any damage caused by any virus transmitted by this email.”
-----Original Message-----
From: Jonathan Valliere [mailto:[email protected]]
Sent: 09 April 2020 18:09
To: Nitin Phuria
Cc: [email protected]
Subject: Re: MINA: ExecutorFilter on NioSocketConnector
Mina is a queue and flush framework. All writes are queued up and scheduled
later using the poll/kqueue system.
The ExecutorFilter uses threading to process incoming messages using a pool
of threads instead of the reactor/processor thread. This frees the
processor thread to do IO.
Have you profiled the application to figure out which code is consuming the
CPU?
On Thu, Apr 9, 2020 at 3:39 AM Nitin Phuria <[email protected]>
wrote:
> Dear All,
>
>
>
> We have Server developed using MINA 2.0.16 where it
> connects
> to two other systems say System-A and System-B.
>
>
>
> The connectivity with both these system is persistence which means our
> server connects and create only single session with each of these systems
> and on the same session multiple message exchanges (Request/Response)
> happens.
>
>
>
> The current flow of message is as below:
>
> Once we connect with System-B and have persistence connection (single
> session), System-B will send us message which we have to process in our
> server and then send processed message to System-A then get response from
> System-A and then pass it back to System-B
>
>
>
> We have provision that if load increases we can have multiple instances of
> System-A and our server can establish persistence connection (single
> session) with each of these System-A instances and manage the load.
>
>
>
> We are facing problem with load coming from System-B to our server as it is
> coming on single session and we could see that the read/write on the
> session
> are slowed down and lot of queue build-up is happening for read/write.
>
>
>
> Below is the code for establishing the Connectivity with System-B where we
> have not added the ExecutorFilter
>
>
>
> SocketAddress socketAddress = new InetSocketAddress(this.hostName,
> this.port);
>
> NioSocketConnector connector = new NioSocketConnector();
>
> connector.setConnectTimeoutMillis(30 * 1000L);
>
> connector.getSessionConfig().setTcpNoDelay(true);
>
>
> connector.getSessionConfig().setIdleTime(IdleStatus.BOTH_IDLE,this.echoTime)
> ;
>
> this.connector.setHandler(systemBHandler); // systemBHandler is our
> IoHandler implementationfor System-B
>
> IoFilter codecFilter = new
> ProtocolCodecFilter((ProtocolCodecFactory)systemBProcFactory);//
> systemBProcFactory is our Codec Implementation
>
> DefaultIoFilterChainBuilder chain = this.connector.getFilterChain();
>
> chain.clear();
>
> chain.addLast("codec", codecFilter);
>
>
>
> https://cwiki.apache.org/confluence/display/MINA/Configuring+Thread+Model
>
>
> From this link we could understand as below
>
>
>
> If We didn't add any ExecutorFilter events are forwarded to an IoHandler
> via
> method invocations. This means your business logic in your IoHandler
> implementation will run in the I/O processor thread. We call this thread
> model a 'single thread model'. The single thread model is known to be
> adequate for low-latency network applications with CPU-bound business logic
> (e.g. game servers).
>
>
>
> Do we need to add the ExecutorFilter for above mentioned NioSocketConnector
> to System-B so that I/O processor thread will only do the I/O job and all
> other business logic will be given to worker threads in ExecutorFilter.
>
>
>
> Is there any way to increase the number of threads for my I/O to do the
> read/write as we have restriction to create only one session with System-B.
>
>
>
> If we add ExecutorFilter with IoEventType for read/write then will that
> make the read/write faster with System-B on same single session.
>
>
>
> Or we add ExecutorFilter with default IoEventType (All Events) will that
> make the read/write faster with System-B on same single session.
>
>
>
> Pls help us understand the working on ExecutorFilter in above scenario
> where
> NioSocketConnector has only single session to do all the read/write with
> huge load of almost 60-70 messages per second with average message size of
> 3-4KiloBytes.
>
>
>
> Currently we have server having 16 Core with 64 GB RAM.
>
>
>
> Thanks And Regards,
>
> Nitin Phuria
>
>
>
> Confidentiality Disclaimer: "The information contained in this electronic
> message (email) and any attachments to this email are intended for the
> exclusive use of the addressee(s) and access to this email by anyone else
> is
> unauthorized. The email may contain proprietary, confidential or privileged
> information or information relating to Integra Group. If you are not the
> intended recipient, please notify the sender by telephone, fax, or return
> email and delete this communication and any attachments thereto,
> immediately
> from your computer. Any dissemination, distribution, or copying of this
> communication and the attachments thereto (in whole or part), in any
> manner,
> is strictly prohibited and actionable at law. The recipient acknowledges
> that emails are susceptible to alteration and their integrity cannot be
> guaranteed and that Company does not guarantee that any e-mail is
> virus-free
> and accept no liability for any damage caused by any virus transmitted by
> this email."
>
>
> --
> *Confidentiality Disclaimer**: "The information contained in this
> electronic message
> (email) and any attachments to this email are intended
> for the exclusive use of
> the addressee(s) and access to this email by
> anyone else is unauthorized. The
> email may contain proprietary,
> confidential or privileged information or
> information relating to Integra
> Group. If you are not the intended recipient,
> please notify the sender by
> telephone, fax, or return email and delete this
> communication and any
> attachments thereto, immediately from your computer. Any
> dissemination,
> distribution, or copying of this communication and the
> attachments thereto
> (in whole or part), in any manner, is strictly prohibited
> and actionable at
> law. The recipient acknowledges that emails are susceptible
> to alteration
> and their integrity cannot be guaranteed and that Company does
> not
> guarantee that any e-mail is virus-free and accept no liability for any
>
> damage caused by any virus transmitted by this email."*
>
--
CONFIDENTIALITY NOTICE: The contents of this email message and any
attachments are intended solely for the addressee(s) and may contain
confidential and/or privileged information and may be legally protected
from disclosure.
--
*Confidentiality Disclaimer**: “The information contained in this
electronic message
(email) and any attachments to this email are intended
for the exclusive use of
the addressee(s) and access to this email by
anyone else is unauthorized. The
email may contain proprietary,
confidential or privileged information or
information relating to Integra
Group. If you are not the intended recipient,
please notify the sender by
telephone, fax, or return email and delete this
communication and any
attachments thereto, immediately from your computer. Any
dissemination,
distribution, or copying of this communication and the
attachments thereto
(in whole or part), in any manner, is strictly prohibited
and actionable at
law. The recipient acknowledges that emails are susceptible
to alteration
and their integrity cannot be guaranteed and that Company does
not
guarantee that any e-mail is virus-free and accept no liability for any
damage caused by any virus transmitted by this email.”*
--
CONFIDENTIALITY NOTICE: The contents of this email message and any attachments
are intended solely for the addressee(s) and may contain confidential and/or
privileged information and may be legally protected from disclosure.
Confidentiality Disclaimer: “The information contained in this electronic
message (email) and any attachments to this email are intended for the
exclusive use of the addressee(s) and access to this email by anyone else is
unauthorized. The email may contain proprietary, confidential or privileged
information or information relating to Integra Group. If you are not the
intended recipient, please notify the sender by telephone, fax, or return email
and delete this communication and any attachments thereto, immediately from
your computer. Any dissemination, distribution, or copying of this
communication and the attachments thereto (in whole or part), in any manner, is
strictly prohibited and actionable at law. The recipient acknowledges that
emails are susceptible to alteration and their integrity cannot be guaranteed
and that Company does not guarantee that any e-mail is virus-free and accept no
liability for any damage caused by any virus transmitted by this email.”
--
CONFIDENTIALITY NOTICE: The contents of this email message and any attachments
are intended solely for the addressee(s) and may contain confidential and/or
privileged information and may be legally protected from disclosure.
Confidentiality Disclaimer: “The information contained in this electronic
message (email) and any attachments to this email are intended for the
exclusive use of the addressee(s) and access to this email by anyone else is
unauthorized. The email may contain proprietary, confidential or privileged
information or information relating to Integra Group. If you are not the
intended recipient, please notify the sender by telephone, fax, or return email
and delete this communication and any attachments thereto, immediately from
your computer. Any dissemination, distribution, or copying of this
communication and the attachments thereto (in whole or part), in any manner, is
strictly prohibited and actionable at law. The recipient acknowledges that
emails are susceptible to alteration and their integrity cannot be guaranteed
and that Company does not guarantee that any e-mail is virus-free and accept no
liability for any damage caused by any virus transmitted by this email.”
--
CONFIDENTIALITY NOTICE: The contents of this email message and any attachments
are intended solely for the addressee(s) and may contain confidential and/or
privileged information and may be legally protected from disclosure.
--
*Confidentiality Disclaimer**: “The information contained in this
electronic message
(email) and any attachments to this email are intended
for the exclusive use of
the addressee(s) and access to this email by
anyone else is unauthorized. The
email may contain proprietary,
confidential or privileged information or
information relating to Integra
Group. If you are not the intended recipient,
please notify the sender by
telephone, fax, or return email and delete this
communication and any
attachments thereto, immediately from your computer. Any
dissemination,
distribution, or copying of this communication and the
attachments thereto
(in whole or part), in any manner, is strictly prohibited
and actionable at
law. The recipient acknowledges that emails are susceptible
to alteration
and their integrity cannot be guaranteed and that Company does
not
guarantee that any e-mail is virus-free and accept no liability for any
damage caused by any virus transmitted by this email.”*
