Make sure you enable BBR TCP on your host machines to help mitigate the network problems.
On Thu, Apr 9, 2020 at 10:58 AM Nitin Phuria <nit...@integramicro.com> wrote: > Dear Jonathan, > > > > Thanks for bringing more clarity on Processor Thread and > it's relation with IoSession. We are also suspecting the network connection > and in a crisis one should pull all the strings to check where you get the > lead to identify the problem and that's what we are doing. > > > > Thanks a tone for all your replies and giving very useful information. We > will check further on network connection. > > > > Thanks And Regards, > > Nitin Phuria > > > > *Confidentiality Disclaimer**: “The information contained in this > electronic message (email) and any attachments to this email are intended > for the exclusive use of the addressee(s) and access to this email by > anyone else is unauthorized. The email may contain proprietary, > confidential or privileged information or information relating to Integra > Group. If you are not the intended recipient, please notify the sender by > telephone, fax, or return email and delete this communication and any > attachments thereto, immediately from your computer. Any dissemination, > distribution, or copying of this communication and the attachments thereto > (in whole or part), in any manner, is strictly prohibited and actionable at > law. The recipient acknowledges that emails are susceptible to alteration > and their integrity cannot be guaranteed and that Company does not > guarantee that any e-mail is virus-free and accept no liability for any > damage caused by any virus transmitted by this email.”* > > > > *From:* Jonathan Valliere [mailto:jon.valli...@emoten.com] > *Sent:* 09 April 2020 20:03 > *To:* Nitin Phuria > *Cc:* users@mina.apache.org > *Subject:* Re: MINA: ExecutorFilter on NioSocketConnector > > > > Every IoSession is bound to one Processor Thread. So only by creating > more IoSessions can you take advantage of multiple Processor Threads. > > > > ExecutorFilter only works for received messages. So it generally does > nothing on the client side however it will cause the received messages to > be executed in multiple threads. > > > > The purpose of the ExecutorFilter is to do potentially blocking tasks in > there so as not to block the Processor Thread from handling tasks from > other IoSessions. > > > > Since you’re seeing a huge jump from 15ms to several minutes and a very > small number of messages it is most likely a problem with your network > connection. > > > > On Thu, Apr 9, 2020 at 10:26 AM Nitin Phuria <nit...@integramicro.com> > wrote: > > Dear Jonathan, > > Our Server and System-B are on WAN network with leased line. > > > > One more clarification required As per documentation > > > > NioSocketConnector > <http://mina.apache.org/mina-project/gen-docs/2.0.10/apidocs/org/apache/mina/transport/socket/nio/NioSocketConnector.html#NioSocketConnector--> > () Constructor for NioSocketConnector > <http://mina.apache.org/mina-project/gen-docs/2.0.10/apidocs/org/apache/mina/transport/socket/nio/NioSocketConnector.html> > with default configuration (multiple thread model). > > > > So when we create NioSocketConnector connector = new NioSocketConnector(); I > expect it to create n+1 processor > > where n is number of core on the hardware. In our case it is 17 (16 core > machine) > > > > When we took threaddump of our Server process I could see only one thread > for this connector and in the log (log4j) when we print the thread name it > prints NioProcessor-x where x is some number. > > > > So does it means that n+1 processors use single thread to work. > > > > Also you mentioned that The ExecutorFilter only works for the Server > receiver side. > > > > > http://apache-mina.10907.n7.nabble.com/OrderedThreadPoolExecutor-shutdown-problem-in-MINA-2-0-0-M4-td34417.html > > > > But I could see from this above URL that we can add it to Connector also > and it says "*My understanding is that the Executor filter just adds an > executor in the chain which will be used to spread the load on many > threads. This is an optimization, rather than something you need to use. If > you don't use it, your program will work fine.*" > > > > Could you provide more insight and your thoughts. > > > > Thanks And Regards, > > Nitin Phuria > > > > *Confidentiality Disclaimer**: “The information contained in this > electronic message (email) and any attachments to this email are intended > for the exclusive use of the addressee(s) and access to this email by > anyone else is unauthorized. The email may contain proprietary, > confidential or privileged information or information relating to Integra > Group. If you are not the intended recipient, please notify the sender by > telephone, fax, or return email and delete this communication and any > attachments thereto, immediately from your computer. Any dissemination, > distribution, or copying of this communication and the attachments thereto > (in whole or part), in any manner, is strictly prohibited and actionable at > law. The recipient acknowledges that emails are susceptible to alteration > and their integrity cannot be guaranteed and that Company does not > guarantee that any e-mail is virus-free and accept no liability for any > damage caused by any virus transmitted by this email.”* > > > > *From:* Jonathan Valliere [mailto:jon.valli...@emoten.com] > *Sent:* 09 April 2020 19:34 > *To:* Nitin Phuria > *Cc:* users@mina.apache.org > *Subject:* Re: MINA: ExecutorFilter on NioSocketConnector > > > > How far are the servers? TCP Packet loss could explain this. > > > > You can get the Queue size from the IoSession object. > > > > On Thu, Apr 9, 2020 at 10:02 AM Nitin Phuria <nit...@integramicro.com> > wrote: > > Dear Jonathan, > > > > Anytime we write message to System-B session with session.write method the > messageSent event has to be generated with it actually gets written on > TCP/IP > > > > org.apache.mina.core.session.IoSession.write(Object) > > > > We see that there is time difference of 10-14 seconds between timestamp of > session.write for particular message and messageSent event getting > generated for the same message. > > > > In non-peak load this time difference is in 10-15 milliseconds and in peak > load (60-70 messages per second) it goes till 10-14 seconds. > > > > Is there anything that we can log to see what is happening or see the > queue size etc.. for further better understaning. > > > > Thanks And Regards, > > Nitin Phuria > > > > *Confidentiality Disclaimer**: “The information contained in this > electronic message (email) and any attachments to this email are intended > for the exclusive use of the addressee(s) and access to this email by > anyone else is unauthorized. The email may contain proprietary, > confidential or privileged information or information relating to Integra > Group. If you are not the intended recipient, please notify the sender by > telephone, fax, or return email and delete this communication and any > attachments thereto, immediately from your computer. Any dissemination, > distribution, or copying of this communication and the attachments thereto > (in whole or part), in any manner, is strictly prohibited and actionable at > law. The recipient acknowledges that emails are susceptible to alteration > and their integrity cannot be guaranteed and that Company does not > guarantee that any e-mail is virus-free and accept no liability for any > damage caused by any virus transmitted by this email.”* > > > > *From:* Jonathan Valliere [mailto:jon.valli...@emoten.com] > *Sent:* 09 April 2020 18:31 > *To:* Nitin Phuria > *Cc:* Nitin Phuria; users@mina.apache.org > *Subject:* Re: MINA: ExecutorFilter on NioSocketConnector > > > > The ExecutorFilter only works for the Server receiver side. > > > > When you say the responses are slow, what do you consider slow? > > > > All writes are put into a queue dedicated for each session. The Session > is then enabled for writing by adding the write interest to the poll > mechanism. Reads and Writes IO happen in the same thread. After the > Session finishes reading it will then call the poll mechanism for new > events to process. Hopefully it should get the write event for your Session > then begin to flush the queue. > > > > Processor Thread Loop —> Select Poll Events —> Process Events > (read/write) —> Loop > > > > On Thu, Apr 9, 2020 at 8:53 AM Nitin Phuria <nit...@integramicro.com> > wrote: > > Dear Jonathan, > > Thanks for the reply, can you provide more details on the "All > writes are queued up and scheduled > later using the poll/kqueue system." How this poll & kqueue system works? > Do we have any control on this from MINA framework? > > We are not facing any problem with the higher CPU consumption but we see > that when there is high load from System-B to our server the reads from > System-B and write back to System-B on the single persistence session is > becoming slow. So we were looking at i/o operations and how to make them > fast. > > With your explanation provided in below email about ExecutorFilter, If we > add it in our connector after codec filter will help us reduce the I/O > overhead on reactor/processor thread. I hope my understanding is right > kindly confirm? > > > Thanks And Regards, > Nitin Phuria > > Confidentiality Disclaimer: “The information contained in this electronic > message (email) and any attachments to this email are intended for the > exclusive use of the addressee(s) and access to this email by anyone else > is unauthorized. The email may contain proprietary, confidential or > privileged information or information relating to Integra Group. If you are > not the intended recipient, please notify the sender by telephone, fax, or > return email and delete this communication and any attachments thereto, > immediately from your computer. Any dissemination, distribution, or copying > of this communication and the attachments thereto (in whole or part), in > any manner, is strictly prohibited and actionable at law. The recipient > acknowledges that emails are susceptible to alteration and their integrity > cannot be guaranteed and that Company does not guarantee that any e-mail is > virus-free and accept no liability for any damage caused by any virus > transmitted by this email.” > > -----Original Message----- > From: Jonathan Valliere [mailto:jon.valli...@emoten.com] > Sent: 09 April 2020 18:09 > To: Nitin Phuria > Cc: users@mina.apache.org > Subject: Re: MINA: ExecutorFilter on NioSocketConnector > > Mina is a queue and flush framework. All writes are queued up and scheduled > later using the poll/kqueue system. > > The ExecutorFilter uses threading to process incoming messages using a pool > of threads instead of the reactor/processor thread. This frees the > processor thread to do IO. > > Have you profiled the application to figure out which code is consuming the > CPU? > > On Thu, Apr 9, 2020 at 3:39 AM Nitin Phuria <nit...@integramicro.com > .invalid> > wrote: > > > Dear All, > > > > > > > > We have Server developed using MINA 2.0.16 where it > > connects > > to two other systems say System-A and System-B. > > > > > > > > The connectivity with both these system is persistence which means our > > server connects and create only single session with each of these systems > > and on the same session multiple message exchanges (Request/Response) > > happens. > > > > > > > > The current flow of message is as below: > > > > Once we connect with System-B and have persistence connection (single > > session), System-B will send us message which we have to process in our > > server and then send processed message to System-A then get response from > > System-A and then pass it back to System-B > > > > > > > > We have provision that if load increases we can have multiple instances > of > > System-A and our server can establish persistence connection (single > > session) with each of these System-A instances and manage the load. > > > > > > > > We are facing problem with load coming from System-B to our server as it > is > > coming on single session and we could see that the read/write on the > > session > > are slowed down and lot of queue build-up is happening for read/write. > > > > > > > > Below is the code for establishing the Connectivity with System-B where > we > > have not added the ExecutorFilter > > > > > > > > SocketAddress socketAddress = new InetSocketAddress(this.hostName, > > this.port); > > > > NioSocketConnector connector = new NioSocketConnector(); > > > > connector.setConnectTimeoutMillis(30 * 1000L); > > > > connector.getSessionConfig().setTcpNoDelay(true); > > > > > > > connector.getSessionConfig().setIdleTime(IdleStatus.BOTH_IDLE,this.echoTime) > > ; > > > > this.connector.setHandler(systemBHandler); // systemBHandler is our > > IoHandler implementationfor System-B > > > > IoFilter codecFilter = new > > ProtocolCodecFilter((ProtocolCodecFactory)systemBProcFactory);// > > systemBProcFactory is our Codec Implementation > > > > DefaultIoFilterChainBuilder chain = this.connector.getFilterChain(); > > > > chain.clear(); > > > > chain.addLast("codec", codecFilter); > > > > > > > > > https://cwiki.apache.org/confluence/display/MINA/Configuring+Thread+Model > > > > > > From this link we could understand as below > > > > > > > > If We didn't add any ExecutorFilter events are forwarded to an IoHandler > > via > > method invocations. This means your business logic in your IoHandler > > implementation will run in the I/O processor thread. We call this thread > > model a 'single thread model'. The single thread model is known to be > > adequate for low-latency network applications with CPU-bound business > logic > > (e.g. game servers). > > > > > > > > Do we need to add the ExecutorFilter for above mentioned > NioSocketConnector > > to System-B so that I/O processor thread will only do the I/O job and all > > other business logic will be given to worker threads in ExecutorFilter. > > > > > > > > Is there any way to increase the number of threads for my I/O to do the > > read/write as we have restriction to create only one session with > System-B. > > > > > > > > If we add ExecutorFilter with IoEventType for read/write then will that > > make the read/write faster with System-B on same single session. > > > > > > > > Or we add ExecutorFilter with default IoEventType (All Events) will that > > make the read/write faster with System-B on same single session. > > > > > > > > Pls help us understand the working on ExecutorFilter in above scenario > > where > > NioSocketConnector has only single session to do all the read/write with > > huge load of almost 60-70 messages per second with average message size > of > > 3-4KiloBytes. > > > > > > > > Currently we have server having 16 Core with 64 GB RAM. > > > > > > > > Thanks And Regards, > > > > Nitin Phuria > > > > > > > > Confidentiality Disclaimer: "The information contained in this electronic > > message (email) and any attachments to this email are intended for the > > exclusive use of the addressee(s) and access to this email by anyone else > > is > > unauthorized. The email may contain proprietary, confidential or > privileged > > information or information relating to Integra Group. If you are not the > > intended recipient, please notify the sender by telephone, fax, or return > > email and delete this communication and any attachments thereto, > > immediately > > from your computer. Any dissemination, distribution, or copying of this > > communication and the attachments thereto (in whole or part), in any > > manner, > > is strictly prohibited and actionable at law. The recipient acknowledges > > that emails are susceptible to alteration and their integrity cannot be > > guaranteed and that Company does not guarantee that any e-mail is > > virus-free > > and accept no liability for any damage caused by any virus transmitted by > > this email." > > > > > > -- > > *Confidentiality Disclaimer**: "The information contained in this > > electronic message > > (email) and any attachments to this email are intended > > for the exclusive use of > > the addressee(s) and access to this email by > > anyone else is unauthorized. The > > email may contain proprietary, > > confidential or privileged information or > > information relating to Integra > > Group. If you are not the intended recipient, > > please notify the sender by > > telephone, fax, or return email and delete this > > communication and any > > attachments thereto, immediately from your computer. Any > > dissemination, > > distribution, or copying of this communication and the > > attachments thereto > > (in whole or part), in any manner, is strictly prohibited > > and actionable at > > law. The recipient acknowledges that emails are susceptible > > to alteration > > and their integrity cannot be guaranteed and that Company does > > not > > guarantee that any e-mail is virus-free and accept no liability for any > > > > damage caused by any virus transmitted by this email."* > > > -- > > CONFIDENTIALITY NOTICE: The contents of this email message and any > attachments are intended solely for the addressee(s) and may contain > confidential and/or privileged information and may be legally protected > from disclosure. > > > -- > *Confidentiality Disclaimer**: “The information contained in this > electronic message > (email) and any attachments to this email are intended > for the exclusive use of > the addressee(s) and access to this email by > anyone else is unauthorized. The > email may contain proprietary, > confidential or privileged information or > information relating to Integra > Group. If you are not the intended recipient, > please notify the sender by > telephone, fax, or return email and delete this > communication and any > attachments thereto, immediately from your computer. Any > dissemination, > distribution, or copying of this communication and the > attachments thereto > (in whole or part), in any manner, is strictly prohibited > and actionable at > law. The recipient acknowledges that emails are susceptible > to alteration > and their integrity cannot be guaranteed and that Company does > not > guarantee that any e-mail is virus-free and accept no liability for any > > damage caused by any virus transmitted by this email.”* > > -- > > > > CONFIDENTIALITY NOTICE: The contents of this email message and any > attachments are intended solely for the addressee(s) and may > contain confidential and/or privileged information and may be legally > protected from disclosure. > > > *Confidentiality Disclaimer**: “The information contained in this > electronic message (email) and any attachments to this email are intended > for the exclusive use of the addressee(s) and access to this email by > anyone else is unauthorized. The email may contain proprietary, > confidential or privileged information or information relating to Integra > Group. If you are not the intended recipient, please notify the sender by > telephone, fax, or return email and delete this communication and any > attachments thereto, immediately from your computer. Any dissemination, > distribution, or copying of this communication and the attachments thereto > (in whole or part), in any manner, is strictly prohibited and actionable at > law. The recipient acknowledges that emails are susceptible to alteration > and their integrity cannot be guaranteed and that Company does not > guarantee that any e-mail is virus-free and accept no liability for any > damage caused by any virus transmitted by this email.”* > > -- > > > > CONFIDENTIALITY NOTICE: The contents of this email message and any > attachments are intended solely for the addressee(s) and may > contain confidential and/or privileged information and may be legally > protected from disclosure. > > > *Confidentiality Disclaimer**: “The information contained in this > electronic message (email) and any attachments to this email are intended > for the exclusive use of the addressee(s) and access to this email by > anyone else is unauthorized. The email may contain proprietary, > confidential or privileged information or information relating to Integra > Group. If you are not the intended recipient, please notify the sender by > telephone, fax, or return email and delete this communication and any > attachments thereto, immediately from your computer. Any dissemination, > distribution, or copying of this communication and the attachments thereto > (in whole or part), in any manner, is strictly prohibited and actionable at > law. The recipient acknowledges that emails are susceptible to alteration > and their integrity cannot be guaranteed and that Company does not > guarantee that any e-mail is virus-free and accept no liability for any > damage caused by any virus transmitted by this email.”* > > -- > > > > CONFIDENTIALITY NOTICE: The contents of this email message and any > attachments are intended solely for the addressee(s) and may > contain confidential and/or privileged information and may be legally > protected from disclosure. > > *Confidentiality Disclaimer**: “The information contained in this > electronic message (email) and any attachments to this email are intended > for the exclusive use of the addressee(s) and access to this email by > anyone else is unauthorized. The email may contain proprietary, > confidential or privileged information or information relating to Integra > Group. If you are not the intended recipient, please notify the sender by > telephone, fax, or return email and delete this communication and any > attachments thereto, immediately from your computer. Any dissemination, > distribution, or copying of this communication and the attachments thereto > (in whole or part), in any manner, is strictly prohibited and actionable at > law. The recipient acknowledges that emails are susceptible to alteration > and their integrity cannot be guaranteed and that Company does not > guarantee that any e-mail is virus-free and accept no liability for any > damage caused by any virus transmitted by this email.”* -- CONFIDENTIALITY NOTICE: The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure.
