No matter how we tune JVM, there's still a certain threshold that when it is reached, a server should stop taking new requests or even close the session to free up resources, depends on the logic in the server.
I observed and categorized 2 OOM scenarios in our Mina-based server: [1] Incoming reqs from SocketAcceptors are too fast and too many. Input queue blows up. Particularly when the Mina-based server has persistent or long-lived connections w/ the clients which just keep pumping in reqs w/o even needing time to establish a connection. [2] Outgoing reqs from SocketConnectors to a service provider are too fast/too many, much faster than the service provider can handle. Then the output queue (writeRequestQueue) blows up. I reported this problem earlier but I can live with this now by tweaking the ReadThrottle filter, though it is very hard to determine the threshold, especially when the mina-based server has multiple Acceptors. I end up assigning a "quota" per Acceptor. Regards On 4/28/07, Mark Webb <[EMAIL PROTECTED]> wrote:
have you looked into tweaking your JVM? http://java.sun.com/performance/reference/whitepapers/tuning.html On 4/28/07, mat <[EMAIL PROTECTED]> wrote: > I also observed that the memory increaed way too fast when heavy loading. > > 2007/4/26, Gaston Dombiak <[EMAIL PROTECTED]>: > > > > Hey Mark, > > > > Based on the statistics I've been collecting from MINA I can tell that > > our OOM problems happen when: > > > > 1) the eventQueue instance variable inside of ExecutorFilter fills up > > (that would be the case for incoming traffic) or > > 2) the SocketSessionImpl#getScheduledWriteRequests() queue starts to > > fill up (that would be the case for outgoing traffic) > > > > Regards, > > > > -- Gato > > > > -----Original Message----- > > From: Mark Webb [mailto:[EMAIL PROTECTED] > > Sent: Wednesday, April 25, 2007 1:53 PM > > To: dev@mina.apache.org > > Subject: Re: Permanent solution for OOM errors > > > > Maybe one approach would be to categorize the ways in which people > > reach OOM conditions. Do these conditions happen only when the input > > and/or output queues get really large, or are there other reasons. > > Once we can start to categorize the OOM conditions that users are > > experiencing, we can properly move forward with solutions. > > > > Basically what I mean is, if the OOM happens when queues fill up, then > > throttle filters would be best. If the OOM happen for other reasons, > > people could look at other areas of MINA. Just not sure there is one > > solution that will fix all the problems. > > > > ...just my 2 cents. > > > > -- > > ..Cheers > > Mark > > > > > > On 4/25/07, Gaston Dombiak <[EMAIL PROTECTED]> wrote: > > > Hey Marcin, > > > > > > I see what you mean. I'm really fine with either option as long as > > MINA > > > handles the OOM problem. As I posted a month or so ago the throttling > > > per connection is not enough. We (Openfire and probably others) also > > > need a global throttling. > > > > > > Thanks, > > > > > > -- Gato > > > > > > -----Original Message----- > > > From: Marcin Waldowski [mailto:[EMAIL PROTECTED] > > > Sent: Wednesday, April 25, 2007 1:36 PM > > > To: dev@mina.apache.org > > > Subject: Re: Permanent solution for OOM errors > > > > > > Hej Gaston, > > > > > > Gaston Dombiak wrote: > > > > Hey Marcin, > > > > > > > > I'm curious to hear from you why do you think so? I absolutely agree > > > > that MINA has to provide easy ways to handle OOM problems due to > > heavy > > > > incoming or outgoing traffic. > > > > > > > > This is by far the more common problem people are reporting with > > > > Openfire now that we moved to MINA (when under heavy load). > > > > > > > > > > Hmm, my first thought was that it introduce the complexity of > > > configuration like ThreadModel. But probably I'm wrong. > > > > > > I agree definitely that MINA need solution for OOM, but it could be > > also > > > > > > ReadThrottleFilterBuilder and WriteThrottleFilter. They are easy way > > to > > > handle OOM and (what is important) users are aware that they use it. > > > > > > But I'm not agains integrate it as default to Acceptor/Connector, > > really > > > > > > :) Meybe I even souldn't post my doubts to this subject because I > > > haven't contributed to MINA so far... > > > > > > Regards, Marcin > > > > > > > > > > > > -- ..Cheers Mark
