After curbing the read rate (one quota per acceptor) at a conservative rate, the write problem is automatically gone.
This is not an approach that leads to optimal performance, but just a way to protect our Mina-based server not run into OOM. Cheers On 4/28/07, mat <[EMAIL PROTECTED]> wrote:
Does ReadThrottle filter only solve the "Read" problem? What about "Write"? BTW, what is the max concurrent users you can reach? 2007/4/29, Paul Chen <[EMAIL PROTECTED]>: > > No matter how we tune JVM, there's still a certain threshold that when it > is > reached, a server should stop taking new requests or even close the > session > to free up resources, depends on the logic in the server. > > I observed and categorized 2 OOM scenarios in our Mina-based server: > > [1] Incoming reqs from SocketAcceptors are too fast and too many. Input > queue blows up. Particularly when the Mina-based server has persistent or > long-lived connections w/ the clients which just keep pumping in reqs w/o > even needing time to establish a connection. > > [2] Outgoing reqs from SocketConnectors to a service provider are too > fast/too many, much faster > than the service provider can handle. Then the output queue > (writeRequestQueue) blows up. > > I reported this problem earlier but I can live with this now by tweaking > the > ReadThrottle filter, though > it is very hard to determine the threshold, especially when the > mina-based > server has multiple > Acceptors. I end up assigning a "quota" per Acceptor. > > Regards > > On 4/28/07, Mark Webb <[EMAIL PROTECTED]> wrote: > > > > have you looked into tweaking your JVM? > > > > http://java.sun.com/performance/reference/whitepapers/tuning.html > > > > On 4/28/07, mat <[EMAIL PROTECTED]> wrote: > > > I also observed that the memory increaed way too fast when heavy > > loading. > > > > > > 2007/4/26, Gaston Dombiak <[EMAIL PROTECTED]>: > > > > > > > > Hey Mark, > > > > > > > > Based on the statistics I've been collecting from MINA I can tell > that > > > > our OOM problems happen when: > > > > > > > > 1) the eventQueue instance variable inside of ExecutorFilter fills > up > > > > (that would be the case for incoming traffic) or > > > > 2) the SocketSessionImpl#getScheduledWriteRequests() queue starts to > > > > fill up (that would be the case for outgoing traffic) > > > > > > > > Regards, > > > > > > > > -- Gato > > > > > > > > -----Original Message----- > > > > From: Mark Webb [mailto:[EMAIL PROTECTED] > > > > Sent: Wednesday, April 25, 2007 1:53 PM > > > > To: dev@mina.apache.org > > > > Subject: Re: Permanent solution for OOM errors > > > > > > > > Maybe one approach would be to categorize the ways in which people > > > > reach OOM conditions. Do these conditions happen only when the > input > > > > and/or output queues get really large, or are there other reasons. > > > > Once we can start to categorize the OOM conditions that users are > > > > experiencing, we can properly move forward with solutions. > > > > > > > > Basically what I mean is, if the OOM happens when queues fill up, > then > > > > throttle filters would be best. If the OOM happen for other > reasons, > > > > people could look at other areas of MINA. Just not sure there is > one > > > > solution that will fix all the problems. > > > > > > > > ...just my 2 cents. > > > > > > > > -- > > > > ..Cheers > > > > Mark > > > > > > > > > > > > On 4/25/07, Gaston Dombiak <[EMAIL PROTECTED]> wrote: > > > > > Hey Marcin, > > > > > > > > > > I see what you mean. I'm really fine with either option as long as > > > > MINA > > > > > handles the OOM problem. As I posted a month or so ago the > > throttling > > > > > per connection is not enough. We (Openfire and probably others) > also > > > > > need a global throttling. > > > > > > > > > > Thanks, > > > > > > > > > > -- Gato > > > > > > > > > > -----Original Message----- > > > > > From: Marcin Waldowski [mailto:[EMAIL PROTECTED] > > > > > Sent: Wednesday, April 25, 2007 1:36 PM > > > > > To: dev@mina.apache.org > > > > > Subject: Re: Permanent solution for OOM errors > > > > > > > > > > Hej Gaston, > > > > > > > > > > Gaston Dombiak wrote: > > > > > > Hey Marcin, > > > > > > > > > > > > I'm curious to hear from you why do you think so? I absolutely > > agree > > > > > > that MINA has to provide easy ways to handle OOM problems due to > > > > heavy > > > > > > incoming or outgoing traffic. > > > > > > > > > > > > This is by far the more common problem people are reporting with > > > > > > Openfire now that we moved to MINA (when under heavy load). > > > > > > > > > > > > > > > > Hmm, my first thought was that it introduce the complexity of > > > > > configuration like ThreadModel. But probably I'm wrong. > > > > > > > > > > I agree definitely that MINA need solution for OOM, but it could > be > > > > also > > > > > > > > > > ReadThrottleFilterBuilder and WriteThrottleFilter. They are easy > way > > > > to > > > > > handle OOM and (what is important) users are aware that they use > it. > > > > > > > > > > But I'm not agains integrate it as default to Acceptor/Connector, > > > > really > > > > > > > > > > :) Meybe I even souldn't post my doubts to this subject because I > > > > > haven't contributed to MINA so far... > > > > > > > > > > Regards, Marcin > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > ..Cheers > > Mark > > >
