Hi, folks,
Here is what I did to solve the OOM problem caused by writeRequest Queues
piling up
when many SocketConnector sessions bombard a service provider thus making
it very slow in responding.
It is the sum of all writeRequest queues that blows up the memory.
The idea is simple, but, alas, it is not that generic. But you get the idea
when reading
the code comments.
Not sure what's the best way/place to modify the Mina src to create global
views
across sessions?
Thanks for comments
package ....
import org.apache.mina.common.IoFilterAdapter;
import org.apache.mina.common.IoSession;
/**
* Prevent out of memory caused by output write queue piling up
*
* This filter is attached on a Mina SocketConnector session to:
*
* - Check outputQ quota of a service provider, the quota implicitly
indicates the max capacity of the
* service provider
*
* This filter complements the Mina ReadThrottleFilter which I attach to
each acceptor opened by my
* Mina-based server, and assign a max quota to each. I preconfigure them to
make sure the
* total of all acceptor quota values don't exceed my system memory limit.
*/
public class WriteThrottleFilter extends IoFilterAdapter {
/** Write a request to a service provider via a SocketConnector */
@Override
public void filterWrite(NextFilter nextFilter, IoSession session,
WriteRequest writeRequest) throws Exception {
// A codec turned it to my Request obj already
Request req = (Request) message;
int reqSize = req.size();
String svc = req.getServiceName();
// Check outputQ quota for the svc
//
// Make sure we don't exceed max capacity pre-configured for this
service provider
// There can be many SocketConnectors sending reqs to the service
provider. When
// it is slow, all write queues connected to this service pile up in
Mina. Thus I
// need a global view instead of a single
session.getScheduledWriteBytes()
if (reqSize + svcProviders.outQueueSize(svc) >
svcProviders.outQueueQuota(svc)) {
// Mark the service as busy, reply an error response to the
client
return;
}
svcProviders.incrOutQueue(svc, reqSize);
nextFilter.filterWrite(session, writeRequest);
}
} // End of class
On 5/3/07, Martin Ritchie <[EMAIL PROTECTED]> wrote:
That is great. I look forward to using the new filter as we have been
having a lot more OOM problems on our clients under load testing
(although that is as much a fault of the test app).
On 03/05/07, Vinod Panicker <[EMAIL PROTECTED]> wrote:
> I've already done that. It's being tested right now. We'll be
> contributing that as soon as we are happy about the performance under
> high load.
>
> On 5/2/07, Martin Ritchie <[EMAIL PROTECTED]> wrote:
> > On 30/04/07, Paul Chen <[EMAIL PROTECTED]> wrote:
> > > I'm refining that to make it more presentable and generic, will
publish
> > that
> > > in 1 week or 2.
> > >
> > > On 4/30/07, Gaston Dombiak <[EMAIL PROTECTED]> wrote:
> > > >
> > > > Hey Paul,
> > > >
> > > > That is exactly what I was planning to implement for our case. Can
you
> > > > send me your modified version? I would like to include it in
Openfire.
> > > >
> > > > Thanks,
> > > >
> > > > -- Gato
> > > >
> > > > -----Original Message-----
> > > > From: Paul Chen [mailto:[EMAIL PROTECTED]
> > > > Sent: Monday, April 30, 2007 1:07 PM
> > > > To: dev@mina.apache.org
> > > > Subject: Re: Permanent solution for OOM errors
> > > >
> > > > I actually *tweaked* the read throttle to take the output Queue
size
> > > > into account to avoid the slow write syndrome. Maybe I should call
> > > > it traffic control throttle instead.
> > > >
> > > > Cheers
> > > >
> > > > On 4/29/07, Vinod Panicker <[EMAIL PROTECTED]> wrote:
> > > > >
> > > > > On 4/29/07, Paul Chen <[EMAIL PROTECTED]> wrote:
> > > > > > After curbing the read rate (one quota per acceptor) at a
> > > > conservative
> > > > > rate,
> > > > > > the write problem is automatically gone.
> > > > >
> > > > > Not really. The read quota can be applied per acceptor, but the
> > > > > server might have connectors that could be writing to slower
clients,
> > > > > leading to another OOM.
> >
> > Hi,
> >
> > I've been watching this thread grow but not had time to contribute
> > (Sorry about that).
> >
> > The Qpid project is hoping to move the latest released version of mina
> > and so I would be happy to take another look at the
> > WriteThrottleFilter (on DIRMINA-302) to see how it can be tidied up.
> > Specifically the filter currently limits by number of requests whereas
> > the number of bytes on the queue would be far better solution.
> >
> > --
> > Martin Ritchie
> >
>
--
Martin Ritchie
