The Mina team are well aware of the problem and are aiming to address it in the 2.0 release.
As a starting point you may want to have a look at this filter. Currently it only limits on message count but could quite easily get the message size and limit on that. I do recall doing that but perhaps never uploaded that version. https://issues.apache.org/jira/browse/DIRMINA-302 On 12/09/2007, Rajith Attapattu <[EMAIL PROTECTED]> wrote: > On 9/11/07, Rafael Schloming <[EMAIL PROTECTED]> wrote: > > > > > > > > Rajith Attapattu wrote: > > > On 9/11/07, Rafael Schloming <[EMAIL PROTECTED]> wrote: > > >> Rajith Attapattu wrote: > > >>> Jodi, > > >>> > > >>> Thanks for the feedback. > > >>> Comments inline > > >>> > > >>> Regards, > > >>> > > >>> Rajith > > >>> > > >>> On 9/11/07, Jodi Moran <[EMAIL PROTECTED]> wrote: > > >>>> Hi all, > > >>>> > > >>>> I'm having problems using the Qpid M2 Java client (JMS to AMQP) for > > >>>> publishing messages in a load test. When I publish messages as > > quickly > > >>>> as possible, the publishing client runs out of memory (no matter what > > >>>> limit I set). I am using only the minimum of the JMS interface: > > >>>> > > >>>> InitialContext jndiContext = new > > >>>> InitialContext(additionalJNDIProps); > > >>>> connectionFactory = (ConnectionFactory) > > >>>> jndiContext.lookup(connectionFactoryJNDIName); > > >>>> destination = (Destination) jndiContext.lookup > > (topicName); > > >>>> connection = connectionFactory.createConnection(); > > >>>> jmsSession = connection.createSession(false, > > >>>> Session.AUTO_ACKNOWLEDGE); > > >>>> jmsMessageProducer = jmsSession.createProducer > > >> (destination); > > >>>> jmsMessageProducer.setDeliveryMode(DeliveryMode.NON_PERSISTENT); > > >>>> > > >>>> And later, in a loop: > > >>>> > > >>>> BytesMessage message = jmsSession.createBytesMessage(); > > >>>> message.writeBytes(messageContent); > > >>>> jmsMessageProducer.send(message); > > >>>> > > >>>> After making use of profilers and heap dumps, it appears that the OOM > > >> is > > >>>> caused by the fact that the publishing thread by default does not > > block > > >>>> during the send but just adds the write request to an (unbounded) > > queue > > >>>> internal to Mina. Since in my case (it seems) the I/O is slower than > > >> the > > >>>> publishing thread, the write request queue continues to grow until it > > >>>> causes the OOM. > > >>>> > > >>>> I've noticed that there is functionality in the BasicMessageProducer > > >>>> that allows the user to block on writes (_waitUntilSent), but it > > seems > > >>>> that this functionality is not even exposed via the extended > > interfaces > > >>>> (i.e. org.apache.qpid.jms.MessageProducer or > > >>>> org.apache.qpid.jms.Session) and so requires a cast to > > >>>> BasicMessageProducer or to AMQSession to use. Is the only way to get > > >>>> flow control in my publishing client to make use of _waitUntilSent or > > >> is > > >>>> there some other way I can achieve the same effect? > > >>> > > >>> Currently this is the only way to set this. > > >>> However we could provide a jvm argument to set it, so that u don't > > have > > >> to > > >>> cast it to any AMQ specific class. > > >>> We might repsin the M2 release again. We can add this feature if it > > >> helps. > > >>> Doing this block will slow down your application. > > >>> Without the block atleast your application can continue publishing at > > a > > >>> higher rate, until the internal MINA queue starts to grow due to IO > > >> being > > >>> slow. > > >>> Is there any way you can throttle the publish rate in your > > application? > > >>> After some experimenting you maybe able to find a sweet spot that is > > >> right > > >>> for your environment. This might yeild a higher average publish rate > > >> than a > > >>> block for every publish type of solution. > > >> We should really do the throttling automatically, i.e. we should block > > >> when the MINA queue exceeds a certain limit, but not if it is below > > that > > >> limit > > > > > > > > > Rafi, I thought about this initially, but since this queue is internal > > to > > > MINA, I wasn't sure if we can know the current queue size etc ? > > > > Well you can modify MINA per Robert's suggestion in another post, > > however I don't think you actually care about the queue size. What you > > care about is how much memory the queue consumes, and this is strictly > > speaking independent of the queue size. > > > I agree that a byte limit is the proper solution. > My suggestion of queue size was a very quick hack for Jodi, based on two > simple assumptions. > a) message sizes are fairly simillar. (for Jodi's use case) > b) no of objects in queue * message size will give a rough idea about the > memory consumption. > > I believe in most cases where you have a high publishing rate, will involve > small messages that are approximately simillar in size. > So a queue size will give a rough estimation of the memory consumption. > So if MINA can provide us a bounded queue, we can implement this as a simple > solution. > > --rajith > -- Martin Ritchie
