On 6/6/06, D Howe <[EMAIL PROTECTED]> wrote:
We are getting similar problems in ActiveMQ 3.2.1. We have a similar scenario to Frank, where our consumers may be down (e.g. for maintenance) but our producers will continue to produce messages.
Unfortunately 3.x is cannot handle large queues that don't fit in memory - its only strategy once the RAM buffer is used up is to block producers until the consumers catch up. So if you want to deal with fast producers and slow consumers on durable queues well I'd recommend upgrading to 4.0.
I was expecting that with the amount of journal capacity and the Oracle database that I would be able to get in the millions of entries into the queue when there was no consumers, and that they would simply "queue up".
The reason is the design trade off of performance; we've designed ActiveMQ to be very fast (rather than doing a database query every time a message is required to be dispatched etc). So 3.x keeps the pending messages on a queue in RAM. As it happens its amazingly complex efficiently dispatching messages from a massive queue efficiently to an arbitrary number of consumers who maybe using filters so it took us until 4.x to truly figure out how to do it well, fast and to deal with arbitrarily massive queues. The dispatch model of massive queues is one of the major reasons for the changes in 4.x
What actually happens is that around 60,000 messages I start getting lots of messages in the log: 2006-06-02 16:21:22,369 [Thread-0 ] WARN ientQueueBoundedMessageManager - memory limit low - forced to remove expired messages: publish.request Around about 65,000 messages I start getting lots of: 2006-06-02 17:19:05,159 [ocalport=61616]] WARN MemoryBoundedQueue - Queue is full, waiting for it to be dequeued. At this point, my producer seems to lock up and everything stops.
Yes - you've hit the limitation of 3.x - I"d recommend going to 4.x. We've gone tests with millions of messages on a queue and things still work fine.
Is there any way to configure ActiveMQ (3.2.x) to reliably queue up a number of messages, only limited by disk/database capacity?
No - you need to move to 4.x.
The application that we are developing could produce average loads of 30 messages/second, so 65,000 messages is nothing. Even if we up the maxMem, we will still have a limit - and the bigger the maxMem, the worse the consequences are if we reach it as we will lose more messages. Upgrading to 4 is not an option for us at the moment, as we are in final stages of testing and don't want to have to re-test with a version that is not final release.
4.0 is final? -- James ------- http://radio.weblogs.com/0112098/
