Some things you may want to consider in case you haven't already done so.
1. Enable asynchronous sends for the producers. With async sends, you'll also have to deal with flow control, which you could always turn off. If you can't tolerate lost messages, then consider grouping async sends w/in a transaction. 2. Enable asynchronous dispatching to the consumers and maybe increase the prefetch limit? 3. Can you assign more consumers to each queue thus dispersing the load across more consumers? Joe Dan Washusen wrote: > > Hey All, > I'm currently load testing our application with ActiveMQ and I'm seeing > some strange behavior that I'm hoping someone can offer some assistance > with. > > Basically, there are 7 machines participating in the load test; > 1. Running JMeter. > 2. Running JMeter. > 3. Running ActiveMQ 4.1.1. (mel1u105). > 4. Running Producer (mel1u108). > 5. Running Producer (mel1u109). > 6. Running Consumer (mel1u110). > 7. Running Consumer (mel1u111). > 8. Running PostgreSQL (mel1u113). > > ActiveMQ is allocated 1500Mb of memory running on a AMD64 X2, 3.5GB RAM, > ~350GB Raid 5 type machine. For a period the producers are adding around > 1000 messages per second to a queue until (according to the JMX console) > there are around 600000 messages queued. At this point ActiveMQ basically > locks up and consumers slow down to a trickle. > > I've attached CPU utilization graphs of each of the nodes and as you can > see the node running ActiveMQ keeps consuming CPU time while all the other > nodes basically idle (the other nodes do more than just interact with > ActiveMQ). > > I've tried this with both journaledJDBC and kahaPersistenceAdapter (see > attached activemq.xml) to basically the same result. I was hoping that > the persistence adapter would handle the peak periods when the producers > are creating messages at a much higher rate than they can be consumed. > > So my questions are; > 1. Is there a limit to the number of messages ActiveMQ can queue even with > disk based persistence? > 2. Can someone suggest the configuration that would handle the above? > It's very important that the producers be allowed to operate as normal > (public facing) and if necessary we could have ActiveMQ ditch the messages > it can't deal with. > > Also, please note that when the consumers can keep up with the producers > everything works fine. > > http://www.nabble.com/file/p13740826/sar_cpu_mel1u105.sar.sadf.png > http://www.nabble.com/file/p13740826/sar_cpu_mel1u108.sar.sadf.png > http://www.nabble.com/file/p13740826/sar_cpu_mel1u109.sar.sadf.png > http://www.nabble.com/file/p13740826/sar_cpu_mel1u110.sar.sadf.png > http://www.nabble.com/file/p13740826/sar_cpu_mel1u111.sar.sadf.png > http://www.nabble.com/file/p13740826/sar_cpu_mel1u113.sar.sadf.png > > http://www.nabble.com/file/p13740826/activemq.xml activemq.xml > -- View this message in context: http://www.nabble.com/performance-issues-with-slow-consumers-tf4802510s2354.html#a13796032 Sent from the ActiveMQ - User mailing list archive at Nabble.com.
