I'm experiencing an issue with ActiveMQ 5.3.1 broker. Here's my setup:
- 1 queue that experiences heavy usage (the only relevant destination in
this case)
- 80 concurrentConsumers on the queue
- consumer.prefetchSize not set (default = 1000, I believe)
Occasionally my service sees a spike in traffic, and the queue may
experience a backlog of tens of thousands of messages (which is obviously
undesirable, but it's ok). As traffic subsides, the consumers are able to
catch up with the backlog and all is well.
BUT...here's the problem...
When the # of messages (N) in the queue rises, there comes a point where the
broker CPU spikes to 100%.
if N >= (concurrentConsumers * prefetchSize) {
cpu spikes;
}
In my case, 80,000 messages is the magic number where the CPU sikes. And
the math adds up: 80 concurrent consumers * 1000 prefetch size. You can
literally watch the CPU spike and stay spiked if N >= 80,000. And when N
drops back down < 80,000, you can see the CPU come back down to nominal
values. Predictable & reliable. But I'm never happy seeing 100% CPU or
high load on my servers! :-)
1. What is ActiveMQ doing to cause the CPU spike?
2. Is this expected or is it a bug?
3. I'm obviously tempted to crank consumer.prefetchSize way up to avoid
this, but what other problem(s) might I cause as a result?
4. I realize this is an amorphous question, but here goes anyway...which is
worse: cranking up prefetchSize, or adding more consumers?
Attached are thread dumps both in normal conditions (CPU nominal) and in the
CPU spiked condition.
http://old.nabble.com/file/p28133368/thread-dump-normal.log
thread-dump-normal.log
http://old.nabble.com/file/p28133368/thread-dump-cpu-spiked.log
thread-dump-cpu-spiked.log
Please let me know if you need any more detail or config.
Thanks!
Dan
--
View this message in context:
http://old.nabble.com/100--CPU-spike-with-ActiveMQ-5.3.1-tp28133368p28133368.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
