Hi,
We were stressing activemq 5.4.2 with few persistent queues, few
non-persistent queues with TimeToLive (TTL) of 2 mins, and few non
persistent queues without any TTL. We came across 2 issues from which we
couldn’t recover to normal state without cleaning tmp-store and restarting
activemq.
Stress Details:
1. Usecase
a. Senders sending messages to one master queue
b. Consumers for master queue is integrated along with activemq thru camel
integration with vm protocol. Consumer is called as router, which routes to
other sub queues based on some routing information based on event type. By,
this same message could be dropped to more than one sub queues. We had 4 sub
queues, on average one message to master queue can result into 2.3 messages
in all sub queues because of routing same messages to sub queues.
c. Consumers to sub queues are in different JVM which drain the messages.
2. Stats
a. 2000-4000 msgs per second sent to master queue
b. 2000-3300 msgs per second of processing at router
c. 30% cpu in activemq
3. Hardware
a. ActiveMQ: one node 6 CPU machine with 24 virtual cpus because of core
and
hyper threading.
b. 4 Senders: Each node having 2 CPU with 4 virtual cpus because of dual
core. Each sender was to send 1000 msgs per second
c. 4 Receivers: Each node having 2 CPU with 4 virtual cpus because of dual
core. 1 receiver per sub queue.
Issue1: Temp store usage hits more than 100%
After temp store usage hits more than 100% all senders and consumers were
blocked. There were pending messages in both non-persistent queues with TTL
and without TTL. We tried purging in non-persistent queue, thinking purging
messages in non-persistent queue can free up temp store, but all purge
requests from activemq console and jmx were also blocked with no response.
Only way to recover was to clean the tmp store and restart the activemq. We
had tmp store configured to 2 GB.
Issue2: Leak in temp store
As Issue 1 was hit, and we couldn’t proceed with stress, so stopped the
stress when temp store reached 96% (timeframe-1), after all messages where
drained in 4-5 mins and there no pending messages in any queue, but temp
store didn’t drop to 0, it dropped only to till 79% (time-frame2). We
started stress again messages piled and temp store again went to 90, we
again stopped stress,
Below has chain of events with temp store stats, queue stats and temp store
files names and its size.
Please let me know any pointers on how to recover from this kind of issue
without cleaning tempstore and restart activemq. Is there anyway
configuration thru which we can avoid memory leak in temp store. Also let me
know if you need any other details on stress usecase.
Thanks
siva
TempStore stats
2011-04-21 07:18:08,760 - MEMU STOREU TEMPU ENQ# DEQ# TOT_MSG#
2011-04-21 07:18:08,779 - 0 0 0 6934 6480 454
Started one sender
2011-04-21 07:52:10,539 - 0 1 0 13370361 13370361 0
2011-04-21 07:54:10,737 - 0 0 11 14155760 14115898
39861 Started three
more sender
2011-04-21 07:56:10,829 - 0 1 26 15484214 15375457
108759
2011-04-21 07:58:10,899 - MEMU STOREU TEMPU ENQ# DEQ# TOT#
2011-04-21 07:58:10,902 - 0 1 41 16822317 16653849
168468
2011-04-21 08:00:10,999 - 0 2 59 18097270 17851082
246192
2011-04-21 08:02:11,111 - 0 2 75 19411640 19097534
314109
2011-04-21 08:04:11,188 - 0 0 96 20630853 20223204
407652 Stopped stress;
consumers are still draining 400k msgs
2011-04-21 08:06:11,244 - 0 1 87 21618914 21436362
182551
2011-04-21 08:08:11,301 - 0 0 79 22002217 22002217
0 Dropped to
79 in 6 mins; 0 pending msgs
2011-04-21 08:18:11,734 - 0 1 79 25445840 25445574
269 started only
one senders
2011-04-21 08:20:11,806 - 0 1 81 26328252 26311965
16285
2011-04-21 08:22:11,901 - 0 2 82 27674442 27597605
76846 started three
more senders
2011-04-21 08:24:11,974 - 0 0 85 29008172 28864086
144093
2011-04-21 08:26:12,034 - 0 1 88 30325245 30124132
201114
2011-04-21 08:28:12,097 - 0 1 91 31664800 31398125
266677
2011-04-21 08:30:12,173 - 0 2 93 32999706 32671816
327890
2011-04-21 08:32:12,232 - 0 2 95 34307528 33946200
361331 Stopped again
2011-04-21 08:34:12,296 - 0 0 79 35082703 35082702
1 Dropped to
79 within 2 mins
Master Queue stats
2011-04-21 07:18:08,867 - masterEventQueue PEND# ENQ# Chng_ENQ#
DEQ#
Chng__DEQ# EXP# MEM% DSPTCH# Stress start
2011-04-21 07:18:08,872 - masterEventQueue 0 2237 18
2237 18 0 0 2237
Started one sender
2011-04-21 07:52:10,564 - masterEventQueue 0 4313020 2185 4313020
2185 0 0
4313020
2011-04-21 07:54:10,767 - masterEventQueue 39769 4593381 2336 4553610
2004 0
0 4555338
2011-04-21 07:56:10,845 - masterEventQueue 108697 5068561 3959 4959863
3385
0 0 4961116 Started 3 more senders. Msgs enqued spiked from 2k to
4k,
after which it stayed constent at 4k;
2011-04-21 07:58:10,926 - masterEventQueue 168461 5540674 3934 5372213
3436
0 0 5373512
2011-04-21 08:00:11,028 - masterEventQueue 246185 6004699 3866 5758517
3219
0 0 5760118
2011-04-21 08:02:11,137 - masterEventQueue 314148 6474728 3916 6160580
3350
0 0 6162105
2011-04-21 08:04:11,205 - masterEventQueue 407616 6931316 3804 6523701
3026
0 0 6525303 Stopped stress; senders were sending 4k msg per
second but
router was able to process only till 3.3-3.4k msg per second.
2011-04-21 08:06:11,261 - masterEventQueue 182463 7097490 1384 6915027
3261
0 0 6920115
2011-04-21 08:08:11,368 - masterEventQueue 0 7097490 0 7097490
1520 0 0
7097490 Pending msg is 0; but Tmp store dropped only to 79%
2011-04-21 08:18:11,758 - masterEventQueue 16 8208393 2084 8208378
2084 0 0
8208397 started only one senders
2011-04-21 08:20:11,836 - masterEventQueue 15958 8503901 2462 8487948
2329 0
0 8490179
2011-04-21 08:22:11,929 - masterEventQueue 76455 8979141 3960 8902691
3456 0
0 8903985 started three more senders
2011-04-21 08:24:11,993 - masterEventQueue 143969 9455091 3966 9311128
3403
0 0 9312780
2011-04-21 08:26:12,053 - masterEventQueue 200591 9918299 3860 9717712
3388
0 0 9719987
2011-04-21 08:28:12,124 - masterEventQueue 266382 10395026 3972
10128647
3424 0 0 10130777
2011-04-21 08:30:12,192 - masterEventQueue 327391 10866887 3932
10539495
3423 0 0 10541609
2011-04-21 08:32:12,252 - masterEventQueue 360928 11311497 3705
10950569
3425 0 0 10952774 Stopped senders again
2011-04-21 08:34:12,320 - masterEventQueue 0 11317002 45
11317002 3053 0 0
11317002 Dropped again to zero
Content of temp store
ls -la tmp-data/
33 mb Apr 21 08:56 db-253.log
33 mb Apr 21 08:57 db-254.log
33 mb Apr 21 08:57 db-255.log
33 mb Apr 21 08:57 db-256.log
0 Apr 21 07:53 lock
1.67 gb Apr 21 08:57 tmpDB.data 1.7GB
3 mb Apr 21 08:57 tmpDB.redo
--
View this message in context:
http://activemq.2283324.n4.nabble.com/Senders-blocked-when-temp-store-100-and-memory-leak-in-temp-store-tp3467749p3467749.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
