Jörg, I finally did some analysis with yourkit today and indeed there is a memory leak, I captured the issue and resolution with reference to this thread in the issue: https://issues.apache.org/activemq/browse/AMQ-2143
I think the resolution of the leak should help with more stable enqueue rates. I will leave a steady state test running over night to validate. thanks for the analysis, Gary. 2009/2/21 Cybexion <cybex...@email.de>: > > Hi Gary, > > we just made a new test which startet yesterday. > JVM settings of AMQ where 3 GB heap size. > We send about 800 msg / sec into the queue, the size was limited to 3 GB. > During that time sending messages to the queue got slower and slower. Even > after the queue was full and new messages have been thrown away. Futhermore, > the memory consimption was increasing and increasing. I checked with > visualvm and the heap was totally used. A full GC did not change anything. > I tried to get a memdump with jmap but the jvm crashed :-( > My current state is, that at least in my case AMQ is not usable for two > reasons: > 1. Obiously a mem leak > 2. Sending messages to the queue takes more and more time the more messages > have been send already. > > Best regards > > Jörg > > > > > Gary Tully wrote: >> >> Hi Jörg, >> I did some analysis of the per message allocations from your email. It >> is clear that each store write allocates a bunch of objects, however, >> from code inspection, I cannot see a point where they are leaked >> through a reference. >> There are two relevant points that may make static analysis a little >> difficult. First is the fact that writes are asynchronous, so there >> will be a small delay until write allocations are free for GC. This >> should be very small in reality though. >> Second is that store pages remain in memory until they are pushed out >> of a LRU cache, a page has the possibility of referencing batch writes >> and indexes with location information, ie: most of the per write >> allocated objects. >> >> In short, we need to repeat your experiment with some more depth by >> taking caching issues into consideration. This is on my to do list. >> >> From a more high level view point, is it the case that a long running >> test, with say 2000m/s in and out of the broker will eventually run >> out of memory? >> >> I ask this because there are two sequential questions: 1) is there >> really a memory leak (long running test gets into trouble) 2) if there >> is a leak, where is it? >> >> Gary. >> >> 2009/2/13 Cybexion <cybex...@email.de>: >>> >>> Hello, >>> >>> I just tried something with the Netbeans memory profiler. >>> My assumption is: >>> If nothing hass been added to an application then after 2 full runs of >>> the >>> Garbace Collector the memory should be the same. >>> In the past we have seen that memory was increasing while messages have >>> been >>> processed by AMQ. For me that means that something stays in memory even >>> after those messages have been dequeued. >>> >>> My test looks like this: >>> 1. start a fresh AMQ instance, send some messages to a queue for >>> initialisation >>> 2. do a full GC >>> 3. Take a memory snapshot >>> 4. send exactly 1000 message to the queue >>> 5. consume all 1000 messages from the queue >>> 6. do a full GC >>> 7. Take a another memory snapshot >>> >>> Assumption: >>> I would expect that no references to messages do exist any more in the >>> AMQ >>> JVM because they have all been processed. The memory should look more or >>> less like it was before the test. >>> Now I did a memory comparision of my two snapshots. >>> The result looks like this: >>> http://www.nabble.com/file/p22003872/AMQClasses.jpg >>> From within the "org.apache.activemq" package there are exactly 1000 >>> MessageKeys, 1000 RemoveMessageCommands and 1000 AdMessageCommands left >>> over. >>> Shouldn't those have disappeared because the queue is empty??? >>> >>> Additionaly, if we look at the "org.apache.kaha" package: >>> http://www.nabble.com/file/p22003872/KahaClasses.jpg >>> There are 2000 DataByteArrayOutputstreams, 2000 Location objects and 2000 >>> WriteCommands still inside the memory. I would also have expected that >>> those >>> do not exist after the queue is empty. >>> >>> Well, personally this does not help me, but maybe one of the developers >>> can >>> convert this information into a possible bug inside the code? >>> >>> Help would really be appreciated. >>> >>> Thanks >>> >>> Jörg >>> >>> >>> >>> -- >>> View this message in context: >>> http://www.nabble.com/Memoryleak-when-consuming-messages-from-KahaDBStore-tp21991929p22003872.html >>> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >>> >>> >> >> >> >> -- >> http://blog.garytully.com >> >> Open Source SOA >> http://FUSESource.com >> >> > > -- > View this message in context: > http://www.nabble.com/Memoryleak-when-consuming-messages-from-KahaDBStore-tp21991929p22141262.html > Sent from the ActiveMQ - User mailing list archive at Nabble.com. > > -- http://blog.garytully.com Open Source SOA http://FUSESource.com
