Hi Gary, we just made a new test which startet yesterday. JVM settings of AMQ where 3 GB heap size. We send about 800 msg / sec into the queue, the size was limited to 3 GB. During that time sending messages to the queue got slower and slower. Even after the queue was full and new messages have been thrown away. Futhermore, the memory consimption was increasing and increasing. I checked with visualvm and the heap was totally used. A full GC did not change anything. I tried to get a memdump with jmap but the jvm crashed :-( My current state is, that at least in my case AMQ is not usable for two reasons: 1. Obiously a mem leak 2. Sending messages to the queue takes more and more time the more messages have been send already.
Best regards Jörg Gary Tully wrote: > > Hi Jörg, > I did some analysis of the per message allocations from your email. It > is clear that each store write allocates a bunch of objects, however, > from code inspection, I cannot see a point where they are leaked > through a reference. > There are two relevant points that may make static analysis a little > difficult. First is the fact that writes are asynchronous, so there > will be a small delay until write allocations are free for GC. This > should be very small in reality though. > Second is that store pages remain in memory until they are pushed out > of a LRU cache, a page has the possibility of referencing batch writes > and indexes with location information, ie: most of the per write > allocated objects. > > In short, we need to repeat your experiment with some more depth by > taking caching issues into consideration. This is on my to do list. > > From a more high level view point, is it the case that a long running > test, with say 2000m/s in and out of the broker will eventually run > out of memory? > > I ask this because there are two sequential questions: 1) is there > really a memory leak (long running test gets into trouble) 2) if there > is a leak, where is it? > > Gary. > > 2009/2/13 Cybexion <cybex...@email.de>: >> >> Hello, >> >> I just tried something with the Netbeans memory profiler. >> My assumption is: >> If nothing hass been added to an application then after 2 full runs of >> the >> Garbace Collector the memory should be the same. >> In the past we have seen that memory was increasing while messages have >> been >> processed by AMQ. For me that means that something stays in memory even >> after those messages have been dequeued. >> >> My test looks like this: >> 1. start a fresh AMQ instance, send some messages to a queue for >> initialisation >> 2. do a full GC >> 3. Take a memory snapshot >> 4. send exactly 1000 message to the queue >> 5. consume all 1000 messages from the queue >> 6. do a full GC >> 7. Take a another memory snapshot >> >> Assumption: >> I would expect that no references to messages do exist any more in the >> AMQ >> JVM because they have all been processed. The memory should look more or >> less like it was before the test. >> Now I did a memory comparision of my two snapshots. >> The result looks like this: >> http://www.nabble.com/file/p22003872/AMQClasses.jpg >> From within the "org.apache.activemq" package there are exactly 1000 >> MessageKeys, 1000 RemoveMessageCommands and 1000 AdMessageCommands left >> over. >> Shouldn't those have disappeared because the queue is empty??? >> >> Additionaly, if we look at the "org.apache.kaha" package: >> http://www.nabble.com/file/p22003872/KahaClasses.jpg >> There are 2000 DataByteArrayOutputstreams, 2000 Location objects and 2000 >> WriteCommands still inside the memory. I would also have expected that >> those >> do not exist after the queue is empty. >> >> Well, personally this does not help me, but maybe one of the developers >> can >> convert this information into a possible bug inside the code? >> >> Help would really be appreciated. >> >> Thanks >> >> Jörg >> >> >> >> -- >> View this message in context: >> http://www.nabble.com/Memoryleak-when-consuming-messages-from-KahaDBStore-tp21991929p22003872.html >> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >> >> > > > > -- > http://blog.garytully.com > > Open Source SOA > http://FUSESource.com > > -- View this message in context: http://www.nabble.com/Memoryleak-when-consuming-messages-from-KahaDBStore-tp21991929p22141262.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
