> Can you please explain to me why I am getting the following suspected memory leaks in the Memory Analyzer tool (for a taken memory heap)?
It's important to note that the tool is only highlighting what it _suspects_ to be a leak. However, it has no real proof that a leak exists. You must investigate further to determine whether or not the suspect is an actual leak. > Is this considered normal? What is "normal" depends on the context. In this case I'd say the 1,006 instances of "org.apache.activemq.artemis.core.server.impl.QueueImpl" in your heap are expected. These represent the 1,002 queues you have configured in broker.xml as well as a few additional internal queues. Likewise, I'd say the 2,000 instances of "org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl" in your heap are also expected since you have 1,000 bridges configured and those bridges will use ChannelImpl for communicating with the remote broker. I'm not seeing a problem here at this point. Justin On Mon, Dec 16, 2024 at 9:57 AM Dragan J <draga...@flipside.team> wrote: > Hey @Justin Bertram <jbert...@apache.org> Undestood. Can you please > explain to me why I am getting the following suspected memory leaks in the > Memory Analyzer tool (for a taken memory heap)? > Is this considered normal? > > Problem Suspect 1 > > 1,006 instances of > *org.apache.activemq.artemis.core.server.impl.QueueImpl*, loaded by > *java.net.URLClassLoader > @ 0x82721c20* occupy *37,566,648 (33.76%)* bytes. > > Most of these instances are referenced from one instance of > *java.util.concurrent.ConcurrentHashMap$Node[]*, loaded by *<system class > loader>*, which occupies *40,496 (0.04%)* bytes. The instance is > referenced by *org.apache.activemq.artemis.utils.collections.ConcurrentHashSet > @ 0x81cf40b0*, loaded by *java.net.URLClassLoader @ 0x82721c20*. > > Thread > *org.apache.activemq.artemis.core.io.aio.AIOSequentialFileFactory$PollerThread > @ 0x81fed740 Apache ActiveMQ Artemis libaio poller* has a local variable > or reference to > *org.apache.activemq.artemis.core.io.aio.AIOSequentialFileFactory > @ 0x81cf68a8* which is on the shortest path to > *java.util.concurrent.ConcurrentHashMap$Node[2048] > @ 0x81cf4100*. The thread > *org.apache.activemq.artemis.core.io.aio.AIOSequentialFileFactory$PollerThread > @ 0x81fed740 Apache ActiveMQ Artemis libaio poller* keeps local variables > with total size *200 (0.00%)* bytes. > > The stacktrace of this Thread is available. See stacktrace. See > stacktrace with involved local variables. > > *Keywords* > > - org.apache.activemq.artemis.core.server.impl.QueueImpl > - java.net.URLClassLoader > - java.util.concurrent.ConcurrentHashMap$Node[] > - org.apache.activemq.artemis.utils.collections.ConcurrentHashSet > > > Leak Suspects > > 2,000 instances of > *org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl*, loaded > by *java.net.URLClassLoader @ 0x82721c20* occupy *16,823,208 (15.12%)* > bytes. > > Most of these instances are referenced from one instance of > *java.util.HashMap$Node[]*, loaded by *<system class loader>*, which > occupies *40,208 (0.04%)* bytes. The instance is referenced by > *org.apache.activemq.artemis.core.server.cluster.ClusterManager > @ 0x81ff0148*, loaded by *java.net.URLClassLoader @ 0x82721c20*. > > Thread > *org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$FailureCheckAndFlushThread > @ 0x81fee598 activemq-failure-check-thread* has a local variable or > reference to > *org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl > @ 0x82757548* which is on the shortest path to *java.util.HashMap$Node[2048] > @ 0x8206b9f8*. The thread > *org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$FailureCheckAndFlushThread > @ 0x81fee598 activemq-failure-check-thread* keeps local variables with > total size *448 (0.00%)* bytes. > > The stacktrace of this Thread is available. See stacktrace. See > stacktrace with involved local variables. > > *Keywords* > > - org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl > - java.net.URLClassLoader > - java.util.HashMap$Node[] > - org.apache.activemq.artemis.core.server.cluster.ClusterManager > > > On Fri, Dec 6, 2024 at 12:01 AM Clebert Suconic <clebert.suco...@gmail.com> > wrote: > >> I hoped you could give me some piece of code isolating the issue, in a >> way I could execute a script and see exactly what you're seeing. >> >> On Wed, Dec 4, 2024 at 11:00 AM Dragan J <draga...@flipside.team> wrote: >> > >> > I have already sent the output of the memory suspect analysis before. >> > 1,254 instances of >> > >> “org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl”, >> > loaded by “java.net.URLClassLoader @ 0x400000000” occupy 527,112,720 >> > (73.68%) bytes. >> > >> > I created the new 2 test servers with 1,000 CoreBridges between them, >> > killed the Java process for the Artemis server, and monitored the memory >> > heap on the client over time (through JConsole). A memory heap increase >> was >> > noticed again. >> > The archived servers (both the client and server) along with a >> screenshot >> > of the memory heap (G1 Old Gen) can be found at: >> > >> https://drive.google.com/file/d/1TBMUV31JbTvZwDF0P2teX0PSCIA_C1kn/view?usp=drive_link >> > >> https://drive.google.com/file/d/1iJ1O9PLlL_8XnIbMvE7mTBrz26mhce0I/view?usp=sharing >> > >> https://drive.google.com/file/d/1-eItVKsubvQXnbyNQI2Jj88hh1kJK0lM/view?usp=sharing >> > >> > Please note that the installation of Artemis 2.38 should be in the /opt >> > folder. >> > >> > Thanks, >> > Dragan >> > >> > >> > On Wed, Dec 4, 2024 at 7:02 AM Clebert Suconic < >> clebert.suco...@gmail.com> >> > wrote: >> > >> > > Do you have a test showing the issue ? >> > > >> > > >> > > check Leak has an interface to count instances. If you provide a way >> to >> > > reproduce the issue (a test with embedded server would be best) I >> will help >> > > out. >> > > >> > > Clebert Suconic >> > > >> > > >> > > On Tue, Dec 3, 2024 at 9:03 AM Dragan J <draga...@flipside.team> >> wrote: >> > > >> > > > Hi guys, >> > > > >> > > > Do you have any updates on this? >> > > > >> > > > Thanks, >> > > > Dragan >> > > > >> > > > On Tue, Nov 26, 2024 at 4:27 PM Dragan J <draga...@flipside.team> >> wrote: >> > > > >> > > > > @justin, please try rebooting the server with an Artemis broker >> (the >> > > > > source) that connects to another broker (the destination), which >> must >> > > run >> > > > > on a different server. >> > > > > The brokers exchange messages between them, but the volume is >> very low >> > > > > since this is a development environment. >> > > > > A memory leak is observed a few hours later on the destination >> server >> > > > > (running Artemis 2.38). >> > > > > >> > > > > @Clebert The provided check-leak code is not straightforward to >> > > > > implement. Could you please refer me to some instructions on how >> to do >> > > > it? >> > > > > >> > > > > >> > > > > >> > > > > On Fri, Nov 22, 2024 at 3:00 PM Dragan J <draga...@flipside.team> >> > > wrote: >> > > > > >> > > > >> @Clebert Unfortunately, the business requirement is to use >> > > CoreBridges. >> > > > >> @Justin We tested with 1000 CoreBridges, and a memory leak >> remains. >> > > > >> Here is "the problem suspect" from the Memory Analyzer (a heap >> dump >> > > > >> analyzed): >> > > > >> >> > > > >> >> > > > >> 1,254 instances of >> > > > >> >> > > > >> > > >> “org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl”, >> > > > >> loaded by “java.net.URLClassLoader @ 0x400000000” occupy >> 527,112,720 >> > > > >> (73.68%) bytes. >> > > > >> >> > > > >> Biggest instances: >> > > > >> >> > > > >> > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> > > > >> @ 0x40d81e128 - 17,669,864 (2.47%) bytes. >> > > > >> >> > > > >> > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> > > > >> @ 0x409926c00 - 17,643,464 (2.47%) bytes. >> > > > >> >> > > > >> > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> > > > >> @ 0x411b0b320 - 17,624,664 (2.46%) bytes. >> > > > >> >> > > > >> > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> > > > >> @ 0x401fdb948 - 17,614,232 (2.46%) bytes. >> > > > >> >> > > > >> > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> > > > >> @ 0x4099afd28 - 17,606,640 (2.46%) bytes. >> > > > >> >> > > > >> > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> > > > >> @ 0x409895e10 - 17,599,608 (2.46%) bytes. >> > > > >> >> > > > >> > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> > > > >> @ 0x40d9e35e0 - 17,597,288 (2.46%) bytes. >> > > > >> >> > > > >> > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> > > > >> @ 0x409b1a978 - 17,581,512 (2.46%) bytes. >> > > > >> >> > > > >> > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> > > > >> @ 0x409969ad8 - 17,579,656 (2.46%) bytes. >> > > > >> >> > > > >> > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> > > > >> @ 0x40d9e5088 - 17,559,432 (2.45%) bytes. >> > > > >> >> > > > >> > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> > > > >> @ 0x410fa8d80 - 17,552,744 (2.45%) bytes. >> > > > >> >> > > > >> > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> > > > >> @ 0x404db4d38 - 17,547,832 (2.45%) bytes. >> > > > >> >> > > > >> > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> > > > >> @ 0x4197b64f0 - 17,529,272 (2.45%) bytes. >> > > > >> >> > > > >> > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> > > > >> @ 0x409a03058 - 17,521,432 (2.45%) bytes. >> > > > >> >> > > > >> > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> > > > >> @ 0x411b55168 - 17,495,208 (2.45%) bytes. >> > > > >> >> > > > >> > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> > > > >> @ 0x40b0b3c30 - 17,481,480 (2.44%) bytes. >> > > > >> >> > > > >> > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> > > > >> @ 0x4084fe4b0 - 17,478,696 (2.44%) bytes. >> > > > >> >> > > > >> > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> > > > >> @ 0x4195a5dc8 - 17,476,840 (2.44%) bytes. >> > > > >> >> > > > >> > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> > > > >> @ 0x419725708 - 17,472,008 (2.44%) bytes. >> > > > >> >> > > > >> > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> > > > >> @ 0x409a441c8 - 17,453,176 (2.44%) bytes. >> > > > >> >> > > > >> > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> > > > >> @ 0x404d89060 - 17,434,424 (2.44%) bytes. >> > > > >> >> > > > >> > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> > > > >> @ 0x40f875d80 - 17,425,144 (2.44%) bytes. >> > > > >> >> > > > >> > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> > > > >> @ 0x4102071f8 - 17,408,104 (2.43%) bytes. >> > > > >> >> > > > >> > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> > > > >> @ 0x40d9e34f8 - 17,383,576 (2.43%) bytes. >> > > > >> >> > > > >> > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> > > > >> @ 0x409926318 - 17,374,568 (2.43%) bytes. >> > > > >> >> > > > >> > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> > > > >> @ 0x409a7cde8 - 17,318,888 (2.42%) bytes. >> > > > >> >> > > > >> > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> > > > >> @ 0x40aadd150 - 17,285,208 (2.42%) bytes. >> > > > >> >> > > > >> > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> > > > >> @ 0x40d81ba08 - 17,273,880 (2.41%) bytes. >> > > > >> >> > > > >> > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> > > > >> @ 0x40070d068 - 17,223,032 (2.41%) bytes. >> > > > >> >> > > > >> > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> > > > >> @ 0x41bea4538 - 17,146,936 (2.40%) bytes. >> > > > >> >> > > > >> Most of these instances are referenced from one instance of >> > > > >> >> > > > >> > > >> “org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl”, >> > > > >> loaded by “java.net.URLClassLoader @ 0x400000000”, which occupies >> > > 7,008 >> > > > >> (0.00%) bytes. >> > > > >> >> > > > >> Thread >> > > > >> >> > > > >> > > >> “org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$FailureCheckAndFlushThread >> > > > >> @ 0x405ffd540 activemq-failure-check-thread” has a local >> variable or >> > > > >> reference to >> > > > >> >> > > > >> > > >> “org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl >> > > > >> @ 0x401a42658” which is on the shortest path to >> > > > >> >> > > > >> > > >> “org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl >> > > > >> @ 0x401a42658”. The thread >> > > > >> >> > > > >> > > >> org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$FailureCheckAndFlushThread >> > > > >> @ 0x405ffd540 activemq-failure-check-thread keeps local >> variables with >> > > > >> total size 328 (0.00%) bytes. >> > > > >> >> > > > >> The stacktrace of this Thread is available. See stacktrace. See >> > > > >> stacktrace with involved local variables. >> > > > >> >> > > > >> Keywords >> > > > >> >> > > > >> > > >> org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> > > > >> java.net.URLClassLoader >> > > > >> >> > > > >> org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl >> > > > >> >> > > > >> Thanks, >> > > > >> Dragan >> > > > >> >> > > > >> >> > > > >> On Tue, Nov 19, 2024 at 12:13 PM Dragan J <draga...@flipside.team >> > >> > > > wrote: >> > > > >> >> > > > >>> Hey Justin, >> > > > >>> >> > > > >>> Here are the answers: >> > > > >>> - Are all the bridges configured the same way? How exactly are >> they >> > > > >>> configured? >> > > > >>> Yes, they are configured the same way. >> > > > >>> The CoreBridge configuration is attached to >> > > > >>> >> > > > >> > > >> https://drive.google.com/file/d/1VEwj1MoB2tbcCp8R4XeQTzi80Q0dQ606/view?usp=sharing >> > > > >>> >> > > > >>> - Is the bridge target available when you see this increase >> happen? >> > > > >>> The bridges are connected from the client side to the server >> (with >> > > the >> > > > >>> 2.38 Artemis version) >> > > > >>> - Are the bridges actually moving messages when you see the >> increase >> > > or >> > > > >>> are they idle? >> > > > >>> Most of the time, messages are idle. >> > > > >>> - Are 3,500 bridges actually required to trigger the increase? >> Do >> > > you, >> > > > >>> for >> > > > >>> example, not see the increase if you use 3,000 bridges? What >> about >> > > > 1,000 >> > > > >>> or >> > > > >>> 100? >> > > > >>> We tested with 3500 CoreBridges in the development environment, >> and I >> > > > >>> should ask for permission to close some connections. >> > > > >>> If you require it, I would ask for that kind of permission. >> > > > >>> >> > > > >>> Thanks, >> > > > >>> Dragan >> > > > >>> >> > > > >>> On Mon, Nov 18, 2024 at 2:43 PM Dragan J <draga...@flipside.team >> > >> > > > wrote: >> > > > >>> >> > > > >>>> Can you access the screenshots? >> > > > >>>> >> > > > >>>> On Thu, Nov 14, 2024 at 10:18 AM Dragan J >> <draga...@flipside.team> >> > > > >>>> wrote: >> > > > >>>> >> > > > >>>>> Sorry, a typo. It's 3,500 CoreBridges >> > > > >>>>> >> > > > >>>>> On Thu, Nov 14, 2024 at 10:14 AM Dragan J >> <draga...@flipside.team> >> > > > >>>>> wrote: >> > > > >>>>> >> > > > >>>>>> @Justin The broker (version 2.38) communicates with other >> brokers >> > > > >>>>>> (version 2.37) through 35,000 CoreBridges. There was minimal >> > > message >> > > > >>>>>> exchange between them. >> > > > >>>>>> Please find the two graphs (from earlier) attached on my >> Google >> > > > >>>>>> Drive. Please let me know if you have an alternative way to >> share >> > > > files. >> > > > >>>>>> >> > > > >>>>>> >> > > > >> > > >> https://drive.google.com/file/d/1Gy8XQi153Fi5Zm9gUdsP_co8fSuh1HjL/view?usp=sharing >> > > > >>>>>> >> > > > >>>>>> >> > > > >> > > >> https://drive.google.com/file/d/1_Fbg8OmWhlsOKfalfIEwKxGIl97a1kQ5/view?usp=sharing >> > > > >>>>>> >> > > > >>>>>> @Arthur When I tried to perform a GC from JConsole I got the >> > > > >>>>>> following: >> > > > >>>>>> >> > > > >>>>>> Exception in thread "MemoryPanel.gc" >> java.lang.SecurityException: >> > > > >>>>>> User not authorized to access operation: gc >> > > > >>>>>> >> > > > >>>>> >> > > > >> > > >> >> >> >> -- >> Clebert Suconic >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@activemq.apache.org >> For additional commands, e-mail: dev-h...@activemq.apache.org >> For further information, visit: https://activemq.apache.org/contact >> >> >>
