You mentioned before that you have a certain number of objects in a Heap that were supposed to be gone.
I usually write a test, where I expect the objects released, and then I will assert if the number of objects in the memory are as expected. from what you described now these seems regular circumstances. Unless you are sure these should have been released and still showing in the heap. On Mon, Dec 16, 2024 at 11:56 AM Justin Bertram <jbert...@apache.org> wrote: > > > Can you please explain to me why I am getting the following suspected > memory leaks in the Memory Analyzer tool (for a taken memory heap)? > > It's important to note that the tool is only highlighting what it > _suspects_ to be a leak. However, it has no real proof that a leak exists. > You must investigate further to determine whether or not the suspect is an > actual leak. > > > Is this considered normal? > > What is "normal" depends on the context. In this case I'd say the 1,006 > instances of "org.apache.activemq.artemis.core.server.impl.QueueImpl" in > your heap are expected. These represent the 1,002 queues you have > configured in broker.xml as well as a few additional internal queues. > Likewise, I'd say the 2,000 instances of > "org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl" in your > heap are also expected since you have 1,000 bridges configured and those > bridges will use ChannelImpl for communicating with the remote broker. > > I'm not seeing a problem here at this point. > > > Justin > > On Mon, Dec 16, 2024 at 9:57 AM Dragan J <draga...@flipside.team> wrote: > > > Hey @Justin Bertram <jbert...@apache.org> Undestood. Can you please > > explain to me why I am getting the following suspected memory leaks in the > > Memory Analyzer tool (for a taken memory heap)? > > Is this considered normal? > > > > Problem Suspect 1 > > > > 1,006 instances of > > *org.apache.activemq.artemis.core.server.impl.QueueImpl*, loaded by > > *java.net.URLClassLoader > > @ 0x82721c20* occupy *37,566,648 (33.76%)* bytes. > > > > Most of these instances are referenced from one instance of > > *java.util.concurrent.ConcurrentHashMap$Node[]*, loaded by *<system class > > loader>*, which occupies *40,496 (0.04%)* bytes. The instance is > > referenced by > > *org.apache.activemq.artemis.utils.collections.ConcurrentHashSet > > @ 0x81cf40b0*, loaded by *java.net.URLClassLoader @ 0x82721c20*. > > > > Thread > > *org.apache.activemq.artemis.core.io.aio.AIOSequentialFileFactory$PollerThread > > @ 0x81fed740 Apache ActiveMQ Artemis libaio poller* has a local variable > > or reference to > > *org.apache.activemq.artemis.core.io.aio.AIOSequentialFileFactory > > @ 0x81cf68a8* which is on the shortest path to > > *java.util.concurrent.ConcurrentHashMap$Node[2048] > > @ 0x81cf4100*. The thread > > *org.apache.activemq.artemis.core.io.aio.AIOSequentialFileFactory$PollerThread > > @ 0x81fed740 Apache ActiveMQ Artemis libaio poller* keeps local variables > > with total size *200 (0.00%)* bytes. > > > > The stacktrace of this Thread is available. See stacktrace. See > > stacktrace with involved local variables. > > > > *Keywords* > > > > - org.apache.activemq.artemis.core.server.impl.QueueImpl > > - java.net.URLClassLoader > > - java.util.concurrent.ConcurrentHashMap$Node[] > > - org.apache.activemq.artemis.utils.collections.ConcurrentHashSet > > > > > > Leak Suspects > > > > 2,000 instances of > > *org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl*, loaded > > by *java.net.URLClassLoader @ 0x82721c20* occupy *16,823,208 (15.12%)* > > bytes. > > > > Most of these instances are referenced from one instance of > > *java.util.HashMap$Node[]*, loaded by *<system class loader>*, which > > occupies *40,208 (0.04%)* bytes. The instance is referenced by > > *org.apache.activemq.artemis.core.server.cluster.ClusterManager > > @ 0x81ff0148*, loaded by *java.net.URLClassLoader @ 0x82721c20*. > > > > Thread > > *org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$FailureCheckAndFlushThread > > @ 0x81fee598 activemq-failure-check-thread* has a local variable or > > reference to > > *org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl > > @ 0x82757548* which is on the shortest path to *java.util.HashMap$Node[2048] > > @ 0x8206b9f8*. The thread > > *org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$FailureCheckAndFlushThread > > @ 0x81fee598 activemq-failure-check-thread* keeps local variables with > > total size *448 (0.00%)* bytes. > > > > The stacktrace of this Thread is available. See stacktrace. See > > stacktrace with involved local variables. > > > > *Keywords* > > > > - org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl > > - java.net.URLClassLoader > > - java.util.HashMap$Node[] > > - org.apache.activemq.artemis.core.server.cluster.ClusterManager > > > > > > On Fri, Dec 6, 2024 at 12:01 AM Clebert Suconic <clebert.suco...@gmail.com> > > wrote: > > > >> I hoped you could give me some piece of code isolating the issue, in a > >> way I could execute a script and see exactly what you're seeing. > >> > >> On Wed, Dec 4, 2024 at 11:00 AM Dragan J <draga...@flipside.team> wrote: > >> > > >> > I have already sent the output of the memory suspect analysis before. > >> > 1,254 instances of > >> > > >> “org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl”, > >> > loaded by “java.net.URLClassLoader @ 0x400000000” occupy 527,112,720 > >> > (73.68%) bytes. > >> > > >> > I created the new 2 test servers with 1,000 CoreBridges between them, > >> > killed the Java process for the Artemis server, and monitored the memory > >> > heap on the client over time (through JConsole). A memory heap increase > >> was > >> > noticed again. > >> > The archived servers (both the client and server) along with a > >> screenshot > >> > of the memory heap (G1 Old Gen) can be found at: > >> > > >> https://drive.google.com/file/d/1TBMUV31JbTvZwDF0P2teX0PSCIA_C1kn/view?usp=drive_link > >> > > >> https://drive.google.com/file/d/1iJ1O9PLlL_8XnIbMvE7mTBrz26mhce0I/view?usp=sharing > >> > > >> https://drive.google.com/file/d/1-eItVKsubvQXnbyNQI2Jj88hh1kJK0lM/view?usp=sharing > >> > > >> > Please note that the installation of Artemis 2.38 should be in the /opt > >> > folder. > >> > > >> > Thanks, > >> > Dragan > >> > > >> > > >> > On Wed, Dec 4, 2024 at 7:02 AM Clebert Suconic < > >> clebert.suco...@gmail.com> > >> > wrote: > >> > > >> > > Do you have a test showing the issue ? > >> > > > >> > > > >> > > check Leak has an interface to count instances. If you provide a way > >> to > >> > > reproduce the issue (a test with embedded server would be best) I > >> will help > >> > > out. > >> > > > >> > > Clebert Suconic > >> > > > >> > > > >> > > On Tue, Dec 3, 2024 at 9:03 AM Dragan J <draga...@flipside.team> > >> wrote: > >> > > > >> > > > Hi guys, > >> > > > > >> > > > Do you have any updates on this? > >> > > > > >> > > > Thanks, > >> > > > Dragan > >> > > > > >> > > > On Tue, Nov 26, 2024 at 4:27 PM Dragan J <draga...@flipside.team> > >> wrote: > >> > > > > >> > > > > @justin, please try rebooting the server with an Artemis broker > >> (the > >> > > > > source) that connects to another broker (the destination), which > >> must > >> > > run > >> > > > > on a different server. > >> > > > > The brokers exchange messages between them, but the volume is > >> very low > >> > > > > since this is a development environment. > >> > > > > A memory leak is observed a few hours later on the destination > >> server > >> > > > > (running Artemis 2.38). > >> > > > > > >> > > > > @Clebert The provided check-leak code is not straightforward to > >> > > > > implement. Could you please refer me to some instructions on how > >> to do > >> > > > it? > >> > > > > > >> > > > > > >> > > > > > >> > > > > On Fri, Nov 22, 2024 at 3:00 PM Dragan J <draga...@flipside.team> > >> > > wrote: > >> > > > > > >> > > > >> @Clebert Unfortunately, the business requirement is to use > >> > > CoreBridges. > >> > > > >> @Justin We tested with 1000 CoreBridges, and a memory leak > >> remains. > >> > > > >> Here is "the problem suspect" from the Memory Analyzer (a heap > >> dump > >> > > > >> analyzed): > >> > > > >> > >> > > > >> > >> > > > >> 1,254 instances of > >> > > > >> > >> > > > > >> > > > >> “org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl”, > >> > > > >> loaded by “java.net.URLClassLoader @ 0x400000000” occupy > >> 527,112,720 > >> > > > >> (73.68%) bytes. > >> > > > >> > >> > > > >> Biggest instances: > >> > > > >> > >> > > > > >> > > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> > > > >> @ 0x40d81e128 - 17,669,864 (2.47%) bytes. > >> > > > >> > >> > > > > >> > > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> > > > >> @ 0x409926c00 - 17,643,464 (2.47%) bytes. > >> > > > >> > >> > > > > >> > > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> > > > >> @ 0x411b0b320 - 17,624,664 (2.46%) bytes. > >> > > > >> > >> > > > > >> > > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> > > > >> @ 0x401fdb948 - 17,614,232 (2.46%) bytes. > >> > > > >> > >> > > > > >> > > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> > > > >> @ 0x4099afd28 - 17,606,640 (2.46%) bytes. > >> > > > >> > >> > > > > >> > > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> > > > >> @ 0x409895e10 - 17,599,608 (2.46%) bytes. > >> > > > >> > >> > > > > >> > > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> > > > >> @ 0x40d9e35e0 - 17,597,288 (2.46%) bytes. > >> > > > >> > >> > > > > >> > > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> > > > >> @ 0x409b1a978 - 17,581,512 (2.46%) bytes. > >> > > > >> > >> > > > > >> > > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> > > > >> @ 0x409969ad8 - 17,579,656 (2.46%) bytes. > >> > > > >> > >> > > > > >> > > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> > > > >> @ 0x40d9e5088 - 17,559,432 (2.45%) bytes. > >> > > > >> > >> > > > > >> > > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> > > > >> @ 0x410fa8d80 - 17,552,744 (2.45%) bytes. > >> > > > >> > >> > > > > >> > > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> > > > >> @ 0x404db4d38 - 17,547,832 (2.45%) bytes. > >> > > > >> > >> > > > > >> > > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> > > > >> @ 0x4197b64f0 - 17,529,272 (2.45%) bytes. > >> > > > >> > >> > > > > >> > > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> > > > >> @ 0x409a03058 - 17,521,432 (2.45%) bytes. > >> > > > >> > >> > > > > >> > > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> > > > >> @ 0x411b55168 - 17,495,208 (2.45%) bytes. > >> > > > >> > >> > > > > >> > > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> > > > >> @ 0x40b0b3c30 - 17,481,480 (2.44%) bytes. > >> > > > >> > >> > > > > >> > > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> > > > >> @ 0x4084fe4b0 - 17,478,696 (2.44%) bytes. > >> > > > >> > >> > > > > >> > > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> > > > >> @ 0x4195a5dc8 - 17,476,840 (2.44%) bytes. > >> > > > >> > >> > > > > >> > > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> > > > >> @ 0x419725708 - 17,472,008 (2.44%) bytes. > >> > > > >> > >> > > > > >> > > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> > > > >> @ 0x409a441c8 - 17,453,176 (2.44%) bytes. > >> > > > >> > >> > > > > >> > > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> > > > >> @ 0x404d89060 - 17,434,424 (2.44%) bytes. > >> > > > >> > >> > > > > >> > > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> > > > >> @ 0x40f875d80 - 17,425,144 (2.44%) bytes. > >> > > > >> > >> > > > > >> > > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> > > > >> @ 0x4102071f8 - 17,408,104 (2.43%) bytes. > >> > > > >> > >> > > > > >> > > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> > > > >> @ 0x40d9e34f8 - 17,383,576 (2.43%) bytes. > >> > > > >> > >> > > > > >> > > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> > > > >> @ 0x409926318 - 17,374,568 (2.43%) bytes. > >> > > > >> > >> > > > > >> > > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> > > > >> @ 0x409a7cde8 - 17,318,888 (2.42%) bytes. > >> > > > >> > >> > > > > >> > > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> > > > >> @ 0x40aadd150 - 17,285,208 (2.42%) bytes. > >> > > > >> > >> > > > > >> > > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> > > > >> @ 0x40d81ba08 - 17,273,880 (2.41%) bytes. > >> > > > >> > >> > > > > >> > > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> > > > >> @ 0x40070d068 - 17,223,032 (2.41%) bytes. > >> > > > >> > >> > > > > >> > > > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> > > > >> @ 0x41bea4538 - 17,146,936 (2.40%) bytes. > >> > > > >> > >> > > > >> Most of these instances are referenced from one instance of > >> > > > >> > >> > > > > >> > > > >> “org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl”, > >> > > > >> loaded by “java.net.URLClassLoader @ 0x400000000”, which occupies > >> > > 7,008 > >> > > > >> (0.00%) bytes. > >> > > > >> > >> > > > >> Thread > >> > > > >> > >> > > > > >> > > > >> “org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$FailureCheckAndFlushThread > >> > > > >> @ 0x405ffd540 activemq-failure-check-thread” has a local > >> variable or > >> > > > >> reference to > >> > > > >> > >> > > > > >> > > > >> “org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl > >> > > > >> @ 0x401a42658” which is on the shortest path to > >> > > > >> > >> > > > > >> > > > >> “org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl > >> > > > >> @ 0x401a42658”. The thread > >> > > > >> > >> > > > > >> > > > >> org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$FailureCheckAndFlushThread > >> > > > >> @ 0x405ffd540 activemq-failure-check-thread keeps local > >> variables with > >> > > > >> total size 328 (0.00%) bytes. > >> > > > >> > >> > > > >> The stacktrace of this Thread is available. See stacktrace. See > >> > > > >> stacktrace with involved local variables. > >> > > > >> > >> > > > >> Keywords > >> > > > >> > >> > > > > >> > > > >> org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> > > > >> java.net.URLClassLoader > >> > > > >> > >> > > > > >> org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl > >> > > > >> > >> > > > >> Thanks, > >> > > > >> Dragan > >> > > > >> > >> > > > >> > >> > > > >> On Tue, Nov 19, 2024 at 12:13 PM Dragan J <draga...@flipside.team > >> > > >> > > > wrote: > >> > > > >> > >> > > > >>> Hey Justin, > >> > > > >>> > >> > > > >>> Here are the answers: > >> > > > >>> - Are all the bridges configured the same way? How exactly are > >> they > >> > > > >>> configured? > >> > > > >>> Yes, they are configured the same way. > >> > > > >>> The CoreBridge configuration is attached to > >> > > > >>> > >> > > > > >> > > > >> https://drive.google.com/file/d/1VEwj1MoB2tbcCp8R4XeQTzi80Q0dQ606/view?usp=sharing > >> > > > >>> > >> > > > >>> - Is the bridge target available when you see this increase > >> happen? > >> > > > >>> The bridges are connected from the client side to the server > >> (with > >> > > the > >> > > > >>> 2.38 Artemis version) > >> > > > >>> - Are the bridges actually moving messages when you see the > >> increase > >> > > or > >> > > > >>> are they idle? > >> > > > >>> Most of the time, messages are idle. > >> > > > >>> - Are 3,500 bridges actually required to trigger the increase? > >> Do > >> > > you, > >> > > > >>> for > >> > > > >>> example, not see the increase if you use 3,000 bridges? What > >> about > >> > > > 1,000 > >> > > > >>> or > >> > > > >>> 100? > >> > > > >>> We tested with 3500 CoreBridges in the development environment, > >> and I > >> > > > >>> should ask for permission to close some connections. > >> > > > >>> If you require it, I would ask for that kind of permission. > >> > > > >>> > >> > > > >>> Thanks, > >> > > > >>> Dragan > >> > > > >>> > >> > > > >>> On Mon, Nov 18, 2024 at 2:43 PM Dragan J <draga...@flipside.team > >> > > >> > > > wrote: > >> > > > >>> > >> > > > >>>> Can you access the screenshots? > >> > > > >>>> > >> > > > >>>> On Thu, Nov 14, 2024 at 10:18 AM Dragan J > >> <draga...@flipside.team> > >> > > > >>>> wrote: > >> > > > >>>> > >> > > > >>>>> Sorry, a typo. It's 3,500 CoreBridges > >> > > > >>>>> > >> > > > >>>>> On Thu, Nov 14, 2024 at 10:14 AM Dragan J > >> <draga...@flipside.team> > >> > > > >>>>> wrote: > >> > > > >>>>> > >> > > > >>>>>> @Justin The broker (version 2.38) communicates with other > >> brokers > >> > > > >>>>>> (version 2.37) through 35,000 CoreBridges. There was minimal > >> > > message > >> > > > >>>>>> exchange between them. > >> > > > >>>>>> Please find the two graphs (from earlier) attached on my > >> Google > >> > > > >>>>>> Drive. Please let me know if you have an alternative way to > >> share > >> > > > files. > >> > > > >>>>>> > >> > > > >>>>>> > >> > > > > >> > > > >> https://drive.google.com/file/d/1Gy8XQi153Fi5Zm9gUdsP_co8fSuh1HjL/view?usp=sharing > >> > > > >>>>>> > >> > > > >>>>>> > >> > > > > >> > > > >> https://drive.google.com/file/d/1_Fbg8OmWhlsOKfalfIEwKxGIl97a1kQ5/view?usp=sharing > >> > > > >>>>>> > >> > > > >>>>>> @Arthur When I tried to perform a GC from JConsole I got the > >> > > > >>>>>> following: > >> > > > >>>>>> > >> > > > >>>>>> Exception in thread "MemoryPanel.gc" > >> java.lang.SecurityException: > >> > > > >>>>>> User not authorized to access operation: gc > >> > > > >>>>>> > >> > > > >>>>> > >> > > > > >> > > > >> > >> > >> > >> -- > >> Clebert Suconic > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: dev-unsubscr...@activemq.apache.org > >> For additional commands, e-mail: dev-h...@activemq.apache.org > >> For further information, visit: https://activemq.apache.org/contact > >> > >> > >> -- Clebert Suconic --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@activemq.apache.org For additional commands, e-mail: dev-h...@activemq.apache.org For further information, visit: https://activemq.apache.org/contact
