I revisited this to check once again that there are no issues. At this point I can confirm that there are no issues, and I believe I understand the source of your concern, Dragan.
I believe this boils down to the fact that you're looking specifically at (and perhaps misunderstanding) the "G1 Old Gen" memory pool. I restarted the proof-of-concept I created previously and performed the same test you outlined (i.e. start a broker with 1,000 bridges, start a broker to receive those 1,000 bridge connections, stop the broker receiving the connections, & monitor memory usage on the broker with 1,000 bridges). I let this run for over an hour and tracked heap usage via JConsole. Like you I saw an increase in the usage of "G1 Old Gen" over this hour. However, this is expected because the "Old Gen" pool is used to store objects that have survived multiple garbage collection cycles in the Young Generation. These objects are considered long-lived and are less likely to be collected frequently so they get their own pool. This is just how the G1 garbage collector manages memory. If you look at the "G1 Eden Space" I imagine you'll see what I see which is an oscillation of heap utilization that reliably stays under a certain limit. In my case it was around 300MB. Again, this is expected. Attempting to reconnect 1,000 bridges over and over creates a lot of garbage which then gets collected. Ultimately memory use is stable and there is no evidence of any leak even with 1,000 bridges attempting to reconnect every 5 seconds. Let me know if you have any further questions. Justin On Wed, Dec 4, 2024 at 10:10 AM Dragan J <draga...@flipside.team> wrote: > I have already sent the output of the memory suspect analysis before. > 1,254 instances of > > “org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl”, > loaded by “java.net.URLClassLoader @ 0x400000000” occupy 527,112,720 > (73.68%) bytes. > > I created the new 2 test servers with 1,000 CoreBridges between them, > killed the Java process for the Artemis server, and monitored the memory > heap on the client over time (through JConsole). A memory heap increase was > noticed again. > The archived servers (both the client and server) along with a screenshot > of the memory heap (G1 Old Gen) can be found at: > > https://drive.google.com/file/d/1TBMUV31JbTvZwDF0P2teX0PSCIA_C1kn/view?usp=drive_link > > https://drive.google.com/file/d/1iJ1O9PLlL_8XnIbMvE7mTBrz26mhce0I/view?usp=sharing > > https://drive.google.com/file/d/1-eItVKsubvQXnbyNQI2Jj88hh1kJK0lM/view?usp=sharing > > Please note that the installation of Artemis 2.38 should be in the /opt > folder. > > Thanks, > Dragan > > > On Wed, Dec 4, 2024 at 7:02 AM Clebert Suconic <clebert.suco...@gmail.com> > wrote: > > > Do you have a test showing the issue ? > > > > > > check Leak has an interface to count instances. If you provide a way to > > reproduce the issue (a test with embedded server would be best) I will > help > > out. > > > > Clebert Suconic > > > > > > On Tue, Dec 3, 2024 at 9:03 AM Dragan J <draga...@flipside.team> wrote: > > > > > Hi guys, > > > > > > Do you have any updates on this? > > > > > > Thanks, > > > Dragan > > > > > > On Tue, Nov 26, 2024 at 4:27 PM Dragan J <draga...@flipside.team> > wrote: > > > > > > > @justin, please try rebooting the server with an Artemis broker (the > > > > source) that connects to another broker (the destination), which must > > run > > > > on a different server. > > > > The brokers exchange messages between them, but the volume is very > low > > > > since this is a development environment. > > > > A memory leak is observed a few hours later on the destination server > > > > (running Artemis 2.38). > > > > > > > > @Clebert The provided check-leak code is not straightforward to > > > > implement. Could you please refer me to some instructions on how to > do > > > it? > > > > > > > > > > > > > > > > On Fri, Nov 22, 2024 at 3:00 PM Dragan J <draga...@flipside.team> > > wrote: > > > > > > > >> @Clebert Unfortunately, the business requirement is to use > > CoreBridges. > > > >> @Justin We tested with 1000 CoreBridges, and a memory leak remains. > > > >> Here is "the problem suspect" from the Memory Analyzer (a heap dump > > > >> analyzed): > > > >> > > > >> > > > >> 1,254 instances of > > > >> > > > > > > “org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl”, > > > >> loaded by “java.net.URLClassLoader @ 0x400000000” occupy 527,112,720 > > > >> (73.68%) bytes. > > > >> > > > >> Biggest instances: > > > >> > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > >> @ 0x40d81e128 - 17,669,864 (2.47%) bytes. > > > >> > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > >> @ 0x409926c00 - 17,643,464 (2.47%) bytes. > > > >> > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > >> @ 0x411b0b320 - 17,624,664 (2.46%) bytes. > > > >> > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > >> @ 0x401fdb948 - 17,614,232 (2.46%) bytes. > > > >> > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > >> @ 0x4099afd28 - 17,606,640 (2.46%) bytes. > > > >> > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > >> @ 0x409895e10 - 17,599,608 (2.46%) bytes. > > > >> > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > >> @ 0x40d9e35e0 - 17,597,288 (2.46%) bytes. > > > >> > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > >> @ 0x409b1a978 - 17,581,512 (2.46%) bytes. > > > >> > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > >> @ 0x409969ad8 - 17,579,656 (2.46%) bytes. > > > >> > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > >> @ 0x40d9e5088 - 17,559,432 (2.45%) bytes. > > > >> > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > >> @ 0x410fa8d80 - 17,552,744 (2.45%) bytes. > > > >> > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > >> @ 0x404db4d38 - 17,547,832 (2.45%) bytes. > > > >> > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > >> @ 0x4197b64f0 - 17,529,272 (2.45%) bytes. > > > >> > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > >> @ 0x409a03058 - 17,521,432 (2.45%) bytes. > > > >> > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > >> @ 0x411b55168 - 17,495,208 (2.45%) bytes. > > > >> > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > >> @ 0x40b0b3c30 - 17,481,480 (2.44%) bytes. > > > >> > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > >> @ 0x4084fe4b0 - 17,478,696 (2.44%) bytes. > > > >> > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > >> @ 0x4195a5dc8 - 17,476,840 (2.44%) bytes. > > > >> > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > >> @ 0x419725708 - 17,472,008 (2.44%) bytes. > > > >> > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > >> @ 0x409a441c8 - 17,453,176 (2.44%) bytes. > > > >> > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > >> @ 0x404d89060 - 17,434,424 (2.44%) bytes. > > > >> > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > >> @ 0x40f875d80 - 17,425,144 (2.44%) bytes. > > > >> > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > >> @ 0x4102071f8 - 17,408,104 (2.43%) bytes. > > > >> > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > >> @ 0x40d9e34f8 - 17,383,576 (2.43%) bytes. > > > >> > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > >> @ 0x409926318 - 17,374,568 (2.43%) bytes. > > > >> > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > >> @ 0x409a7cde8 - 17,318,888 (2.42%) bytes. > > > >> > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > >> @ 0x40aadd150 - 17,285,208 (2.42%) bytes. > > > >> > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > >> @ 0x40d81ba08 - 17,273,880 (2.41%) bytes. > > > >> > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > >> @ 0x40070d068 - 17,223,032 (2.41%) bytes. > > > >> > > > > > > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > >> @ 0x41bea4538 - 17,146,936 (2.40%) bytes. > > > >> > > > >> Most of these instances are referenced from one instance of > > > >> > > > > > > “org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl”, > > > >> loaded by “java.net.URLClassLoader @ 0x400000000”, which occupies > > 7,008 > > > >> (0.00%) bytes. > > > >> > > > >> Thread > > > >> > > > > > > “org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$FailureCheckAndFlushThread > > > >> @ 0x405ffd540 activemq-failure-check-thread” has a local variable or > > > >> reference to > > > >> > > > > > > “org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl > > > >> @ 0x401a42658” which is on the shortest path to > > > >> > > > > > > “org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl > > > >> @ 0x401a42658”. The thread > > > >> > > > > > > org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$FailureCheckAndFlushThread > > > >> @ 0x405ffd540 activemq-failure-check-thread keeps local variables > with > > > >> total size 328 (0.00%) bytes. > > > >> > > > >> The stacktrace of this Thread is available. See stacktrace. See > > > >> stacktrace with involved local variables. > > > >> > > > >> Keywords > > > >> > > > > > > org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > > > >> java.net.URLClassLoader > > > >> > > > > org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl > > > >> > > > >> Thanks, > > > >> Dragan > > > >> > > > >> > > > >> On Tue, Nov 19, 2024 at 12:13 PM Dragan J <draga...@flipside.team> > > > wrote: > > > >> > > > >>> Hey Justin, > > > >>> > > > >>> Here are the answers: > > > >>> - Are all the bridges configured the same way? How exactly are they > > > >>> configured? > > > >>> Yes, they are configured the same way. > > > >>> The CoreBridge configuration is attached to > > > >>> > > > > > > https://drive.google.com/file/d/1VEwj1MoB2tbcCp8R4XeQTzi80Q0dQ606/view?usp=sharing > > > >>> > > > >>> - Is the bridge target available when you see this increase happen? > > > >>> The bridges are connected from the client side to the server (with > > the > > > >>> 2.38 Artemis version) > > > >>> - Are the bridges actually moving messages when you see the > increase > > or > > > >>> are they idle? > > > >>> Most of the time, messages are idle. > > > >>> - Are 3,500 bridges actually required to trigger the increase? Do > > you, > > > >>> for > > > >>> example, not see the increase if you use 3,000 bridges? What about > > > 1,000 > > > >>> or > > > >>> 100? > > > >>> We tested with 3500 CoreBridges in the development environment, > and I > > > >>> should ask for permission to close some connections. > > > >>> If you require it, I would ask for that kind of permission. > > > >>> > > > >>> Thanks, > > > >>> Dragan > > > >>> > > > >>> On Mon, Nov 18, 2024 at 2:43 PM Dragan J <draga...@flipside.team> > > > wrote: > > > >>> > > > >>>> Can you access the screenshots? > > > >>>> > > > >>>> On Thu, Nov 14, 2024 at 10:18 AM Dragan J <draga...@flipside.team > > > > > >>>> wrote: > > > >>>> > > > >>>>> Sorry, a typo. It's 3,500 CoreBridges > > > >>>>> > > > >>>>> On Thu, Nov 14, 2024 at 10:14 AM Dragan J <draga...@flipside.team > > > > > >>>>> wrote: > > > >>>>> > > > >>>>>> @Justin The broker (version 2.38) communicates with other > brokers > > > >>>>>> (version 2.37) through 35,000 CoreBridges. There was minimal > > message > > > >>>>>> exchange between them. > > > >>>>>> Please find the two graphs (from earlier) attached on my Google > > > >>>>>> Drive. Please let me know if you have an alternative way to > share > > > files. > > > >>>>>> > > > >>>>>> > > > > > > https://drive.google.com/file/d/1Gy8XQi153Fi5Zm9gUdsP_co8fSuh1HjL/view?usp=sharing > > > >>>>>> > > > >>>>>> > > > > > > https://drive.google.com/file/d/1_Fbg8OmWhlsOKfalfIEwKxGIl97a1kQ5/view?usp=sharing > > > >>>>>> > > > >>>>>> @Arthur When I tried to perform a GC from JConsole I got the > > > >>>>>> following: > > > >>>>>> > > > >>>>>> Exception in thread "MemoryPanel.gc" > java.lang.SecurityException: > > > >>>>>> User not authorized to access operation: gc > > > >>>>>> > > > >>>>> > > > > > >
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@activemq.apache.org For additional commands, e-mail: dev-h...@activemq.apache.org For further information, visit: https://activemq.apache.org/contact
