> ...the business requirement is to use CoreBridges. I find that a bit strange. Typically business requirements aren't specified in terms of a technology. Typically they're expressed in business terms (e.g. move retail orders from store A to store B). A Core bridge is an implementation specific way to reach a business goal, but I digress.
I did a quick proof-of-concept deployment with 2 instances of 2.38.0. One instance had 1,000 bridges connecting to the other instance. The instance with the bridges show no growth in heap usage over half an hour or so. It was sitting comfortably around 100MB the whole time. At this point you need to come up with a test-case to reliably reproduce the problem you're seeing. I don't want to invest more time into this investigation without that. Thanks! Justin On Fri, Nov 22, 2024 at 8:10 AM Dragan J <draga...@flipside.team> wrote: > @Clebert Unfortunately, the business requirement is to use CoreBridges. > @Justin We tested with 1000 CoreBridges, and a memory leak remains. > Here is "the problem suspect" from the Memory Analyzer (a heap dump > analyzed): > > > 1,254 instances of > > “org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl”, > loaded by “java.net.URLClassLoader @ 0x400000000” occupy 527,112,720 > (73.68%) bytes. > > Biggest instances: > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > @ 0x40d81e128 - 17,669,864 (2.47%) bytes. > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > @ 0x409926c00 - 17,643,464 (2.47%) bytes. > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > @ 0x411b0b320 - 17,624,664 (2.46%) bytes. > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > @ 0x401fdb948 - 17,614,232 (2.46%) bytes. > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > @ 0x4099afd28 - 17,606,640 (2.46%) bytes. > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > @ 0x409895e10 - 17,599,608 (2.46%) bytes. > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > @ 0x40d9e35e0 - 17,597,288 (2.46%) bytes. > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > @ 0x409b1a978 - 17,581,512 (2.46%) bytes. > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > @ 0x409969ad8 - 17,579,656 (2.46%) bytes. > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > @ 0x40d9e5088 - 17,559,432 (2.45%) bytes. > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > @ 0x410fa8d80 - 17,552,744 (2.45%) bytes. > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > @ 0x404db4d38 - 17,547,832 (2.45%) bytes. > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > @ 0x4197b64f0 - 17,529,272 (2.45%) bytes. > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > @ 0x409a03058 - 17,521,432 (2.45%) bytes. > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > @ 0x411b55168 - 17,495,208 (2.45%) bytes. > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > @ 0x40b0b3c30 - 17,481,480 (2.44%) bytes. > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > @ 0x4084fe4b0 - 17,478,696 (2.44%) bytes. > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > @ 0x4195a5dc8 - 17,476,840 (2.44%) bytes. > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > @ 0x419725708 - 17,472,008 (2.44%) bytes. > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > @ 0x409a441c8 - 17,453,176 (2.44%) bytes. > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > @ 0x404d89060 - 17,434,424 (2.44%) bytes. > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > @ 0x40f875d80 - 17,425,144 (2.44%) bytes. > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > @ 0x4102071f8 - 17,408,104 (2.43%) bytes. > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > @ 0x40d9e34f8 - 17,383,576 (2.43%) bytes. > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > @ 0x409926318 - 17,374,568 (2.43%) bytes. > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > @ 0x409a7cde8 - 17,318,888 (2.42%) bytes. > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > @ 0x40aadd150 - 17,285,208 (2.42%) bytes. > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > @ 0x40d81ba08 - 17,273,880 (2.41%) bytes. > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > @ 0x40070d068 - 17,223,032 (2.41%) bytes. > •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > @ 0x41bea4538 - 17,146,936 (2.40%) bytes. > > Most of these instances are referenced from one instance of > > “org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl”, > loaded by “java.net.URLClassLoader @ 0x400000000”, which occupies 7,008 > (0.00%) bytes. > > Thread > > “org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$FailureCheckAndFlushThread > @ 0x405ffd540 activemq-failure-check-thread” has a local variable or > reference to > “org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl > @ 0x401a42658” which is on the shortest path to > “org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl > @ 0x401a42658”. The thread > > org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$FailureCheckAndFlushThread > @ 0x405ffd540 activemq-failure-check-thread keeps local variables with > total size 328 (0.00%) bytes. > > The stacktrace of this Thread is available. See stacktrace. See stacktrace > with involved local variables. > > Keywords > org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > java.net.URLClassLoader > org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl > > Thanks, > Dragan > > > On Tue, Nov 19, 2024 at 12:13 PM Dragan J <draga...@flipside.team> wrote: > > > Hey Justin, > > > > Here are the answers: > > - Are all the bridges configured the same way? How exactly are they > > configured? > > Yes, they are configured the same way. > > The CoreBridge configuration is attached to > > > https://drive.google.com/file/d/1VEwj1MoB2tbcCp8R4XeQTzi80Q0dQ606/view?usp=sharing > > > > - Is the bridge target available when you see this increase happen? > > The bridges are connected from the client side to the server (with the > > 2.38 Artemis version) > > - Are the bridges actually moving messages when you see the increase or > > are they idle? > > Most of the time, messages are idle. > > - Are 3,500 bridges actually required to trigger the increase? Do you, > for > > example, not see the increase if you use 3,000 bridges? What about 1,000 > or > > 100? > > We tested with 3500 CoreBridges in the development environment, and I > > should ask for permission to close some connections. > > If you require it, I would ask for that kind of permission. > > > > Thanks, > > Dragan > > > > On Mon, Nov 18, 2024 at 2:43 PM Dragan J <draga...@flipside.team> wrote: > > > >> Can you access the screenshots? > >> > >> On Thu, Nov 14, 2024 at 10:18 AM Dragan J <draga...@flipside.team> > wrote: > >> > >>> Sorry, a typo. It's 3,500 CoreBridges > >>> > >>> On Thu, Nov 14, 2024 at 10:14 AM Dragan J <draga...@flipside.team> > >>> wrote: > >>> > >>>> @Justin The broker (version 2.38) communicates with other brokers > >>>> (version 2.37) through 35,000 CoreBridges. There was minimal message > >>>> exchange between them. > >>>> Please find the two graphs (from earlier) attached on my Google Drive. > >>>> Please let me know if you have an alternative way to share files. > >>>> > >>>> > https://drive.google.com/file/d/1Gy8XQi153Fi5Zm9gUdsP_co8fSuh1HjL/view?usp=sharing > >>>> > >>>> > https://drive.google.com/file/d/1_Fbg8OmWhlsOKfalfIEwKxGIl97a1kQ5/view?usp=sharing > >>>> > >>>> @Arthur When I tried to perform a GC from JConsole I got the > following: > >>>> > >>>> Exception in thread "MemoryPanel.gc" java.lang.SecurityException: User > >>>> not authorized to access operation: gc > >>>> > >>> >
