@Dragan: what I usually do with memory leaks is by running the test with check-leak.. and finding what objects are growing over time.
I had also written tests like this to identify and prevent leaks: https://github.com/apache/activemq-artemis/blob/14c564a48144b0e052c6e0eeee64ba0f94744335/tests/leak-tests/src/test/java/org/apache/activemq/artemis/tests/leak/RedistributorLeakTest.java#L126-L128 Notice CheckLeak here will count the number of objects... There's also an option to take class histograms and then generate graphs to what's growing overtime. if you provide some testing at any level we can take the test up to that level. On Fri, Nov 22, 2024 at 5:20 PM Justin Bertram <jbert...@apache.org> wrote: > > Also, I strongly encourage you to actually subscribe to this mailing list > [1]. Currently you are not subscribed and every email you send must pass > through a moderation process before being posted to the list. > > > Justin > > [1] https://activemq.apache.org/contact > > On Fri, Nov 22, 2024 at 4:02 PM Justin Bertram <jbert...@apache.org> wrote: > > > > ...the business requirement is to use CoreBridges. > > > > I find that a bit strange. Typically business requirements aren't > > specified in terms of a technology. Typically they're expressed in business > > terms (e.g. move retail orders from store A to store B). A Core bridge is > > an implementation specific way to reach a business goal, but I digress. > > > > I did a quick proof-of-concept deployment with 2 instances of 2.38.0. One > > instance had 1,000 bridges connecting to the other instance. The instance > > with the bridges show no growth in heap usage over half an hour or so. It > > was sitting comfortably around 100MB the whole time. > > > > At this point you need to come up with a test-case to reliably reproduce > > the problem you're seeing. I don't want to invest more time into this > > investigation without that. Thanks! > > > > > > Justin > > > > On Fri, Nov 22, 2024 at 8:10 AM Dragan J <draga...@flipside.team> wrote: > > > >> @Clebert Unfortunately, the business requirement is to use CoreBridges. > >> @Justin We tested with 1000 CoreBridges, and a memory leak remains. > >> Here is "the problem suspect" from the Memory Analyzer (a heap dump > >> analyzed): > >> > >> > >> 1,254 instances of > >> > >> “org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl”, > >> loaded by “java.net.URLClassLoader @ 0x400000000” occupy 527,112,720 > >> (73.68%) bytes. > >> > >> Biggest instances: > >> > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> @ 0x40d81e128 - 17,669,864 (2.47%) bytes. > >> > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> @ 0x409926c00 - 17,643,464 (2.47%) bytes. > >> > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> @ 0x411b0b320 - 17,624,664 (2.46%) bytes. > >> > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> @ 0x401fdb948 - 17,614,232 (2.46%) bytes. > >> > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> @ 0x4099afd28 - 17,606,640 (2.46%) bytes. > >> > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> @ 0x409895e10 - 17,599,608 (2.46%) bytes. > >> > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> @ 0x40d9e35e0 - 17,597,288 (2.46%) bytes. > >> > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> @ 0x409b1a978 - 17,581,512 (2.46%) bytes. > >> > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> @ 0x409969ad8 - 17,579,656 (2.46%) bytes. > >> > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> @ 0x40d9e5088 - 17,559,432 (2.45%) bytes. > >> > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> @ 0x410fa8d80 - 17,552,744 (2.45%) bytes. > >> > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> @ 0x404db4d38 - 17,547,832 (2.45%) bytes. > >> > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> @ 0x4197b64f0 - 17,529,272 (2.45%) bytes. > >> > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> @ 0x409a03058 - 17,521,432 (2.45%) bytes. > >> > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> @ 0x411b55168 - 17,495,208 (2.45%) bytes. > >> > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> @ 0x40b0b3c30 - 17,481,480 (2.44%) bytes. > >> > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> @ 0x4084fe4b0 - 17,478,696 (2.44%) bytes. > >> > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> @ 0x4195a5dc8 - 17,476,840 (2.44%) bytes. > >> > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> @ 0x419725708 - 17,472,008 (2.44%) bytes. > >> > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> @ 0x409a441c8 - 17,453,176 (2.44%) bytes. > >> > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> @ 0x404d89060 - 17,434,424 (2.44%) bytes. > >> > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> @ 0x40f875d80 - 17,425,144 (2.44%) bytes. > >> > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> @ 0x4102071f8 - 17,408,104 (2.43%) bytes. > >> > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> @ 0x40d9e34f8 - 17,383,576 (2.43%) bytes. > >> > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> @ 0x409926318 - 17,374,568 (2.43%) bytes. > >> > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> @ 0x409a7cde8 - 17,318,888 (2.42%) bytes. > >> > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> @ 0x40aadd150 - 17,285,208 (2.42%) bytes. > >> > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> @ 0x40d81ba08 - 17,273,880 (2.41%) bytes. > >> > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> @ 0x40070d068 - 17,223,032 (2.41%) bytes. > >> > >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> @ 0x41bea4538 - 17,146,936 (2.40%) bytes. > >> > >> Most of these instances are referenced from one instance of > >> > >> “org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl”, > >> loaded by “java.net.URLClassLoader @ 0x400000000”, which occupies 7,008 > >> (0.00%) bytes. > >> > >> Thread > >> > >> “org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$FailureCheckAndFlushThread > >> @ 0x405ffd540 activemq-failure-check-thread” has a local variable or > >> reference to > >> “org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl > >> @ 0x401a42658” which is on the shortest path to > >> “org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl > >> @ 0x401a42658”. The thread > >> > >> org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$FailureCheckAndFlushThread > >> @ 0x405ffd540 activemq-failure-check-thread keeps local variables with > >> total size 328 (0.00%) bytes. > >> > >> The stacktrace of this Thread is available. See stacktrace. See stacktrace > >> with involved local variables. > >> > >> Keywords > >> org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl > >> java.net.URLClassLoader > >> org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl > >> > >> Thanks, > >> Dragan > >> > >> > >> On Tue, Nov 19, 2024 at 12:13 PM Dragan J <draga...@flipside.team> wrote: > >> > >> > Hey Justin, > >> > > >> > Here are the answers: > >> > - Are all the bridges configured the same way? How exactly are they > >> > configured? > >> > Yes, they are configured the same way. > >> > The CoreBridge configuration is attached to > >> > > >> https://drive.google.com/file/d/1VEwj1MoB2tbcCp8R4XeQTzi80Q0dQ606/view?usp=sharing > >> > > >> > - Is the bridge target available when you see this increase happen? > >> > The bridges are connected from the client side to the server (with the > >> > 2.38 Artemis version) > >> > - Are the bridges actually moving messages when you see the increase or > >> > are they idle? > >> > Most of the time, messages are idle. > >> > - Are 3,500 bridges actually required to trigger the increase? Do you, > >> for > >> > example, not see the increase if you use 3,000 bridges? What about > >> 1,000 or > >> > 100? > >> > We tested with 3500 CoreBridges in the development environment, and I > >> > should ask for permission to close some connections. > >> > If you require it, I would ask for that kind of permission. > >> > > >> > Thanks, > >> > Dragan > >> > > >> > On Mon, Nov 18, 2024 at 2:43 PM Dragan J <draga...@flipside.team> > >> wrote: > >> > > >> >> Can you access the screenshots? > >> >> > >> >> On Thu, Nov 14, 2024 at 10:18 AM Dragan J <draga...@flipside.team> > >> wrote: > >> >> > >> >>> Sorry, a typo. It's 3,500 CoreBridges > >> >>> > >> >>> On Thu, Nov 14, 2024 at 10:14 AM Dragan J <draga...@flipside.team> > >> >>> wrote: > >> >>> > >> >>>> @Justin The broker (version 2.38) communicates with other brokers > >> >>>> (version 2.37) through 35,000 CoreBridges. There was minimal message > >> >>>> exchange between them. > >> >>>> Please find the two graphs (from earlier) attached on my Google > >> Drive. > >> >>>> Please let me know if you have an alternative way to share files. > >> >>>> > >> >>>> > >> https://drive.google.com/file/d/1Gy8XQi153Fi5Zm9gUdsP_co8fSuh1HjL/view?usp=sharing > >> >>>> > >> >>>> > >> https://drive.google.com/file/d/1_Fbg8OmWhlsOKfalfIEwKxGIl97a1kQ5/view?usp=sharing > >> >>>> > >> >>>> @Arthur When I tried to perform a GC from JConsole I got the > >> following: > >> >>>> > >> >>>> Exception in thread "MemoryPanel.gc" java.lang.SecurityException: > >> User > >> >>>> not authorized to access operation: gc > >> >>>> > >> >>> > >> > > -- Clebert Suconic --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@activemq.apache.org For additional commands, e-mail: dev-h...@activemq.apache.org For further information, visit: https://activemq.apache.org/contact
