Also, I strongly encourage you to actually subscribe to this mailing list [1]. Currently you are not subscribed and every email you send must pass through a moderation process before being posted to the list.
Justin [1] https://activemq.apache.org/contact On Fri, Nov 22, 2024 at 4:02 PM Justin Bertram <jbert...@apache.org> wrote: > > ...the business requirement is to use CoreBridges. > > I find that a bit strange. Typically business requirements aren't > specified in terms of a technology. Typically they're expressed in business > terms (e.g. move retail orders from store A to store B). A Core bridge is > an implementation specific way to reach a business goal, but I digress. > > I did a quick proof-of-concept deployment with 2 instances of 2.38.0. One > instance had 1,000 bridges connecting to the other instance. The instance > with the bridges show no growth in heap usage over half an hour or so. It > was sitting comfortably around 100MB the whole time. > > At this point you need to come up with a test-case to reliably reproduce > the problem you're seeing. I don't want to invest more time into this > investigation without that. Thanks! > > > Justin > > On Fri, Nov 22, 2024 at 8:10 AM Dragan J <draga...@flipside.team> wrote: > >> @Clebert Unfortunately, the business requirement is to use CoreBridges. >> @Justin We tested with 1000 CoreBridges, and a memory leak remains. >> Here is "the problem suspect" from the Memory Analyzer (a heap dump >> analyzed): >> >> >> 1,254 instances of >> >> “org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl”, >> loaded by “java.net.URLClassLoader @ 0x400000000” occupy 527,112,720 >> (73.68%) bytes. >> >> Biggest instances: >> >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> @ 0x40d81e128 - 17,669,864 (2.47%) bytes. >> >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> @ 0x409926c00 - 17,643,464 (2.47%) bytes. >> >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> @ 0x411b0b320 - 17,624,664 (2.46%) bytes. >> >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> @ 0x401fdb948 - 17,614,232 (2.46%) bytes. >> >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> @ 0x4099afd28 - 17,606,640 (2.46%) bytes. >> >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> @ 0x409895e10 - 17,599,608 (2.46%) bytes. >> >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> @ 0x40d9e35e0 - 17,597,288 (2.46%) bytes. >> >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> @ 0x409b1a978 - 17,581,512 (2.46%) bytes. >> >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> @ 0x409969ad8 - 17,579,656 (2.46%) bytes. >> >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> @ 0x40d9e5088 - 17,559,432 (2.45%) bytes. >> >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> @ 0x410fa8d80 - 17,552,744 (2.45%) bytes. >> >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> @ 0x404db4d38 - 17,547,832 (2.45%) bytes. >> >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> @ 0x4197b64f0 - 17,529,272 (2.45%) bytes. >> >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> @ 0x409a03058 - 17,521,432 (2.45%) bytes. >> >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> @ 0x411b55168 - 17,495,208 (2.45%) bytes. >> >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> @ 0x40b0b3c30 - 17,481,480 (2.44%) bytes. >> >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> @ 0x4084fe4b0 - 17,478,696 (2.44%) bytes. >> >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> @ 0x4195a5dc8 - 17,476,840 (2.44%) bytes. >> >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> @ 0x419725708 - 17,472,008 (2.44%) bytes. >> >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> @ 0x409a441c8 - 17,453,176 (2.44%) bytes. >> >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> @ 0x404d89060 - 17,434,424 (2.44%) bytes. >> >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> @ 0x40f875d80 - 17,425,144 (2.44%) bytes. >> >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> @ 0x4102071f8 - 17,408,104 (2.43%) bytes. >> >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> @ 0x40d9e34f8 - 17,383,576 (2.43%) bytes. >> >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> @ 0x409926318 - 17,374,568 (2.43%) bytes. >> >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> @ 0x409a7cde8 - 17,318,888 (2.42%) bytes. >> >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> @ 0x40aadd150 - 17,285,208 (2.42%) bytes. >> >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> @ 0x40d81ba08 - 17,273,880 (2.41%) bytes. >> >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> @ 0x40070d068 - 17,223,032 (2.41%) bytes. >> >> •org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> @ 0x41bea4538 - 17,146,936 (2.40%) bytes. >> >> Most of these instances are referenced from one instance of >> >> “org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl”, >> loaded by “java.net.URLClassLoader @ 0x400000000”, which occupies 7,008 >> (0.00%) bytes. >> >> Thread >> >> “org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$FailureCheckAndFlushThread >> @ 0x405ffd540 activemq-failure-check-thread” has a local variable or >> reference to >> “org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl >> @ 0x401a42658” which is on the shortest path to >> “org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl >> @ 0x401a42658”. The thread >> >> org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$FailureCheckAndFlushThread >> @ 0x405ffd540 activemq-failure-check-thread keeps local variables with >> total size 328 (0.00%) bytes. >> >> The stacktrace of this Thread is available. See stacktrace. See stacktrace >> with involved local variables. >> >> Keywords >> org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl >> java.net.URLClassLoader >> org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl >> >> Thanks, >> Dragan >> >> >> On Tue, Nov 19, 2024 at 12:13 PM Dragan J <draga...@flipside.team> wrote: >> >> > Hey Justin, >> > >> > Here are the answers: >> > - Are all the bridges configured the same way? How exactly are they >> > configured? >> > Yes, they are configured the same way. >> > The CoreBridge configuration is attached to >> > >> https://drive.google.com/file/d/1VEwj1MoB2tbcCp8R4XeQTzi80Q0dQ606/view?usp=sharing >> > >> > - Is the bridge target available when you see this increase happen? >> > The bridges are connected from the client side to the server (with the >> > 2.38 Artemis version) >> > - Are the bridges actually moving messages when you see the increase or >> > are they idle? >> > Most of the time, messages are idle. >> > - Are 3,500 bridges actually required to trigger the increase? Do you, >> for >> > example, not see the increase if you use 3,000 bridges? What about >> 1,000 or >> > 100? >> > We tested with 3500 CoreBridges in the development environment, and I >> > should ask for permission to close some connections. >> > If you require it, I would ask for that kind of permission. >> > >> > Thanks, >> > Dragan >> > >> > On Mon, Nov 18, 2024 at 2:43 PM Dragan J <draga...@flipside.team> >> wrote: >> > >> >> Can you access the screenshots? >> >> >> >> On Thu, Nov 14, 2024 at 10:18 AM Dragan J <draga...@flipside.team> >> wrote: >> >> >> >>> Sorry, a typo. It's 3,500 CoreBridges >> >>> >> >>> On Thu, Nov 14, 2024 at 10:14 AM Dragan J <draga...@flipside.team> >> >>> wrote: >> >>> >> >>>> @Justin The broker (version 2.38) communicates with other brokers >> >>>> (version 2.37) through 35,000 CoreBridges. There was minimal message >> >>>> exchange between them. >> >>>> Please find the two graphs (from earlier) attached on my Google >> Drive. >> >>>> Please let me know if you have an alternative way to share files. >> >>>> >> >>>> >> https://drive.google.com/file/d/1Gy8XQi153Fi5Zm9gUdsP_co8fSuh1HjL/view?usp=sharing >> >>>> >> >>>> >> https://drive.google.com/file/d/1_Fbg8OmWhlsOKfalfIEwKxGIl97a1kQ5/view?usp=sharing >> >>>> >> >>>> @Arthur When I tried to perform a GC from JConsole I got the >> following: >> >>>> >> >>>> Exception in thread "MemoryPanel.gc" java.lang.SecurityException: >> User >> >>>> not authorized to access operation: gc >> >>>> >> >>> >> >
