As the exception suggest, the heap size seems not enough. I suggest to raise it and print some GC logs to check the amount of stop of the world pauses.
Il mer 23 dic 2020, 20:25 Mohan Kumar <[email protected]> ha scritto: > Hi > > I have tried with 2200+ client application connects to Artemis server and > publishing data in an interval of 30 sec(1100 messages per minute). > > At some point Artemis server was not responsive. > Find the below logs. > > 2020-12-23 13:01:33,245 WARN [org.apache.activemq.artemis.core.server] > AMQ222214: Destination *.Ax1.* has an inconsistent and negative address > size=-227,980,495. > 2020-12-23 13:01:33,518 WARN [org.apache.activemq.artemis.core.server] > AMQ222214: Destination *.Ax1.* has an inconsistent and negative address > size=-227,980,559. > 2020-12-23 13:01:33,707 WARN [org.apache.activemq.artemis.core.server] > AMQ222214: Destination *.Ax1.* has an inconsistent and negative address > size=-227,980,495. > 2020-12-23 13:01:33,707 WARN [org.apache.activemq.artemis.core.server] > AMQ222214: Destination *.Ax1.* has an inconsistent and negative address > size=-227,980,431. > 2020-12-23 13:01:36,815 WARN > [org.apache.activemq.artemis.utils.actors.OrderedExecutor] GC overhead > limit exceeded: java.lang.OutOfMemoryError: GC overhead limit exceeded > > 2020-12-23 13:02:01,856 WARN > [org.apache.activemq.artemis.utils.actors.OrderedExecutor] GC overhead > limit exceeded: java.lang.OutOfMemoryError: GC overhead limit exceeded > > 2020-12-23 13:02:04,724 WARN > [org.eclipse.jetty.util.thread.strategy.EatWhatYouKill] : > java.lang.OutOfMemoryError: GC overhead limit exceeded > > 2020-12-23 13:01:45,404 WARN [org.eclipse.jetty.server.HttpChannel] > /console/jolokia/: java.lang.OutOfMemoryError: GC overhead limit exceeded > > 2020-12-23 13:01:55,386 WARN > [io.netty.util.concurrent.SingleThreadEventExecutor] Unexpected exception > from an event executor: : java.lang.OutOfMemoryError: GC overhead limit > exceeded > > 2020-12-23 13:01:52,191 WARN > [org.apache.activemq.artemis.utils.actors.OrderedExecutor] GC overhead > limit exceeded: java.lang.OutOfMemoryError: GC overhead limit exceeded > > 2020-12-23 13:01:35,482 WARN [org.apache.activemq.artemis.core.server] > AMQ222214: Destination *.Ax1.* has an inconsistent and negative address > size=-227,980,495. > 2020-12-23 13:01:42,381 WARN [io.netty.channel.nio.NioEventLoop] > Unexpected exception in the selector loop.: java.lang.OutOfMemoryError: GC > overhead limit exceeded > > 2020-12-23 13:01:50,721 WARN [org.apache.activemq.artemis.core.server] > AMQ222082: error on connection failure check: java.lang.OutOfMemoryError: > GC overhead limit exceeded > > 2020-12-23 13:02:30,180 WARN > [io.netty.channel.AbstractChannelHandlerContext] An exception > 'java.lang.OutOfMemoryError: GC overhead limit exceeded' [enable DEBUG > level for full stacktrace] was thrown by a user handler's exceptionCaught() > method while handling the following exception:: java.lang.OutOfMemoryError: > GC overhead limit exceeded > > 2020-12-23 13:02:03,396 WARN > [org.apache.activemq.artemis.utils.actors.OrderedExecutor] GC overhead > limit exceeded: java.lang.OutOfMemoryError: GC overhead limit exceeded > > 2020-12-23 13:01:55,649 WARN > [io.netty.channel.AbstractChannelHandlerContext] An exception > 'java.lang.OutOfMemoryError: GC overhead limit exceeded' [enable DEBUG > level for full stacktrace] was thrown by a user handler's exceptionCaught() > method while handling the following exception:: java.lang.OutOfMemoryError: > GC overhead limit exceeded > > 2020-12-23 13:01:50,721 WARN > [org.apache.activemq.artemis.utils.actors.OrderedExecutor] GC overhead > limit exceeded: java.lang.OutOfMemoryError: GC overhead limit exceeded > > 2020-12-23 13:02:11,042 WARN > [org.apache.activemq.artemis.utils.actors.OrderedExecutor] GC overhead > limit exceeded: java.lang.OutOfMemoryError: GC overhead limit exceeded > > 2020-12-23 13:02:13,987 WARN [org.apache.activemq.artemis.core.server] > AMQ222231: Failed to flush outstanding data from the connection: > java.lang.OutOfMemoryError: GC overhead limit exceeded > > 2020-12-23 13:02:08,870 WARN > [io.netty.channel.AbstractChannelHandlerContext] An exception > 'java.lang.OutOfMemoryError: GC overhead limit exceeded' [enable DEBUG > level for full stacktrace] was thrown by a user handler's exceptionCaught() > method while handling the following exception:: java.lang.OutOfMemoryError: > GC overhead limit exceeded > > 2020-12-23 13:02:31,963 WARN > [org.apache.activemq.artemis.utils.actors.OrderedExecutor] GC overhead > limit exceeded: java.lang.OutOfMemoryError: GC overhead limit exceeded > > 2020-12-23 13:01:54,186 WARN > [io.netty.util.concurrent.AbstractEventExecutor] A task raised an > exception. Task: > org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext$TickerRunnable@be67394: > java.lang.OutOfMemoryError: GC overhead limit exceeded > > 2020-12-23 13:01:52,811 WARN > [org.apache.activemq.artemis.utils.actors.OrderedExecutor] GC overhead > limit exceeded: java.lang.OutOfMemoryError: GC overhead limit exceeded > > 2020-12-23 13:01:52,895 WARN [io.netty.channel.nio.NioEventLoop] > Unexpected exception in the selector loop.: java.lang.OutOfMemoryError: GC > overhead limit exceeded > > 2020-12-23 13:02:32,050 WARN > [org.apache.activemq.artemis.utils.actors.OrderedExecutor] GC overhead > limit exceeded: java.lang.OutOfMemoryError: GC overhead limit exceeded > > 2020-12-23 13:03:26,702 WARN > [org.apache.activemq.artemis.utils.actors.OrderedExecutor] GC overhead > limit exceeded: java.lang.OutOfMemoryError: GC overhead limit exceeded > > 2020-12-23 13:03:27,348 WARN > [org.apache.activemq.artemis.utils.actors.OrderedExecutor] GC overhead > limit exceeded: java.lang.OutOfMemoryError: GC overhead limit exceeded > > > > Thanks, > Mohan > > ________________________________ > From: Justin Bertram <[email protected]> > Sent: Thursday, December 17, 2020 9:52 PM > To: [email protected] <[email protected]> > Subject: Re: ActiveMQ Artemis Client interface (AMQP vs REST) > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you recognize the sender and know > the content is safe. > > > I recommend you ask about building on Windows on the Qpid user list [1]. > > That said, even if you can't build on Windows you could potentially run it > under WSL or even a full-blown VM. > > > Justin > > [1] https://qpid.apache.org/discussion.html > > On Thu, Dec 17, 2020 at 2:49 AM Mohan Kumar <[email protected]> > wrote: > > > Thanks for your reply. > > > > As per the documentation QPID Dispatcher "Note : Dispatch Router will not > > build on Windows." > > Our applications mainly runs on Windows OS, is there any way to build and > > run dispatcher application in Windows OS. > > > > Thanks, > > Mohan > > > > > > > > -----Original Message----- > > From: Justin Bertram <[email protected]> > > Sent: Thursday, December 17, 2020 7:11 AM > > To: [email protected] > > Subject: Re: ActiveMQ Artemis Client interface (AMQP vs REST) > > > > CAUTION: This email originated from outside of the organization. Do not > > click links or open attachments unless you recognize the sender and know > > the content is safe. > > > > > > You don't really provide any meaningful info about what exactly is > > challenging to manage and troubleshoot with 10,000 connections. > Therefore, > > your idea of moving to REST in the first place could be fundamentally > > misguided. The trouble (whatever it was) could be tied to something other > > than connection management which may not be solved by offloading that > work > > to an HTTP server. > > > > Also, you flatly assert that Artemis wasn't built with the intention of > > serving very large numbers of endpoints but simply with the intention of > > moving messages quickly between endpoints. However, Artemis was designed > > with high concurrency and vertical scalability in mind so I would push > back > > on your assertion here. > > > > There is no arbitrary limit for the number of connections which a broker > > can support. The functional limit will depend on your hardware and your > > use-case. > > > > I don't really understand your question about if the "broker connection > > has any dependency with acceptor." Can you clarify this? > > > > Ultimately I would recommend against using the REST interface as a way to > > offload connection management. It wasn't designed for this purpose so I > > wouldn't expect it to scale well. In other words, I would expect > > performance to be poor. Keep in mind that messaging connections (unlike > > HTTP connections) are *stateful* and that state must be tracked > somewhere. > > You can't just switch to REST and expect that to go away. > > > > If you really need to scale up the number of AMQP connections beyond what > > the broker is able to support I would recommend using Qpid Dispatch > Router > > [1] as a connection concentrator. > > > > > > Justin > > > > [1] https://qpid.apache.org/components/dispatch-router/index.html > > > > > > > > On Mon, Dec 14, 2020 at 1:57 AM Mohan Kumar <[email protected] > > > > wrote: > > > > > Hi, > > > > > > Based on the suggestion from Justin Bertram I am posting my query here. > > > > > > We are new to ActiveMQ Artemis world. > > > When we started in ActiveMQ Artemis Broker we choose AMQP protocol to > > > produce and consume data from broker. > > > But as per the suggestion we received from ActiveMQ Artemis consultant > > > we switched from AMQP to REST interface. > > > > > > Reason to switch from AMQP to REST > > > > > > * As per the suggestion > > > > > > Using AMQP, one of the problem we run into managing the connections > > > come and go it is hard to get lot of insight into what's going on the > > broker. > > > > > > So it is challenging to manage and troubleshoot. > > > > > > broker aren't built with the intension of serving very large numbers > > > of endpoints they built with the intension of moving messages quickly > > > between endpoints. > > > > > > Rest: The tooling which are available in HTTP, and for scaling and for > > > frontend and it is really for superior to broker itself > > > > > > > > > > > > > > > > > > * When we create 10000 connection using AMQP, It creates 10000 > > > connections in Artemis broker(i.e. 10000 clients : 10000 connection in > > > Artemis broker) > > > > > > But using rest, 10000 clients connecting to HTTP server, creates 10000 > > > connection at HTTP server and there is only one connection from HTTP > > > server to REST interface. > > > > > > So there is less load in broker(less number of connection in broker) > > > and connection management comes to REST layer. > > > > > > > > > > > > > > > Our requirement is, > > > sensors (client which connect to Artemis server) in concurrent way we > > > are using AMQP acceptor in broker We would like to know: > > > 1. What would be maximum concurrent connection could be handled by > > > Artemis broker (includes both publisher and subscriber) 2. Does broker > > > connection has any dependency with acceptor (STOMP, AMQP, HTTP etc...) > > > > > > Thanks, > > > Mohan > > > > > > > > >
