Hi I have tried with 2200+ client application connects to Artemis server and publishing data in an interval of 30 sec(1100 messages per minute).
At some point Artemis server was not responsive. Find the below logs. 2020-12-23 13:01:33,245 WARN [org.apache.activemq.artemis.core.server] AMQ222214: Destination *.Ax1.* has an inconsistent and negative address size=-227,980,495. 2020-12-23 13:01:33,518 WARN [org.apache.activemq.artemis.core.server] AMQ222214: Destination *.Ax1.* has an inconsistent and negative address size=-227,980,559. 2020-12-23 13:01:33,707 WARN [org.apache.activemq.artemis.core.server] AMQ222214: Destination *.Ax1.* has an inconsistent and negative address size=-227,980,495. 2020-12-23 13:01:33,707 WARN [org.apache.activemq.artemis.core.server] AMQ222214: Destination *.Ax1.* has an inconsistent and negative address size=-227,980,431. 2020-12-23 13:01:36,815 WARN [org.apache.activemq.artemis.utils.actors.OrderedExecutor] GC overhead limit exceeded: java.lang.OutOfMemoryError: GC overhead limit exceeded 2020-12-23 13:02:01,856 WARN [org.apache.activemq.artemis.utils.actors.OrderedExecutor] GC overhead limit exceeded: java.lang.OutOfMemoryError: GC overhead limit exceeded 2020-12-23 13:02:04,724 WARN [org.eclipse.jetty.util.thread.strategy.EatWhatYouKill] : java.lang.OutOfMemoryError: GC overhead limit exceeded 2020-12-23 13:01:45,404 WARN [org.eclipse.jetty.server.HttpChannel] /console/jolokia/: java.lang.OutOfMemoryError: GC overhead limit exceeded 2020-12-23 13:01:55,386 WARN [io.netty.util.concurrent.SingleThreadEventExecutor] Unexpected exception from an event executor: : java.lang.OutOfMemoryError: GC overhead limit exceeded 2020-12-23 13:01:52,191 WARN [org.apache.activemq.artemis.utils.actors.OrderedExecutor] GC overhead limit exceeded: java.lang.OutOfMemoryError: GC overhead limit exceeded 2020-12-23 13:01:35,482 WARN [org.apache.activemq.artemis.core.server] AMQ222214: Destination *.Ax1.* has an inconsistent and negative address size=-227,980,495. 2020-12-23 13:01:42,381 WARN [io.netty.channel.nio.NioEventLoop] Unexpected exception in the selector loop.: java.lang.OutOfMemoryError: GC overhead limit exceeded 2020-12-23 13:01:50,721 WARN [org.apache.activemq.artemis.core.server] AMQ222082: error on connection failure check: java.lang.OutOfMemoryError: GC overhead limit exceeded 2020-12-23 13:02:30,180 WARN [io.netty.channel.AbstractChannelHandlerContext] An exception 'java.lang.OutOfMemoryError: GC overhead limit exceeded' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception:: java.lang.OutOfMemoryError: GC overhead limit exceeded 2020-12-23 13:02:03,396 WARN [org.apache.activemq.artemis.utils.actors.OrderedExecutor] GC overhead limit exceeded: java.lang.OutOfMemoryError: GC overhead limit exceeded 2020-12-23 13:01:55,649 WARN [io.netty.channel.AbstractChannelHandlerContext] An exception 'java.lang.OutOfMemoryError: GC overhead limit exceeded' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception:: java.lang.OutOfMemoryError: GC overhead limit exceeded 2020-12-23 13:01:50,721 WARN [org.apache.activemq.artemis.utils.actors.OrderedExecutor] GC overhead limit exceeded: java.lang.OutOfMemoryError: GC overhead limit exceeded 2020-12-23 13:02:11,042 WARN [org.apache.activemq.artemis.utils.actors.OrderedExecutor] GC overhead limit exceeded: java.lang.OutOfMemoryError: GC overhead limit exceeded 2020-12-23 13:02:13,987 WARN [org.apache.activemq.artemis.core.server] AMQ222231: Failed to flush outstanding data from the connection: java.lang.OutOfMemoryError: GC overhead limit exceeded 2020-12-23 13:02:08,870 WARN [io.netty.channel.AbstractChannelHandlerContext] An exception 'java.lang.OutOfMemoryError: GC overhead limit exceeded' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception:: java.lang.OutOfMemoryError: GC overhead limit exceeded 2020-12-23 13:02:31,963 WARN [org.apache.activemq.artemis.utils.actors.OrderedExecutor] GC overhead limit exceeded: java.lang.OutOfMemoryError: GC overhead limit exceeded 2020-12-23 13:01:54,186 WARN [io.netty.util.concurrent.AbstractEventExecutor] A task raised an exception. Task: org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext$TickerRunnable@be67394: java.lang.OutOfMemoryError: GC overhead limit exceeded 2020-12-23 13:01:52,811 WARN [org.apache.activemq.artemis.utils.actors.OrderedExecutor] GC overhead limit exceeded: java.lang.OutOfMemoryError: GC overhead limit exceeded 2020-12-23 13:01:52,895 WARN [io.netty.channel.nio.NioEventLoop] Unexpected exception in the selector loop.: java.lang.OutOfMemoryError: GC overhead limit exceeded 2020-12-23 13:02:32,050 WARN [org.apache.activemq.artemis.utils.actors.OrderedExecutor] GC overhead limit exceeded: java.lang.OutOfMemoryError: GC overhead limit exceeded 2020-12-23 13:03:26,702 WARN [org.apache.activemq.artemis.utils.actors.OrderedExecutor] GC overhead limit exceeded: java.lang.OutOfMemoryError: GC overhead limit exceeded 2020-12-23 13:03:27,348 WARN [org.apache.activemq.artemis.utils.actors.OrderedExecutor] GC overhead limit exceeded: java.lang.OutOfMemoryError: GC overhead limit exceeded Thanks, Mohan ________________________________ From: Justin Bertram <[email protected]> Sent: Thursday, December 17, 2020 9:52 PM To: [email protected] <[email protected]> Subject: Re: ActiveMQ Artemis Client interface (AMQP vs REST) CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. I recommend you ask about building on Windows on the Qpid user list [1]. That said, even if you can't build on Windows you could potentially run it under WSL or even a full-blown VM. Justin [1] https://qpid.apache.org/discussion.html On Thu, Dec 17, 2020 at 2:49 AM Mohan Kumar <[email protected]> wrote: > Thanks for your reply. > > As per the documentation QPID Dispatcher "Note : Dispatch Router will not > build on Windows." > Our applications mainly runs on Windows OS, is there any way to build and > run dispatcher application in Windows OS. > > Thanks, > Mohan > > > > -----Original Message----- > From: Justin Bertram <[email protected]> > Sent: Thursday, December 17, 2020 7:11 AM > To: [email protected] > Subject: Re: ActiveMQ Artemis Client interface (AMQP vs REST) > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you recognize the sender and know > the content is safe. > > > You don't really provide any meaningful info about what exactly is > challenging to manage and troubleshoot with 10,000 connections. Therefore, > your idea of moving to REST in the first place could be fundamentally > misguided. The trouble (whatever it was) could be tied to something other > than connection management which may not be solved by offloading that work > to an HTTP server. > > Also, you flatly assert that Artemis wasn't built with the intention of > serving very large numbers of endpoints but simply with the intention of > moving messages quickly between endpoints. However, Artemis was designed > with high concurrency and vertical scalability in mind so I would push back > on your assertion here. > > There is no arbitrary limit for the number of connections which a broker > can support. The functional limit will depend on your hardware and your > use-case. > > I don't really understand your question about if the "broker connection > has any dependency with acceptor." Can you clarify this? > > Ultimately I would recommend against using the REST interface as a way to > offload connection management. It wasn't designed for this purpose so I > wouldn't expect it to scale well. In other words, I would expect > performance to be poor. Keep in mind that messaging connections (unlike > HTTP connections) are *stateful* and that state must be tracked somewhere. > You can't just switch to REST and expect that to go away. > > If you really need to scale up the number of AMQP connections beyond what > the broker is able to support I would recommend using Qpid Dispatch Router > [1] as a connection concentrator. > > > Justin > > [1] https://qpid.apache.org/components/dispatch-router/index.html > > > > On Mon, Dec 14, 2020 at 1:57 AM Mohan Kumar <[email protected]> > wrote: > > > Hi, > > > > Based on the suggestion from Justin Bertram I am posting my query here. > > > > We are new to ActiveMQ Artemis world. > > When we started in ActiveMQ Artemis Broker we choose AMQP protocol to > > produce and consume data from broker. > > But as per the suggestion we received from ActiveMQ Artemis consultant > > we switched from AMQP to REST interface. > > > > Reason to switch from AMQP to REST > > > > * As per the suggestion > > > > Using AMQP, one of the problem we run into managing the connections > > come and go it is hard to get lot of insight into what's going on the > broker. > > > > So it is challenging to manage and troubleshoot. > > > > broker aren't built with the intension of serving very large numbers > > of endpoints they built with the intension of moving messages quickly > > between endpoints. > > > > Rest: The tooling which are available in HTTP, and for scaling and for > > frontend and it is really for superior to broker itself > > > > > > > > > > > > * When we create 10000 connection using AMQP, It creates 10000 > > connections in Artemis broker(i.e. 10000 clients : 10000 connection in > > Artemis broker) > > > > But using rest, 10000 clients connecting to HTTP server, creates 10000 > > connection at HTTP server and there is only one connection from HTTP > > server to REST interface. > > > > So there is less load in broker(less number of connection in broker) > > and connection management comes to REST layer. > > > > > > > > > > Our requirement is, > > sensors (client which connect to Artemis server) in concurrent way we > > are using AMQP acceptor in broker We would like to know: > > 1. What would be maximum concurrent connection could be handled by > > Artemis broker (includes both publisher and subscriber) 2. Does broker > > connection has any dependency with acceptor (STOMP, AMQP, HTTP etc...) > > > > Thanks, > > Mohan > > > > >
