Cezary, Please forgive if this is too simplistic of an thought, but we encountered similar symptoms due to a hung or very-slow consumer. AMQ was batching delivery of N messages, one consumer was getting stuck and so the N-1 pre-delivered messages would not be consumed by other available consumers until a timeout and redelivery.
john [rg] <https://www.redpointglobal.com/> John Lilley Data Management Chief Architect, Redpoint Global Inc. 888 Worcester Street, Suite 200 Wellesley, MA 02482 M: +1 7209385761<tel:+1%207209385761> | john.lil...@redpointglobal.com<mailto:john.lil...@redpointglobal.com> From: Cezary Majchrzak <cezary.majchrza...@gmail.com> Sent: Monday, January 2, 2023 7:15 AM To: users@activemq.apache.org Subject: Consumers stop receiving messages *** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. *** Hello, We are observing strange communication problems with the ActiveMQ Artemis broker in our system. When the problem occurs JmsListener stops receiving further messages despite the fact that previously consuming worked perfectly. The problem can occur on several queues but others at the same time work properly. The Artemis management panel on the problematic queues then indicates that deliveringCount > 0 and this value does not change. Consumer count at this time is non-zero. Restarting the broker or message consuming services does not always help. Sometimes messages are consumed for a short time after which the problem reappears. We noticed that this happens only when sending large messages (size of about 250 KB, Artemis saves them with a size twice as large due to encoding). Problematic queues process large and small messages or only large messages. Queues that work properly process only small messages. At the same time, the problem does not occur with every sending of large messages. We use message grouping, assigning each message a UUID at the beginning of processing, which is then used as a group identifier. We wonder if the large number of such groups (sometimes even several million new messages per day) can have a significant impact on memory consumption. Artemis configuration - Single instance of ActiveMQ Artemis broker (configured for master-slave operation, but only one instance is enabled). - The broker is running on AlmaLinux 8.4 OS. - Artemis version is 2.27.1 (updated from version 2.22.0 where the problem also occurred). - The broker.xml configuration file is attached. - One topic (omitting DLQ and ExpiryQueue) for which queues are created with appropriate filters. Application side configuration - Spring Boot version 2.6.13 with spring-boot-starter-artemis. - Subscriptions configured as durable and shared. - Sessions are transacted. What have we tried to solve the issue - JmsListener used a container with dynamic scaling of the number of consumers, while caching of consumers was enabled. We thought that this might pose a problem for a broker trying to deliver messages to consumers that no longer existed. We disabled caching of consumers and set maxMessagePerTask property, unfortunately this did not solve the problem. - We tried changing Spring Boot's CachingConnectionFactory to JmsPoolConnectionFactory from lib https://github.com/messaginghub/pooled-jms, but again the problem was not solved. - We took thread dumps in the services to make sure that the processing doesn't get stuck when executing business logic and interacting with external services. All threads of type JmsListenerEndpointContainer are in TIMED_WAITING state and the stacktrace indicates that they are waiting for messages from the broker in the receive method of class org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl. - Updated the broker version to the latest 2.27.1, but the same problem still occurs. - We tried changing the parameters of the acceptor in the broker.xml file, such as: amqpMinLargeMessageSize (despite changing this parameter, messages in the broker continue to be seen as large, despite the smaller size than declared), remotingThreads and directDeliver. No apparent effect on broker performance. - TCP dumps of the network traffic between the broker and the services consuming the messages show that the network communication is established and some data is sent from the broker. - We have changed the broker settings related to memory. Previously, the host had 32GB of RAM and the Artemis process was configured with the JVM -Xms and -Xmx parameters equal to 26GB and the global-max-size parameter set by default. We noticed that during a heavy load of large messages, in addition to the problem of not consuming messages, the host would sometimes reset itself through out of out of memory errors. For this reason, we increased the amount of RAM available to the host to 64GB and set the -Xms and -Xmx parameters to 50G, and changed the global-max-size to 10G as recommended by https://activemq.apache.org/components/artemis/documentation/latest/perf-tuning.html. The broker seemed to work more stably (one day processed about 3 million large messages without any problems), unfortunately after about a week of operation the problem of not consuming messages returned. I've attached below graphs of memory consumption during one such problem. I have numbered on them the consecutive times when we restarted the broker (coinciding with high GC time and high committed memory value). During the first three reboots, consuming resumed only for a moment, then stopped again. After the fourth reboot, consuming started working properly and all the messages came off the queues. [cid:image001.png@01D91ED2.DD7BE4F0] [cid:image002.png@01D91ED2.DD7BE4F0] Similar symptoms have been described here<https://stackoverflow.com/questions/74792977/no-data-being-sent-to-consumers-even-though-connection-and-session-are-created> but the proposed solutions do not seem to apply to us. Please provide ideas on how to solve the problem. Many thanks, Cezary Majchrzak PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
