subhajit maiti created AMQ-7066: ----------------------------------- Summary: Using dynamic scaling and message grouping feature together makes messages stuck Key: AMQ-7066 URL: https://issues.apache.org/jira/browse/AMQ-7066 Project: ActiveMQ Issue Type: Bug Components: activemq-camel Affects Versions: 5.15.5 Reporter: subhajit maiti
Recently we observed a strange problem in one of our flows. We could see some messages being stuck in the queue when we are using the message grouping and dynamic scaling feature together. *Usecase:* We want to process the messages belonging to a user (application user) in a sequence. Multiple users can send messages to the same queue, but messages originating from transactions made by a user should be processed sequentially. In order to do so we are using the "JMSXGroupID" header to group the messages and the "username" serves as the group key. *Our configuration and versions:* # We are using ActiveMQ 5.15.5 # We are using camel routing to connect to ActiveMQ, both on the producer and consumer side. Camel version : 2.20.1 # We are using the following properties in the consumer URI to enable dynamic scaling : maxConcurrentConsumers, idleTaskExecutionLimit (defaulted to 10) maxMessagesPerTask (defaulted to 250) # JDK version : 1.8.0_91 *Observations:* # The messages are stuck for some of the groups. Messages belonging to other groups are processing fine. There is no identifiable pattern for the groups that are stuck. During each execution some random groups got stuck. # The messages get processed when the consumer container is restarted and works fine for sometime before the issue recurs. # When we verified from the admin console, we could see that the number of consumers are going beyond the max value set for it. # We verified from the thread dumps that all the consumers are healthy and are in TIMED_WAITING state. Even though there are some messages stuck in the queue, they are not being delivered to the waiting consumers. # We have entry-exit logs for the producers and consumers. We could see the messages are being sent successfully by the producer and are never delivered to the consumer. Expected behaviour : All messages should be delivered to the consumers. Actual behaviour : Some of the messages belonging to random groups get stuck in the queue until the consumer container is restarted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)