Hi all, I finally figured out what was happening. Here are my conclusions, for future reference.
In a nutshell: * Every time a message expires on the consumer, the prefetch extension kicks in and allows even more messages to be dispatched. Because the session is not closed, the extended prefetch limit does not decrease. New expirations will continue to increase it. * In-flight message size metrics were introduced some time ago. The current implementation requires all in-flight messages to be kept in memory. So with the prefetch extension growing uncontrollably, more and more messages are retained in memory. This causes heap exhaustion. * Messages expired on the broker were counted towards the dispatched count. This was being hidden by the prefetch extension. When finally it became possible to disable the prefetch extension, consumers stopped receiving messages after a while. This resulted in two ActiveMQ code changes: Expired messages counting as Dispatched on TopicSubscription https://issues.apache.org/jira/browse/AMQ-6579 TopicSubscription does not honor usePrefetchExtension https://issues.apache.org/jira/browse/AMQ-6577 Once it became possible to disable prefetch extension for topic subscriptions and the message counting bug was fixed, the broker became stable. The only thing I do not understand is the purpose of never decreasing the prefetch extension value while the session is active... Regards, Vasco Veloso -----Original Message----- From: Veloso, Vasco (Coriant - PT/Lisbon) [mailto:vasco.vel...@coriant.com] Sent: sexta-feira, 20 de janeiro de 2017 14:03 To: users@activemq.apache.org Subject: RE: Expired non-persistent messages on topic appear to not be discarded All consumers continue to receive messages. My first and biggest question is whether this JMX counter evolution makes sense? Is there any possibility of expired messages counting towards in flight messages? I am worried that internal data structures may not be released, which will lead either to an OOM condition or to the JVM experiencing full GCs continuously should the situation continue for a long time. In fact this has happened during my tests. Heap usage keeps increasing over several hours, it is then exhausted and the JVM starts to run full GC cycles continuously. Vasco -----Original Message----- From: tbai...@gmail.com [mailto:tbai...@gmail.com] On Behalf Of Tim Bain Sent: sexta-feira, 20 de janeiro de 2017 13:49 To: ActiveMQ Users <users@activemq.apache.org> Subject: Re: Expired non-persistent messages on topic appear to not be discarded Are your slow consumers still receiving messages properly when this happens? Are there any indications that this is a "real" problem rather than just a bug in the JMX counters? Tim On Jan 19, 2017 8:38 AM, "Veloso, Vasco (Coriant - PT/Lisbon)" < vasco.vel...@coriant.com> wrote: > Hello, > > > > I’m using ActiveMQ 5.14.3 to test and try to find a suitable > configuration for a scenario where we have multiple producers and > multiple consumers of a topic, non-durable, in which some consumers > may be quite slow. Also, some messages have TTL defined and others do not. > > > > By looking at the JMX counters for the topic, InFlightCount remains > stable at the prefetch value for consumers. However once messages > begin to expire, it looks like they are not discarded: > > > > · ExpiredCount starts to increase as messages time out; > > · InFlightCount starts to increase at the same rate as > ExpiredCount. > > > > JMX counters for slow consumers follow the same pattern. They remain > stable until messages begin to expire: MessageCountAwaitingAcknowledge > remains near PrefetchSize. When messages expire then > MessageCountAwaitingAcknowledge starts to increase as well. > > > > These counters will never decrease or stabilize as long as there are > messages being produced. > > > > I tried many settings but I still don’t understand what is going on. > It appears as if expired messages are forwarded to consumers or expire > on consumers? However for this test the broker, producers and > consumers are on the same machine so there is no clock drift. And if > messages would expire on consumers, wouldn’t they be dropped from the topic > as well? > > > > My broker configuration and test producer/consumer code is attached > for your reference. > > > > I hope that someone can shed some light on this matter, since I can’t > make heads or tails of it at this time. > > > > Thanks! > > > > Vasco Veloso > > >
