Avoid blocking producers
------------------------
Key: AMQ-688
URL: https://issues.apache.org/activemq/browse/AMQ-688
Project: ActiveMQ
Type: New Feature
Components: Broker
Versions: 4.0 RC 2
Reporter: Christopher A. Larrieu
Fix For: incubation
Our main goal
is to avoid stalled producers by addressing the main culprit: too many
undispatched messages
in the broker's memory. Our motivation is to handle significant --though
temporary-- imbalances
between production and consumption rates.
Reaching this goal entails specific broker modifications:
1. When memory gets tight, start dropping undispatched non-persistent messages.
This is the
first-cut attempt to maintain throughput of persistent messages.
Unlike the approach documented at
http://docs.codehaus.org/display/ACTIVEMQ/Slow+Consumer+Handling,
the message dropping will only occur after the UsageManager reaches capacity.
Non-persistent
messages in dispatch lists will be dropped according to per-destination policy.
Subscriptions
can purge their own messages triggered via callback from the UsageManager.
2. Evict messages if memory remains tight, to be fetched from backing store
prior to dispatch.
ActiveMQ already supports this for persistent messages on Topics with durable
subscriptions.
If a consumer's prefetch buffer is full, the splash-over messages remain as
IndirectMessageReference
objects in the dispatch list, with the actual message body loaded from store on
demand. I
believe we can extend this approach for Queues as well.
3. Improve the efficiency with which evicted messages are loaded back into
memory.
Currently, they are loaded one at a time as needed. It would make sense to
batch-load message
sets periodically. This will require a significant shift in responsibilities
between objects,
since an IndirectMessageReference doesn't know about other instances that can
be loaded in
mass.
The goal will be to keep each subscription dispatch list stocked with a decent
number of messages
in-memory to reasonably trade-off between it's consumer's performance and
resource usage in
the broker. As with everything else, we can implement this as a strategy class
with the first
cut implementing a simple resource allocation strategy: divvy up available
memory amongst
all subscriptions and keep that memory filled with messages for dispatch. I
envision a worker
task assuming responsibility for keeping these lists filled.
4. Even with the above modifications, we still can't entirely avoid blocked
producers, so
we'd like to add client-configurable time-outs to provide a bound for the time
a producer
can remain stalled.
Maybe this should be a new attribute of ActiveMQConnection:
maxProducerFlowControlWait. Calls
to UsageManager.waitForSpace can take this quantity as an argument. Failure to
reach sufficient
space for the new message will throw an exception back up the stack and across
the wire, letting
the producer know that the message was not delivered.
5. Finally, we need to extend disk support for Topics that have only
non-durable subscribers,
otherwise their dispatch lists can fill up with persistent messages. In order
to maintain
compliance with JMS, it would be nice to provide some alternative to dropping
persistent messages.
One possible first cut is to layer this on top of a DurableTopicSubscription by
creating an
anonymous subscriber for every Topic that has only non-durable subscriptions.
When all such
subscriptions terminate, the broker can remove the anonymous subscriber.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
