Rob, Thanks for the heads up. This is a matter of significant interest for us. I'm going to be out until mid next week, but I'd like to hear some more about your plans, so that we can better understand how we can most effectively work together towards this common goal. I'll ping you again next week.
Cheers, Chris > -----Original Message----- > From: Rob Davies (JIRA) [mailto:[EMAIL PROTECTED] > Sent: Thursday, April 13, 2006 3:35 PM > To: activemq-dev@geronimo.apache.org > Subject: [jira] Commented: (AMQ-688) Avoid blocking producers > > [ > https://issues.apache.org/activemq/browse/AMQ-688?page=comment s#action_36051 ] > > Rob Davies commented on AMQ-688: > -------------------------------- > > We are investigating changing the model for persisting > messages for 4.1 - at the moment there is a tight coupling > between producers and consumers and we want to de-couple this > - particularly for the persistent messages. The aim is to > have a staged eventing, where messages will be pulled from > storage to fill to a staging area before being dispatched. > This will have an impact on this particular issue, in that > the dispatch models will be separated for persistent and > non-persistent (including non-durable topic subscribers of > persistent messages). - Just wanted to give you a heads up :) > > > Avoid blocking producers > > ------------------------ > > > > Key: AMQ-688 > > URL: https://issues.apache.org/activemq/browse/AMQ-688 > > Project: ActiveMQ > > Type: New Feature > > > Components: Broker > > Versions: 4.0 RC 2 > > Reporter: Christopher A. Larrieu > > Assignee: Christopher A. Larrieu > > Fix For: incubation > > > > > Original Estimate: 8 weeks > > Remaining: 8 weeks > > > > Our main goal > > is to avoid stalled producers by addressing the main > culprit: too many > > undispatched messages in the broker's memory. Our motivation is to > > handle significant --though temporary-- imbalances between > production and consumption rates. > > Reaching this goal entails specific broker modifications: > > 1. When memory gets tight, start dropping undispatched > non-persistent > > messages. This is the first-cut attempt to maintain > throughput of persistent messages. > > Unlike the approach documented at > > http://docs.codehaus.org/display/ACTIVEMQ/Slow+Consumer+Handling, > > the message dropping will only occur after the UsageManager reaches > > capacity. Non-persistent messages in dispatch lists will > be dropped > > according to per-destination policy. Subscriptions can > purge their own messages triggered via callback from the UsageManager. > > 2. Evict messages if memory remains tight, to be fetched > from backing store prior to dispatch. > > ActiveMQ already supports this for persistent messages on > Topics with durable subscriptions. > > If a consumer's prefetch buffer is full, the splash-over messages > > remain as IndirectMessageReference objects in the dispatch > list, with > > the actual message body loaded from store on demand. I > believe we can extend this approach for Queues as well. > > 3. Improve the efficiency with which evicted messages are > loaded back into memory. > > Currently, they are loaded one at a time as needed. It would make > > sense to batch-load message sets periodically. This will require a > > significant shift in responsibilities between objects, since an > > IndirectMessageReference doesn't know about other instances > that can be loaded in mass. > > > > The goal will be to keep each subscription dispatch list > stocked with > > a decent number of messages in-memory to reasonably > trade-off between > > it's consumer's performance and resource usage in the > broker. As with > > everything else, we can implement this as a strategy class with the > > first cut implementing a simple resource allocation > strategy: divvy up > > available memory amongst all subscriptions and keep that > memory filled with messages for dispatch. I envision a > worker task assuming responsibility for keeping these lists filled. > > 4. Even with the above modifications, we still can't entirely avoid > > blocked producers, so we'd like to add client-configurable > time-outs > > to provide a bound for the time a producer can remain stalled. > > Maybe this should be a new attribute of ActiveMQConnection: > > maxProducerFlowControlWait. Calls to UsageManager.waitForSpace can > > take this quantity as an argument. Failure to reach > sufficient space > > for the new message will throw an exception back up the > stack and across the wire, letting the producer know that the > message was not delivered. > > 5. Finally, we need to extend disk support for Topics that > have only > > non-durable subscribers, otherwise their dispatch lists can fill up > > with persistent messages. In order to maintain compliance > with JMS, it would be nice to provide some alternative to > dropping persistent messages. > > One possible first cut is to layer this on top of a > > DurableTopicSubscription by creating an anonymous > subscriber for every > > Topic that has only non-durable subscriptions. When all > such subscriptions terminate, the broker can remove the > anonymous subscriber. > > -- > This message is automatically generated by JIRA. > - > If you think it was sent incorrectly contact one of the > administrators: > https://issues.apache.org/activemq/secure/Administrators.jspa > - > For more information on JIRA, see: > http://www.atlassian.com/software/jira >
