Re: Sender flow control via resource-limit-exceeded
On 12/17/2009 08:13 AM, Rafael Schloming wrote: Andrew Stitcher wrote: On Wed, 2009-12-16 at 09:56 -0500, Alan Conway wrote: I'm looking at some code that does sender flow control as follows: - set a queue size limit - sender catches 'resource-limit-exceeded' exception, sleeps for a bit, creates a new session & re-tries The essential problem with this approach is that the queue size is not directly caused by any single producer. So it's entirely possible in general for a producer that is producing at a low rate to be entirely killed by another producer sending messages into the same queue. The only way that this overall scheme can work fairly is if there is a 1-1 producer-queue mapping. If that is the case then it should work. Why wouldn't we use flow control as specified by the protocol? It doesn't help you figure out *who* to flow control, but once you've made that decision, it provides a much better way to actually flow control an individual producer. Agreed, I'm talking about what a user can do with the code as it stands today. We definitely need to implement some producer flow control. I'm still not sure how this would work. As you point out its tricky to give any exact guarantees since you have no idea which queues a client may publish to (at least in 0.10) You can bound some notion of "total messages in the system", but you need to figure out a) what should that bound be? It probably varies with number of queues, producers/consumers and queue limits but I don't know what the relationship would be. b) how to divide up the credit so fast producers with slow consumers can't hog the show. I don't think it's possible with the protocol flow control to enforce a depth limit on individual queues which is unfortunate. So retry after resource-limit-exceeded is still probably necessary for a robust producer. - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
Re: Sender flow control via resource-limit-exceeded
Andrew Stitcher wrote: On Wed, 2009-12-16 at 09:56 -0500, Alan Conway wrote: I'm looking at some code that does sender flow control as follows: - set a queue size limit - sender catches 'resource-limit-exceeded' exception, sleeps for a bit, creates a new session & re-tries The essential problem with this approach is that the queue size is not directly caused by any single producer. So it's entirely possible in general for a producer that is producing at a low rate to be entirely killed by another producer sending messages into the same queue. The only way that this overall scheme can work fairly is if there is a 1-1 producer-queue mapping. If that is the case then it should work. Why wouldn't we use flow control as specified by the protocol? It doesn't help you figure out *who* to flow control, but once you've made that decision, it provides a much better way to actually flow control an individual producer. --Rafael - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
Re: Sender flow control via resource-limit-exceeded
On Wed, 2009-12-16 at 09:56 -0500, Alan Conway wrote: > I'm looking at some code that does sender flow control as follows: > - set a queue size limit > - sender catches 'resource-limit-exceeded' exception, sleeps for a bit, > creates a new session & re-tries The essential problem with this approach is that the queue size is not directly caused by any single producer. So it's entirely possible in general for a producer that is producing at a low rate to be entirely killed by another producer sending messages into the same queue. The only way that this overall scheme can work fairly is if there is a 1-1 producer-queue mapping. If that is the case then it should work. Andrew - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
Sender flow control via resource-limit-exceeded
I'm looking at some code that does sender flow control as follows: - set a queue size limit - sender catches 'resource-limit-exceeded' exception, sleeps for a bit, creates a new session & re-tries Is there a better way to flow control senders to bound broker resources? I can't think of one as the broker & protocol currently stand. - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org