On 12/20/2013 05:27 PM, Herndon, John Luke wrote:
On Dec 20, 2013, at 8:48 AM, Julien Danjou <jul...@danjou.info>
wrote:
Anyway, my main concern here is that I am not very enthusiast
about using the executor to do that. I wonder if there is not a way
to ask the broker to get as many as message as it has up to a
limit?
You would have 100 messages waiting in the notifications.info
queue, and you would be able to tell to oslo.messaging that you
want to read up to 10 messages at a time. If the underlying
protocol (e.g. AMQP) can support that too, it would be more
efficient too.
Yeah, I like this idea. As far as I can tell, AMQP doesn’t support
grabbing more than a single message at a time, but we could
definitely have the broker store up the batch before sending it
along.
AMQP (in all it's versions) allows for a subscription with a
configurable amount of 'prefetch', which means the broker can send lots
of messages without waiting for the client to request them one at a time.
That's not quite the same as the batching I think you are looking for,
but it does allow the broker to do its own batching. My guess is the
rabbit driver is already using basic.consume rather than basic.get
anyway(?), so the broker is free to batch as it sees fit. (I haven't
actually dug into the kombu code to verify that however, perhaps someone
else here can confirm?)
However you still need the client to have some way of batching up the
messages and then processing them together.
Other protocols may support bulk consumption. My one concern
with this approach is error handling. Currently the executors treat
each notification individually. So let’s say the broker hands 100
messages at a time. When client is done processing the messages, the
broker needs to know if message 25 had an error or not. We would
somehow need to communicate back to the broker which messages failed.
I think this may take some refactoring of executors/dispatchers. What
do you think?
I've have some related questions, that I haven't yet satisfactorily
answered yet. The extra context here may be useful in doing so.
(1) What are the expectations around message delivery guarantees for
insertion into a store? I.e. if there is a failure, is it ok to get
duplicate entries for notifications? (I'm assuming losing notifications
is not acceptable).
(2) What would you want the broker to do with the failed messages? What
sort of things might fail? Is it related to the message content itself?
Or is it failures suspected to be of a temporal nature?
(3) How important is ordering ? If a failure causes some notifications
to be inserted out of order is that a problem at all?
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
