Hi Nadya, Yep, that’s right, the notifications stick around on the server until they are acknowledged so there is extra overhead involved. I only have experience with rabbitmq, so I can’t speak for other transports, but we have used this strategy internally for other purposes, and have reached > 10k messages/second on a single consumer using batch message consumption (i.e., consume N messages, process them, then ack all N at once). We’ve found that being able to acknowledge the entire batch of messages at a time leads to a huge performance increase. This is another motivating factor for moving towards batches. But to your point, making this configurable is the right way to go just in case other transports don’t react as well.
Thanks, -john From: Nadya Privalova <[email protected]> Reply-To: "OpenStack Development Mailing List (not for usage questions)" <[email protected]> Date: Fri, 20 Dec 2013 15:25:55 +0400 To: "OpenStack Development Mailing List (not for usage questions)" <[email protected]> Subject: Re: [openstack-dev] [Ceilometer][Oslo] Consuming Notifications in Batches Hi John, As for me your ideas look very interesting. As I understood notification messages will be kept in MQ for some time (during batch-basket is being filled), right? I'm concerned about the additional load that will be on MQ (Rabbit). Thanks, Nadya On Fri, Dec 20, 2013 at 3:31 AM, Herndon, John Luke <[email protected]> wrote: > Hi Folks, > > The Rackspace-HP team has been putting a lot of effort into performance > testing event collection in the ceilometer storage drivers[0]. Based on > some results of this testing, we would like to support batch consumption > of notifications, as it will greatly improve insertion performance. Batch > consumption in this case means waiting for a certain number of > notifications to arrive before sending to the storage > driver. > > I¹d like to get feedback from the community about this feature, and how we > are planning to implement it. Here is what I’m currently thinking: > > 1) This seems to fit well into oslo.messaging - batching may be a feature > that other projects will find useful. After reviewing the changes that > sileht has been working on in oslo.messaging, I think the right way to > start off is to create a new executor that builds up a batch of > notifications, and sends the batch to the dispatcher. We’d also add a > timeout, so if a certain amount of time passes and the batch isn’t filled > up, the notifications will be dispatched anyway. I’ve started a > blueprint for this change and am filling in the details as I go along [1]. > > 2) In ceilometer, initialize the notification listener with the batch > executor instead of the eventlet executor (this should probably be > configurable)[2]. We can then send the entire batch of notifications to > the storage driver to be processed as events, while maintaining the > current method for converting notifications into samples. > > 3) Error handling becomes more difficult. The executor needs to know if > any of the notifications should be requeued. I think the right way to > solve this is to return a list of notifications to requeue from the > handler. Any better ideas? > > Is this the right approach to take? I¹m not an oslo.messaging expert, so > if there is a proper way to implement this change, I¹m all ears! > > Thanks, happy holidays! > -john > > 0: https://etherpad.openstack.org/p/ceilometer-data-store-scale-testing > 1: > https://blueprints.launchpad.net/oslo.messaging/+spec/bulk-consume-messages > 2: https://blueprints.launchpad.net/ceilometer/+spec/use-bulk-notification > > _______________________________________________ > OpenStack-dev mailing list > [email protected] > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > _______________________________________________ OpenStack-dev mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ OpenStack-dev mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
