Hi Folks, The Rackspace-HP team has been putting a lot of effort into performance testing event collection in the ceilometer storage drivers[0]. Based on some results of this testing, we would like to support batch consumption of notifications, as it will greatly improve insertion performance. Batch consumption in this case means waiting for a certain number of notifications to arrive before sending to the storage driver.
I¹d like to get feedback from the community about this feature, and how we are planning to implement it. Here is what I’m currently thinking: 1) This seems to fit well into oslo.messaging - batching may be a feature that other projects will find useful. After reviewing the changes that sileht has been working on in oslo.messaging, I think the right way to start off is to create a new executor that builds up a batch of notifications, and sends the batch to the dispatcher. We’d also add a timeout, so if a certain amount of time passes and the batch isn’t filled up, the notifications will be dispatched anyway. I’ve started a blueprint for this change and am filling in the details as I go along [1]. 2) In ceilometer, initialize the notification listener with the batch executor instead of the eventlet executor (this should probably be configurable)[2]. We can then send the entire batch of notifications to the storage driver to be processed as events, while maintaining the current method for converting notifications into samples. 3) Error handling becomes more difficult. The executor needs to know if any of the notifications should be requeued. I think the right way to solve this is to return a list of notifications to requeue from the handler. Any better ideas? Is this the right approach to take? I¹m not an oslo.messaging expert, so if there is a proper way to implement this change, I¹m all ears! Thanks, happy holidays! -john 0: https://etherpad.openstack.org/p/ceilometer-data-store-scale-testing 1: https://blueprints.launchpad.net/oslo.messaging/+spec/bulk-consume-messages 2: https://blueprints.launchpad.net/ceilometer/+spec/use-bulk-notification
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev