On Wed, 28 Oct 2015, Emilien Macchi wrote:
As a user[1], I would like to functionally test OpenStack services.
Bless you. Let's all be more like that.
Until now I was happy, until this bug [4] (TL;DR: Aodh can't be testing with Tempest which is a bug I'm working on, and not really related to this thread). I realized Aodh [5] (and apparently some other projects like Ceilometer) were using something else (gabbi [6]) for testing.
There's a fair amount of history here for which I don't know all the details so I'll just try to address the gabbi related questions below after this initial comment: aodh is not yet in tempest, but that's simply because it is not yet done, not because there are not plans to do it. Tomorrow morning at 9am in the Kiri room we're having a design session about functional and integration testing in all three of ceilometer, aodh and gnocchi and one of the primary topics is: getting all three to have tempest plugins. One of the other primary topics is making new and existing in-tree functional test as good and useful as possible. A lot of those tests are not tempest because they've been extracted from pre-existing so-called unit tests but were determined to not be because they use a database. A small segment are API tests driven by gabbi as a result of this spec[1] If you or anyone else have time to show up to that session, that would be great.
How come some big tent projects do not use Tempest anymore for functional testing? I thought there was/is a move with tempest plugins that will help projects to host their tempest tests in their repos.
Ceilometer projects started a migration to more in-tree functional tests in advance of tempest-lib existing. These manifest as tests that require a storage backend and have their own non-tempest job description in project-config.
Am I missing something? Any official decision taken? Is gabbi supported by OpenStack?
When I first released gabbi there was talk about it becoming either part of QA or oslo but after showing it around the world a bit I had feedback from several non-openstack people that they would be _far_ more likely to contribute to it if it was not subject to the openstack development model[0]. So it lives now as a project on github to which people submit pull requests and travis takes care of CI. To avoid bus termination errors, there are two other admins in addition to me who have all the same rights with regard to merging and releasing and maintaining on python. One of them is another OpenStack contributor (jasonamyers). I'm obviously biased, but I think gabbi is teh ossum and makes writing API tests easy and perhaps more importantly makes reading them later fantastic. Within gnocchi and aodh, gabbi has been so successful at making it easy to write API tests that it is now being used for writing integration tests of aodh+gnocchi+ceilometer+heat.
I feel like there is currently 2 paths that try to do the same thing and as a user, I'm not happy.
Yeah, that's perhaps a problem. One thing I'm hoping to explore (or hoping someone else will explore) is making it easy to do gabbi tests within a tempest plugin. This ought to be possible but there are some humps to deal with regard to how gabbi orders and groups tests. Again, I'm biased, but I think that gabbi would be a _huge_ asset for API tests in tempest because by its very nature it is very close to HTTP without a notion of a (python) client being involved. I think this is an excellent guard for ensuring that OpenStack APIs are sufficiently agnostic about their context.
Please help me to understand,
I hope that adds a bit more info. I know it doesn't actually answer the real question though. I hope some other voices will come along. [0] I'm happy to discuss this elsewhere (in person, on another thread, whatever) but it would be bad to let it distract this current thread. [1] http://specs.openstack.org/openstack/ceilometer-specs/specs/kilo/declarative-http-tests.html -- Chris Dent tw:@anticdent freenode:cdent https://tank.peermore.com/tanks/cdent __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev