On Wed, Apr 22, 2015 at 02:03:13PM -0400, Emilien Macchi wrote: > > > On 04/22/2015 11:53 AM, Spencer Krum wrote: > > Emillen, > > > > Do you see shelling out to openstackclient in the keystone test to > > verify that the keystone resources have been created? Do you see trying > > to hit the api from something like aviator? Ultimately I'd like to see > > us spin up an entire openstack in one test then hit it with tempest. > > * shell testing: yes because it's the way we wrote our providers. > * aviator: no because we gave up some months ago in favor of using > openstackclient. I'm not in favor of using aviator which would add yet > another dependency and complexity. Maybe I'm wrong though. > * tempest: well... tempest aims to test API features while we only want > to check Keystone is actually running. I think serverspec + some > shelling could help. Having tempest is (to me) overkill and could slow > down our CI if something's wrong in Tempest.
I kinda take issue with this comment, more likely than not when there are issues with running tempest it's either a config or service problem. That's not to say there aren't tempest bugs, I just don't think tempest breaking your CI should be a primary concern. (especially how often it's used for this exact purpose) If it did break something it's not like fixing it is a complicated procedure. We've been running tempest as the gating tests for some time so we've got some experience fixing issues with fixing the test suite when things really go haywire. I actually think running tempest against against a deployment using the puppet modules would probably be very useful. It'll let you test that everything comes together and actually works. It's used for gating devstack commits to verify basically the same thing. I'm also interested in doing this because I think it'll help us improve tempest by having direct feedback loop from running it against a non-devstack environment. That's something I'd really like to have because it's something I think we're missing right now. If you need any help with setting something like this up, I'm definitely available to give a hand with it. > > > It may be possible to use a very narrow version of tempest to validate > > just keystone. > > Like, only a small set of tests that we would run with testr? So this is the intent of the smoke tag in tempest, to provide a small set of tests to do a quick sanity check that a deployment is working. Unfortunately right now the tag doesn't do that because it got conflated as part of the neutron testing bring up. We just need to spend some time sitting down and rationalizing the use of the tag to provide this. Also, if you want to just test service specific testing tempest already does service tagging of tests. So you can run a subset by using the service name as the filter. (ie identity for tests which talk to keystone or compute for nova tests) -Matt Treinish > > On Wed, Apr 22, 2015 at 7:51 AM, Emilien Macchi <emil...@redhat.com > > <mailto:emil...@redhat.com>> wrote: > > > > Hi, > > > > Some important work is being done on Keystone v3 API support in > > puppet-keystone. > > We've clearly seen there is a lack of review and I think we all worry > > about breaking something. > > Spencer & I are working on beaker tests lately and the jobs are > > non-voting for now. > > > > I propose: > > * to review (and eventually merge) the beaker-tests patches [1] [2] for > > Keystone & openstacklib. > > * to patch project-config [3] to make vote Beaker jobs in Puppet > > OpenStack gate for puppet-keystone & puppet-openstacklib. Why voting? > > Because otherwise I'm not sure people will notice the failure and some > > patches will be merged while beaker is red. > > > > So we can have a good set of tests that will help us to detect some > > issues in the future. > > I don't think we will catch all mistakes we can do, but this is a good > > start. > > > > To vote this proposal, you can use the gerrit patches and let any > > feedback. > > > > Thanks, > > > > [1] puppet-keystone: https://review.openstack.org/#/c/155873/ > > [2] puppet-openstacklib: https://review.openstack.org/#/c/176098/ > > [3] project-config: https://review.openstack.org/176343 > > -- > > Emilien Macchi > > > > -- > > > > To unsubscribe from this group and stop receiving emails from it, > > send an email to puppet-openstack+unsubscr...@puppetlabs.com > > <mailto:puppet-openstack%2bunsubscr...@puppetlabs.com>. > > > > > > > > > > -- > > Spencer Krum > > (619)-980-7820 > > > > -- > > > > To unsubscribe from this group and stop receiving emails from it, send > > an email to puppet-openstack+unsubscr...@puppetlabs.com > > <mailto:puppet-openstack+unsubscr...@puppetlabs.com>. > > -- > Emilien Macchi > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
pgpedLSQPCQQo.pgp
Description: PGP signature
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev