Re: [openstack-dev] [puppet] Moving forward with puppet-keystone CI (beaker tests)

2015-04-27 Thread Martin Mágr 


On 04/22/2015 08:03 PM, Emilien Macchi wrote:

* shell testing: yes because it's the way we wrote our providers.
We don't necessary need to shell out, but instead implement resources 
for Serverspec which will use openstackclient to test Keystone/Nova/... 
resources. I'm currently in process of merging Serverspec tests I made 
and will submit patches as soon as I will have something working. What I 
have currently is just Serverspec resource for testing INI file content, 
but openstackclient Serverspec resources are next on my TODO list.


Regards,
Martin

--

Martin Mágr

IRC nick: mmagr / para
Freenode channels: #openstack-dev, #packstack-dev, #puppet-openstack, #rdo, 
#rdo-puppet


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] Moving forward with puppet-keystone CI (beaker tests)

2015-04-22 Thread Adam Young 

On 04/22/2015 10:51 AM, Emilien Macchi wrote:

Hi,

Some important work is being done on Keystone v3 API support in
puppet-keystone.
We've clearly seen there is a lack of review and I think we all worry
about breaking something.
Spencer & I are working on beaker tests lately and the jobs are
non-voting for now.

I propose:
* to review (and eventually merge) the beaker-tests patches [1] [2] for
Keystone & openstacklib.
* to patch project-config [3] to make vote Beaker jobs in Puppet
OpenStack gate for puppet-keystone & puppet-openstacklib. Why voting?
Because otherwise I'm not sure people will notice the failure and some
patches will be merged while beaker is red.

So we can have a good set of tests that will help us to detect some
issues in the future.
I don't think we will catch all mistakes we can do, but this is a good
start.

To vote this proposal, you can use the gerrit patches and let any feedback.

Thanks,

[1] puppet-keystone: https://review.openstack.org/#/c/155873/
[2] puppet-openstacklib: https://review.openstack.org/#/c/176098/
[3] project-config: https://review.openstack.org/176343



I added Keystone-core (13 People) to each of these reviews.  If things 
are critical, and concert Keystone, we really should be notified so 
someone on the Keystone side can voice an opinion.





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] Moving forward with puppet-keystone CI (beaker tests)

2015-04-22 Thread Matthew Treinish 
On Wed, Apr 22, 2015 at 02:03:13PM -0400, Emilien Macchi wrote:
> 
> 
> On 04/22/2015 11:53 AM, Spencer Krum wrote:
> > Emillen,
> > 
> > Do you see shelling out to openstackclient in the keystone test to
> > verify that the keystone resources have been created? Do you see trying
> > to hit the api from something like aviator? Ultimately I'd like to see
> > us spin up an entire openstack in one test then hit it with tempest.
> 
> * shell testing: yes because it's the way we wrote our providers.
> * aviator: no because we gave up some months ago in favor of using
> openstackclient. I'm not in favor of using aviator which would add yet
> another dependency and complexity. Maybe I'm wrong though.
> * tempest: well... tempest aims to test API features while we only want
> to check Keystone is actually running. I think serverspec + some
> shelling could help. Having tempest is (to me) overkill and could slow
> down our CI if something's wrong in Tempest.

I kinda take issue with this comment, more likely than not when there are issues
with running tempest it's either a config or service problem. That's not to say
there aren't tempest bugs, I just don't think tempest breaking your CI should be
a  primary concern. (especially how often it's used for this exact purpose) If
it did break something it's not like fixing it is a complicated procedure. We've
been running tempest as the gating tests for some time so we've got some 
experience
fixing issues with fixing the test suite when things really go haywire. 

I actually think running tempest against against a deployment using the puppet
modules would probably be very useful. It'll let you test that everything comes
together and actually works. It's used for gating devstack commits to verify 
basically the same thing. I'm also interested in doing this because I think
it'll help us improve tempest by having direct feedback loop from running it
against a non-devstack environment. That's something I'd really like to have
because it's something I think we're missing right now.

If you need any help with setting something like this up, I'm definitely
available to give a hand with it.

> 
> > It may be possible to use a very narrow version of tempest to validate
> > just keystone.
> 
> Like, only a small set of tests that we would run with testr?

So this is the intent of the smoke tag in tempest, to provide a small set of
tests to do a quick sanity check that a deployment is working. Unfortunately
right now the tag doesn't do that because it got conflated as part of the
neutron testing bring up. We just need to spend some time sitting down 
and rationalizing the use of the tag to provide this. 

Also, if you want to just test service specific testing tempest already does
service tagging of tests. So you can run a subset by using the service name as
the filter. (ie identity for tests which talk to keystone or compute for nova
tests)

-Matt Treinish

> > On Wed, Apr 22, 2015 at 7:51 AM, Emilien Macchi  > > wrote:
> > 
> > Hi,
> > 
> > Some important work is being done on Keystone v3 API support in
> > puppet-keystone.
> > We've clearly seen there is a lack of review and I think we all worry
> > about breaking something.
> > Spencer & I are working on beaker tests lately and the jobs are
> > non-voting for now.
> > 
> > I propose:
> > * to review (and eventually merge) the beaker-tests patches [1] [2] for
> > Keystone & openstacklib.
> > * to patch project-config [3] to make vote Beaker jobs in Puppet
> > OpenStack gate for puppet-keystone & puppet-openstacklib. Why voting?
> > Because otherwise I'm not sure people will notice the failure and some
> > patches will be merged while beaker is red.
> > 
> > So we can have a good set of tests that will help us to detect some
> > issues in the future.
> > I don't think we will catch all mistakes we can do, but this is a good
> > start.
> > 
> > To vote this proposal, you can use the gerrit patches and let any
> > feedback.
> > 
> > Thanks,
> > 
> > [1] puppet-keystone: https://review.openstack.org/#/c/155873/
> > [2] puppet-openstacklib: https://review.openstack.org/#/c/176098/
> > [3] project-config: https://review.openstack.org/176343
> > --
> > Emilien Macchi
> > 
> > --
> > 
> > To unsubscribe from this group and stop receiving emails from it,
> > send an email to puppet-openstack+unsubscr...@puppetlabs.com
> > .
> > 
> > 
> > 
> > 
> > -- 
> > Spencer Krum
> > (619)-980-7820
> > 
> > -- 
> > 
> > To unsubscribe from this group and stop receiving emails from it, send
> > an email to puppet-openstack+unsubscr...@puppetlabs.com
> > .
> 
> -- 
> Emilien Macchi
> 



> __
> OpenStack Development Mailing List (n

Re: [openstack-dev] [puppet] Moving forward with puppet-keystone CI (beaker tests)

2015-04-22 Thread Emilien Macchi 


On 04/22/2015 11:53 AM, Spencer Krum wrote:
> Emillen,
> 
> Do you see shelling out to openstackclient in the keystone test to
> verify that the keystone resources have been created? Do you see trying
> to hit the api from something like aviator? Ultimately I'd like to see
> us spin up an entire openstack in one test then hit it with tempest.

* shell testing: yes because it's the way we wrote our providers.
* aviator: no because we gave up some months ago in favor of using
openstackclient. I'm not in favor of using aviator which would add yet
another dependency and complexity. Maybe I'm wrong though.
* tempest: well... tempest aims to test API features while we only want
to check Keystone is actually running. I think serverspec + some
shelling could help. Having tempest is (to me) overkill and could slow
down our CI if something's wrong in Tempest.

> It may be possible to use a very narrow version of tempest to validate
> just keystone.

Like, only a small set of tests that we would run with testr?

> 
> Thanks,
> Spencer
> 
> On Wed, Apr 22, 2015 at 7:51 AM, Emilien Macchi  > wrote:
> 
> Hi,
> 
> Some important work is being done on Keystone v3 API support in
> puppet-keystone.
> We've clearly seen there is a lack of review and I think we all worry
> about breaking something.
> Spencer & I are working on beaker tests lately and the jobs are
> non-voting for now.
> 
> I propose:
> * to review (and eventually merge) the beaker-tests patches [1] [2] for
> Keystone & openstacklib.
> * to patch project-config [3] to make vote Beaker jobs in Puppet
> OpenStack gate for puppet-keystone & puppet-openstacklib. Why voting?
> Because otherwise I'm not sure people will notice the failure and some
> patches will be merged while beaker is red.
> 
> So we can have a good set of tests that will help us to detect some
> issues in the future.
> I don't think we will catch all mistakes we can do, but this is a good
> start.
> 
> To vote this proposal, you can use the gerrit patches and let any
> feedback.
> 
> Thanks,
> 
> [1] puppet-keystone: https://review.openstack.org/#/c/155873/
> [2] puppet-openstacklib: https://review.openstack.org/#/c/176098/
> [3] project-config: https://review.openstack.org/176343
> --
> Emilien Macchi
> 
> --
> 
> To unsubscribe from this group and stop receiving emails from it,
> send an email to puppet-openstack+unsubscr...@puppetlabs.com
> .
> 
> 
> 
> 
> -- 
> Spencer Krum
> (619)-980-7820
> 
> -- 
> 
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-openstack+unsubscr...@puppetlabs.com
> .

-- 
Emilien Macchi



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] Moving forward with puppet-keystone CI (beaker tests)

2015-04-22 Thread Spencer Krum 
Emillen,

Do you see shelling out to openstackclient in the keystone test to verify
that the keystone resources have been created? Do you see trying to hit the
api from something like aviator? Ultimately I'd like to see us spin up an
entire openstack in one test then hit it with tempest.

It may be possible to use a very narrow version of tempest to validate just
keystone.

Thanks,
Spencer

On Wed, Apr 22, 2015 at 7:51 AM, Emilien Macchi  wrote:

> Hi,
>
> Some important work is being done on Keystone v3 API support in
> puppet-keystone.
> We've clearly seen there is a lack of review and I think we all worry
> about breaking something.
> Spencer & I are working on beaker tests lately and the jobs are
> non-voting for now.
>
> I propose:
> * to review (and eventually merge) the beaker-tests patches [1] [2] for
> Keystone & openstacklib.
> * to patch project-config [3] to make vote Beaker jobs in Puppet
> OpenStack gate for puppet-keystone & puppet-openstacklib. Why voting?
> Because otherwise I'm not sure people will notice the failure and some
> patches will be merged while beaker is red.
>
> So we can have a good set of tests that will help us to detect some
> issues in the future.
> I don't think we will catch all mistakes we can do, but this is a good
> start.
>
> To vote this proposal, you can use the gerrit patches and let any feedback.
>
> Thanks,
>
> [1] puppet-keystone: https://review.openstack.org/#/c/155873/
> [2] puppet-openstacklib: https://review.openstack.org/#/c/176098/
> [3] project-config: https://review.openstack.org/176343
> --
> Emilien Macchi
>
> --
>
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-openstack+unsubscr...@puppetlabs.com.
>



-- 
Spencer Krum
(619)-980-7820
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [puppet] Moving forward with puppet-keystone CI (beaker tests)

2015-04-22 Thread Emilien Macchi 
Hi,

Some important work is being done on Keystone v3 API support in
puppet-keystone.
We've clearly seen there is a lack of review and I think we all worry
about breaking something.
Spencer & I are working on beaker tests lately and the jobs are
non-voting for now.

I propose:
* to review (and eventually merge) the beaker-tests patches [1] [2] for
Keystone & openstacklib.
* to patch project-config [3] to make vote Beaker jobs in Puppet
OpenStack gate for puppet-keystone & puppet-openstacklib. Why voting?
Because otherwise I'm not sure people will notice the failure and some
patches will be merged while beaker is red.

So we can have a good set of tests that will help us to detect some
issues in the future.
I don't think we will catch all mistakes we can do, but this is a good
start.

To vote this proposal, you can use the gerrit patches and let any feedback.

Thanks,

[1] puppet-keystone: https://review.openstack.org/#/c/155873/
[2] puppet-openstacklib: https://review.openstack.org/#/c/176098/
[3] project-config: https://review.openstack.org/176343
-- 
Emilien Macchi



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev