Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-27 Thread Christopher Yeoh
On Wed, 26 Mar 2014 21:02:58 -0700 Maru Newby wrote: > > > > If an API is made and documented properly what domain expertise > > would be needed to use it? The opposite is true for tempest and the > > tests themselves. The tempest team focuses on just tests so they > > know how to write good test

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-27 Thread Joe Gordon
On Wed, Mar 26, 2014 at 9:02 PM, Maru Newby wrote: > > On Mar 26, 2014, at 12:59 PM, Joe Gordon wrote: > > > > > > > > > On Tue, Mar 25, 2014 at 1:49 AM, Maru Newby wrote: > > > > On Mar 21, 2014, at 9:01 AM, David Kranz wrote: > > > > > On 03/20/2014 04:19 PM, Rochelle.RochelleGrober wrote: >

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-27 Thread Malini Kamalambal
> >The beginning of this thread largely came from the fact that Marconi >clearly doing most of their QA not in an upstream way. To be integrated, >that needs to change. Marconi has very good test coverage within the project. These tests guarantee functionality at a project level (i.e the API wor

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-26 Thread Maru Newby
On Mar 26, 2014, at 1:52 PM, Sean Dague wrote: > On 03/25/2014 04:49 AM, Maru Newby wrote: >> >> On Mar 21, 2014, at 9:01 AM, David Kranz wrote: >> >>> On 03/20/2014 04:19 PM, Rochelle.RochelleGrober wrote: > -Original Message- > From: Malini Kamalambal [mailto:malini.ka

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-26 Thread Maru Newby
On Mar 26, 2014, at 12:59 PM, Joe Gordon wrote: > > > > On Tue, Mar 25, 2014 at 1:49 AM, Maru Newby wrote: > > On Mar 21, 2014, at 9:01 AM, David Kranz wrote: > > > On 03/20/2014 04:19 PM, Rochelle.RochelleGrober wrote: > >> > >>> -Original Message- > >>> From: Malini Kamalambal [

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-26 Thread Sean Dague
On 03/25/2014 04:49 AM, Maru Newby wrote: > > On Mar 21, 2014, at 9:01 AM, David Kranz wrote: > >> On 03/20/2014 04:19 PM, Rochelle.RochelleGrober wrote: >>> -Original Message- From: Malini Kamalambal [mailto:malini.kamalam...@rackspace.com] Sent: Thursday, March 20, 2014

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-26 Thread Joe Gordon
On Tue, Mar 25, 2014 at 1:49 AM, Maru Newby wrote: > > On Mar 21, 2014, at 9:01 AM, David Kranz wrote: > > > On 03/20/2014 04:19 PM, Rochelle.RochelleGrober wrote: > >> > >>> -Original Message- > >>> From: Malini Kamalambal [mailto:malini.kamalam...@rackspace.com] > >>> Sent: Thursday, M

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-26 Thread Maglana, Mark
This is really interesting discussion but was thrown off by the different use of ‘functional testing.’ I decided to reconcile it with my understanding and ended up with this two-pager. Sharing it in case it helps: https://docs.google.com/document/d/1ILxfoJlov9lBfuuZtvwW_7bmlGaYYw0UsE1R9lo6XwQ/ed

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-25 Thread Malini Kamalambal
We are talking about different levels of testing, 1. Unit tests - which everybody agrees should be in the individual project itself 2. System Tests - 'System' referring to (& limited to), all the components that make up the project. These are also the functional tests for the project. 3. Integr

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-25 Thread Maru Newby
On Mar 21, 2014, at 9:01 AM, David Kranz wrote: > On 03/20/2014 04:19 PM, Rochelle.RochelleGrober wrote: >> >>> -Original Message- >>> From: Malini Kamalambal [mailto:malini.kamalam...@rackspace.com] >>> Sent: Thursday, March 20, 2014 12:13 PM >>> >>> 'project specific functional testi

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-24 Thread Joe Gordon
On Fri, Mar 21, 2014 at 9:38 AM, Malini Kamalambal < malini.kamalam...@rackspace.com> wrote: > > > On 3/21/14 12:01 PM, "David Kranz" wrote: > > >On 03/20/2014 04:19 PM, Rochelle.RochelleGrober wrote: > >> > >>> -Original Message- > >>> From: Malini Kamalambal [mailto:malini.kamalam...@ra

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-24 Thread Malini Kamalambal
On 3/21/14 3:49 PM, "Rochelle.RochelleGrober" wrote: > >> From: Malini Kamalambal [mailto:malini.kamalam...@rackspace.com] > >> >> We are talking about different levels of testing, >> >> 1. Unit tests - which everybody agrees should be in the individual >> project >> itself >> 2. System Tests

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-21 Thread Rochelle.RochelleGrober
> From: Malini Kamalambal [mailto:malini.kamalam...@rackspace.com] > > We are talking about different levels of testing, > > 1. Unit tests - which everybody agrees should be in the individual > project > itself > 2. System Tests - 'System' referring to (& limited to), all the > components > tha

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-21 Thread Malini Kamalambal
On 3/21/14 12:01 PM, "David Kranz" wrote: >On 03/20/2014 04:19 PM, Rochelle.RochelleGrober wrote: >> >>> -Original Message- >>> From: Malini Kamalambal [mailto:malini.kamalam...@rackspace.com] >>> Sent: Thursday, March 20, 2014 12:13 PM >>> >>> 'project specific functional testing' in t

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-21 Thread David Kranz
On 03/20/2014 04:19 PM, Rochelle.RochelleGrober wrote: -Original Message- From: Malini Kamalambal [mailto:malini.kamalam...@rackspace.com] Sent: Thursday, March 20, 2014 12:13 PM 'project specific functional testing' in the Marconi context is treating Marconi as a complete system, maki

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-21 Thread Malini Kamalambal
I have an etherpad started to document QA requirements https://etherpad.openstack.org/p/Tempest-Graduation-Criteria I hope Sean and the rest of QA team can add their thoughts here. I am also looking for inputs from the Sahara team, while the path to graduation is still fresh in their minds. We can

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-20 Thread Joe Gordon
On Thu, Mar 20, 2014 at 1:19 PM, Rochelle.RochelleGrober < rochelle.gro...@huawei.com> wrote: > > > > -Original Message- > > From: Malini Kamalambal [mailto:malini.kamalam...@rackspace.com] > > Sent: Thursday, March 20, 2014 12:13 PM > > > > 'project specific functional testing' in the Mar

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-20 Thread Rochelle.RochelleGrober
> -Original Message- > From: Malini Kamalambal [mailto:malini.kamalam...@rackspace.com] > Sent: Thursday, March 20, 2014 12:13 PM > > 'project specific functional testing' in the Marconi context is > treating > Marconi as a complete system, making Marconi API calls & verifying the > resp

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-20 Thread Sean Dague
I will agree that the TC language is not as strong as it should be (and really should be clarified, but I don't think that's going to happen until the release is looking solid). Honestly, though, I think Sahara is a good example here of the level of that we expect. They have actively engaged with u

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-20 Thread Malini Kamalambal
Thanks Matt for your response !! It has clarified some of the 'cloudy areas' ;) "So having only looked at the Marconi ML thread and not the actual TC meeting minutes I might be missing the whole picture. But, from what I saw when I looked at both a marconi commit and a tempest commit is that ther

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-20 Thread Matthew Treinish
On Thu, Mar 20, 2014 at 11:35:15AM +, Malini Kamalambal wrote: > Hello all, > > I have been working on adding tests in Tempest for Marconi, for the last few > months. > While there are many amazing people to work with, the process has been more > difficult than I expected. > > Couple of pai

[openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-20 Thread Malini Kamalambal
Hello all, I have been working on adding tests in Tempest for Marconi, for the last few months. While there are many amazing people to work with, the process has been more difficult than I expected. Couple of pain-points and suggestions to make the process easier for myself & future contributo