From: GHANSHYAM MANN <ghanshyamm...@gmail.com> Subject: Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest Date: Fri, 18 Mar 2016 10:05:39 +0900
> On Fri, Mar 18, 2016 at 9:06 AM, Ken'ichi Ohmichi <ken1ohmi...@gmail.com> > wrote: >> 2016-03-17 4:05 GMT-07:00 Andrea Frittoli <andrea.fritt...@gmail.com>: >>> On Thu, Mar 17, 2016 at 2:57 AM Ken'ichi Ohmichi <ken1ohmi...@gmail.com> >>> wrote: >>>> >>>> 2016-03-16 19:41 GMT-07:00 Jim Rollenhagen <j...@jimrollenhagen.com>: >>>> > On Wed, Mar 16, 2016 at 06:20:11PM -0700, Ken'ichi Ohmichi wrote: >>>> >> Hi >>>> >> >>>> >> I have one proposal[1] related to negative tests in Tempest, and >>>> >> hoping opinions before doing that. >>>> >> >>>> >> Now Tempest contains negative tests and sometimes patches are being >>>> >> posted for adding more negative tests, but I'd like to propose >>>> >> removing them from Tempest instead. >>>> >> >>>> >> Negative tests verify surfaces of REST APIs for each component without >>>> >> any integrations between components. That doesn't seem integration >>>> >> tests which are scope of Tempest. >>>> >> In addition, we need to spend the test operating time on different >>>> >> component's gate if adding negative tests into Tempest. For example, >>>> >> we are operating negative tests of Keystone and more >>>> >> components on the gate of Nova. That is meaningless, so we need to >>>> >> avoid more negative tests into Tempest now. >>>> >> >>>> >> If wanting to add negative tests, it is a nice option to implement >>>> >> these tests on each component repo with Tempest plugin interface. We >>>> >> can avoid operating negative tests on different component gates and >>>> >> each component team can decide what negative tests are valuable on the >>>> >> gate. >>>> >> >>>> >> In long term, all negative tests will be migrated into each component >>>> >> repo with Tempest plugin interface. We will be able to operate >>>> >> valuable negative tests only on each gate. >>>> > >>>> > So, positive tests in tempest, negative tests as a plugin. >>>> > >>>> > Is there any longer term goal to have all tests for all projects in a >>>> > plugin for that project? Seems odd to separate them. >>>> >>>> Yeah, from implementation viewpoint, that seems a little odd. >>>> but from the main scope of Tempest and to avoid unnecessary gate >>>> operation time, that can be acceptable, I feel. >>>> Negative tests can be corner cases in most cases, they don't seem >>>> integration tests. >>> >>> I think it's difficult to define a single black and white criteria for >>> negative tests, as they encompass a wide range of types of tests. >>> >>> I agree that things that only testing the API level of a service (not even a >>> DB behind) do not necessarily belong in tempest - i.e. testing of input >>> validation done by an API. We could have a guideline for such tests to be >>> implemented as unit/functional tests in tree of the service. > > Yes, this is key point here. If we see ~70% of the negative tests are > just checking API surface level (wrong input validation), which > defiantly > not belong to Tempest scope. Those should be in respective projects > repo either by functional/unit/plugin. > But in that case we have to define a very clear criteria about what > level of negative testing should be in scope of Tempest. > > Also another key point is that, as we have lot of surface level > negative testing in Tempest, should we reject the new one? > For me sometime it makes difficult to reject those as we already have > some in tempest. > > My vote here is we reject the new surface level negative tests and try > to move all existing negative tests(surface level) out of Tempest > ASAP. > And those can be just moved to projects functional/unit tests. > >> >> Yeah, it is difficult to distinguish corner cases or not on negative >> tests as the criteria. >> If necessary to do that, we(QA team) need to read the implementation >> code of the core six services deeply during Tempest reviews. Then I >> rushed to remove all of them. My first proposal is not good according >> to feedback, but I'm happy to get feedback to see our direction :-) >> >> The guideline is a nice idea. >> If necessary to add more negative tests into Tempest, how about >> requiring to write the reason which explains new tests are not corner >> cases in the commit message? >> We can know the merit of new negative ones when reviewing. >> >>> However Tempest is also interoperability, so we should keep at least a few >>> negative API checks in tempest (for the core six services) to enforce that >>> return codes do not change inadvertently in negative cases, which could >>> break existing clients and applications. >> >> This also is a nice point. >> How to change error return codes is unclear to me at this time. >> In Nova, there are some exceptions for changing error return code >> without microversion bumping as [1]. This kind of guideline will be >> discussed later. > > This makes Tempest scope little bit unclear again. If we want to > verify all error codes in Tempest then it leads to have all surface > level negative testing also in Tempest. There are lot of scenario > where error codes can be verified and will be difficult to cover all > in Tempest. > > Current negative tests does not cover all error codes for all APIs. If > we try to implement all then it will be huge tests number. > I think its project which should be verifying those. > > In Summary - > > 1. If we choose to have only valid negative tests (other than surface > level negative testing) which can verify stability/integration of API > and used by Def-core too then, > - We should remove all existing tests which only touch surface of > APIs (wrong input validation). > 2. If we want to verify error codes also in Tempest then, > - first point becomes invalid and we need to implement all > possible error code testing. > > Having mix of those always leads to have issue in reviews/development etc. I think it doesn't depend on surface or deeply but it depends on the interface is stable or not. Because Tempest is a blackbox testing suite, Tempest shouldn't care about the individual project implementation, basically. So the issue is we don't know that which interface(especially negative case) is stable or not, actually. I think individual projects (and/or Def-Core) can define that, though. In other words, when we have a tempest negative/positive test case, it defines as a stable interface for its project, IMO. Best Regards, -- Masayuki Igawa (IRC: masayukig) > >> >> Now I am fine to keep the existing negative tests in Tempest, anyway. >> >> Thanks >> Ken Ohmichi >> >> --- >> [1]: >> https://github.com/openstack/nova/blob/master/doc/source/api_microversion_dev.rst#f2 >> __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev