On Wed, May 3, 2017 at 21:57 Andrea Frittoli <andrea.fritt...@gmail.com> wrote:
> On Tue, May 2, 2017 at 2:41 PM Jordan Pittier <jordan.pitt...@scality.com> > wrote: > >> On Tue, May 2, 2017 at 7:42 AM, Ghanshyam Mann <ghanshyamm...@gmail.com> >> wrote: >> >>> In Cinder, there are many features/APIs which are backend specific and >>> will return 405 or 501 if same is not implemented on any backend [1]. >>> If such tests are implemented in Tempest, then it will break some gate >>> where that backend job is voting. like ceph job in glance_store gate. >>> >>> There been many such cases recently where ceph jobs were broken due to >>> such tests and recently it is for force-delete backup feature[2]. >>> Reverting force-delete tests in [3]. To resolve such cases at some >>> extend, Jon is going to add a white/black list of tests which can run >>> on ceph job [4] depends on what all feature ceph implemented. But this >>> does not resolve it completely due to many reason like >>> 1. External use of Tempest become difficult where user needs to know >>> what all tests to skip for which backend >>> 2. Tempest tests become too specific to backend. >>> >>> Now there are few options to resolve this: >>> 1. Tempest should not tests such API/feature which are backend >>> specific like mentioned by api-ref like[1]. >>> >> So basically, if one of the 50 Cinder driver doesn't support a feature, >> we should never test that feature ? What about the 49 other drivers ? If a >> feature exists and can be tested in the Gate (with whatever default >> config/driver is shipped) then I think we should test it. >> >> >>> 2. Tempest test can be disabled/skip based on backend. - This is not >>> good idea as it increase config options and overhead of setting those. >>> >> Using regex and blacklist, any 3rd party CI can skip any test based on >> the test ID. Without introducing a config flag. See: >> https://github.com/openstack-infra/project-config/blob/1cea31f402b6b0cccc47cde203c12184b5392c90/jenkins/jobs/devstack-gate.yaml#L1871 >> > > This way each 3rd party system has to maintain its own list, which has the > advantage that > different teams maintain their own list (which is nice from an ownership > and scale pov). > > However I think such list of tests are not as re-usable as having a > devstack plugin (or an > ansible or puppet module) changing a few tempest config options. > Humm, I am little bit hesitate to go that way. For gate and CI it might be good solution but for production cloud testing it makes Tenpest difficult to use. If I use Tempest to test my cloud, few tests going to fail as those were not supported by cinder driver my cloud has or going to have. I do not have any way to configure something so that test can be disabled. Instead I need to maintain list of tests to run or skip. And that list is not static, it grows dynamically. This makes Tempest difficult to use. > >> >>> 3. Tempest test can verify behavior with if else condition as per >>> backend. This is bad idea and lose the test strength. >>> >> Yeah, that's bad. >> >>> >>> IMO options 1 is better options. More feedback are welcome. >> >> >>> ..1 >>> https://developer.openstack.org/api-ref/block-storage/v3/?expanded=force-delete-a-backup-detail#force-delete-a-backup >>> ..2 https://bugs.launchpad.net/glance/+bug/1687538 >>> ..3 https://review.openstack.org/#/c/461625/ >>> ..4 >>> http://lists.openstack.org/pipermail/openstack-dev/2017-April/115229.html >>> >>> -gmann >>> >>> >>> __________________________________________________________________________ >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: >>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- -gmann
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev