On 05/02/2017 11:49 AM, Sean McGinnis wrote:
On Tue, May 02, 2017 at 03:36:20PM +0200, Jordan Pittier wrote:
On Tue, May 2, 2017 at 7:42 AM, Ghanshyam Mann <ghanshyamm...@gmail.com>
wrote:
In Cinder, there are many features/APIs which are backend specific and
will return 405 or 501 if same is not implemented on any backend [1].
If such tests are implemented in Tempest, then it will break some gate
where that backend job is voting. like ceph job in glance_store gate.
There been many such cases recently where ceph jobs were broken due to
such tests and recently it is for force-delete backup feature[2].
Reverting force-delete tests in [3]. To resolve such cases at some
extend, Jon is going to add a white/black list of tests which can run
on ceph job [4] depends on what all feature ceph implemented. But this
does not resolve it completely due to many reason like
1. External use of Tempest become difficult where user needs to know
what all tests to skip for which backend
2. Tempest tests become too specific to backend.
Now there are few options to resolve this:
1. Tempest should not tests such API/feature which are backend
specific like mentioned by api-ref like[1].
So basically, if one of the 50 Cinder driver doesn't support a feature, we
should never test that feature ? What about the 49 other drivers ? If a
feature exists and can be tested in the Gate (with whatever default
config/driver is shipped) then I think we should test it.
50? Over 100 as of Ocata.
Well, is tempest's purpose in life to provide complete gate test coverage,
or is tempest's purpose in life to give operators a tool to validate that
their deployment is working as expected?
I'd actually like to suggest that such a scenario actually points out a
thing that is ultimately potential pain passed to the end user in the
real world, so this question about what/how to test this in tempest is a
good one to have.
If there is a feature which is only provisionally available depending on
the backend driver such that it's hard to test in tempest without an out
of band config - then it's a feature that a user will have no clue
whether it works on a given cloud.
As we find these, I'd love it if we could expose discovery in the API
for viability of the feature. Like:
GET /capabilities
{
"capabilities": {
"has_force_delete": true
}
}
(I know we've talked about that concept generally, but this is a
specific example)
If such a thing existed, then the user can know whether they can use a
thing .. and so can tempest. A tempest test to validate force_delete
working could check the capability reported by the API and validate that
if the API says "true" that the feature work as expected, and if it says
"false" validate that attempting to call it returns a 405 (or whatever
is appropriate)
Ultimately, every config we need to feed to tempest is potentially a
place where an end user is unable to know whether or not to expect a
call to work - and an opportunity for us to provide our API consumers
with a richer experience.
In attempting to do things in the past, I've received push back based on
the argument that it was the latter. For this reason, in-tree tempest tests
were added to Cinder to give us a way to get better test coverage for our
own sake.
Now that this is all in place, I think it's working well and I would like
to see it continue that way. IMO, tempest proper should not have anything
that isn't universally applicable to real world deployments. Not just for
things like Ceph, but things like the manage/unmanage backend specific
tests that were added and broke a large majority of third party CI.
Backend specific things should not be part of tempest in my opinion. We
should cover those things through in-tree tempest plugins and our own
testing.
2. Tempest test can be disabled/skip based on backend. - This is not
good idea as it increase config options and overhead of setting those.
Using regex and blacklist, any 3rd party CI can skip any test based on the
test ID. Without introducing a config flag. See:
https://github.com/openstack-infra/project-config/blob/1cea31f402b6b0cccc47cde203c12184b5392c90/jenkins/jobs/devstack-gate.yaml#L1871
3. Tempest test can verify behavior with if else condition as per
backend. This is bad idea and lose the test strength.
Yeah, that's bad.
IMO options 1 is better options. More feedback are welcome.
..1 https://developer.openstack.org/api-ref/block-storage/v3/?
expanded=force-delete-a-backup-detail#force-delete-a-backup
..2 https://bugs.launchpad.net/glance/+bug/1687538
..3 https://review.openstack.org/#/c/461625/
..4 http://lists.openstack.org/pipermail/openstack-dev/2017-
April/115229.html
-gmann
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
