On 03/20/2014 04:19 PM, Rochelle.RochelleGrober wrote:
-----Original Message-----
From: Malini Kamalambal [mailto:malini.kamalam...@rackspace.com]
Sent: Thursday, March 20, 2014 12:13 PM
'project specific functional testing' in the Marconi context is
treating
Marconi as a complete system, making Marconi API calls & verifying the
response - just like an end user would, but without keystone. If one of
these tests fail, it is because there is a bug in the Marconi code ,
and
not because its interaction with Keystone caused it to fail.
"That being said there are certain cases where having a project
specific
functional test makes sense. For example swift has a functional test
job
that
starts swift in devstack. But, those things are normally handled on a
per
case
basis. In general if the project is meant to be part of the larger
OpenStack
ecosystem then Tempest is the place to put functional testing. That way
you know
it works with all of the other components. The thing is in openstack
what
seems
like a project isolated functional test almost always involves another
project
in real use cases. (for example keystone auth with api requests)
"
One of the concerns we heard in the review was 'having the functional
tests elsewhere (I.e within the project itself) does not count and they
have to be in Tempest'.
This has made us as a team wonder if we should migrate all our
functional
tests to Tempest.
But from Matt's response, I think it is reasonable to continue in our
current path & have the functional tests in Marconi coexist along with
the tests in Tempest.
I think that what is being asked, really is that the functional tests could be a single set of
tests that would become a part of the tempest repository and that these tests would have an ENV
variable as part of the configuration that would allow either "no Keystone" or
"Keystone" or some such, if that is the only configuration issue that separates running
the tests isolated vs. integrated. The functional tests need to be as much as possible a single
set of tests to reduce duplication and remove the likelihood of two sets getting out of sync with
each other/development. If they only run in the integrated environment, that's ok, but if you want
to run them isolated to make debugging easier, then it should be a configuration option and a
separate test job.
So, if my assumptions are correct, QA only requires functional tests for
integrated runs, but if the project QAs/Devs want to run isolated for dev and
devtest purposes, more power to them. Just keep it a single set of functional
tests and put them in the Tempest repository so that if a failure happens,
anyone can find the test and do the debug work without digging into a separate
project repository.
Hopefully, the tests as designed could easily take a new configuration
directive and a short bit of work with OS QA will get the integrated FTs
working as well as the isolated ones.
--Rocky
This issue has been much debated. There are some active members of our
community who believe that all the functional tests should live outside
of tempest in the projects, albeit with the same idea that such tests
could be run either as part of today's "real" tempest runs or mocked in
various ways to allow component isolation or better performance. Maru
Newby posted a patch with an example of one way to do this but I think
it expired and I don't have a pointer.
IMO there are valid arguments on both sides, but I hope every one could
agree that functional tests should not be arbitrarily split between
projects and tempest as they are now. The Tempest README states a desire
for "complete coverage of the OpenStack API" but Tempest is not close to
that. We have been discussing and then ignoring this issue for some time
but I think the recent action to say that Tempest will be used to
determine if something can use the OpenStack trademark will force more
completeness on tempest (more tests, that is). I think we need to
resolve this issue but it won't be easy and modifying existing api tests
to be more flexible will be a lot of work. But at least new projects
could get on the right path sooner.
-David
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev