On 10/30/2014 11:12 AM, Sean Dague wrote:
On 10/30/2014 10:47 AM, Eoghan Glynn wrote:
Matthew wrote:
This would have the advantage of making tempest slimmer for every project
and begin the process of getting projects to take responsibility for their
functional testing rather than relying on tempest.
[much snipping]
Sean wrote:
Ok, so part of this remains to be seen about what the biggest bang for the
buck is. The class of bugs I feel like we need to nail in Nova right now are
going to require tests that bring up pieces of the wsgi stack, but are
probably not runable on a real deploy. Again, this is about debugability.
So this notion of the biggest bang for our buck is an aspect of the drive
for in-tree functional tests, that's not entirely clear to me as yet.
i.e. whether individual projects should be prioritizing within this effort:
(a) the creation of net-new coverage for scenarios (especially known or
suspected bugs) that were not previously tested, in a non-unit sense
(b) the relocation of existing integration test coverage from Tempest to
the project trees, in order to make the management of Tempest more
tractable
It feels like there may be a tension between (a) and (b) in terms of the
pay-off for this effort. I'd interested in hearing other opinions on this,
on what aspect projects are expecting (and expected) to concentrate on
initially.
For what it's worth I have a bunch of early targets listed for Nova for
our summit session -
https://etherpad.openstack.org/p/kilo-nova-functional-testing
My focus in kilo is going to be first about A), as that provides value
out of the gate (pun intended). Then peel off some stuff from B as makes
sense.
-Sean
That seems sensible from the nova point of view and overall health, and
not all projects have to pursue the same priorities at the same time.
But a big part of the benefit of (b) is the impact it has on all the
other projects in that other projects will stop getting as many gate
failures, and that benefit could be achieved right now by simply
changing the set of tempest tests that run against each project.
-David
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
