Excerpts from Graham Hayes's message of 2018-01-18 15:33:12 +0000: > I had hoped for more of a discussion on this before I jumped back into > this debate - but it seams to be stalled still, so here it goes. > > I proposed this initially as we were unclear on where the tests should > go - we had a resolution that said all tests go into openstack/tempest > (with a list of reasons why), and the guidance and discussion that been > had in various summits was that "add-ons" should stay in plugins. > > So right now, we (by the governance rules) should be pushing tests to > tempest for the new programs. > > In the resolution that placed the tests in tempest there was a few > reasons proposed: > > For example, API and behavioral changes must be carefully managed, as > must mundane aspects such as test and module naming and location > within the test suite. Even changes that leave tests functionally > equivalent may cause unexpected consequences for their use in DefCore > processes and validation. Placing the tests in a central repository > will make it easier to maintain consistency and avoid breaking the > trademark enforcement tool. > > This still applies, and even more so for teams that traditionally do not > have a strong QE contributor / reviewer base (aka projects not in > "core"). > > Centralizing the tests also makes it easier for anyone running the > validation tool against their cloud or cloud distribution to use the > tests. It is easier to install the test suite and its dependencies, > and it is easier to read and understand a set of tests following a > consistent implementation pattern. > > Apparently users do not need central tests anymore, feedback from > RefStack is that people who run these tests are comfortable dealing > with extra python packages. > > The point about a single set of tests, in a single location and style > still stands. > > Finally, having the tests in a central location makes it easier to > ensure that all members of the community have equal input into what > the tests do and how they are implemented and maintained. > > Seems like a good value to strive for. > > One of the items that has been used to push back against adding > "add-ons" to tempest has been that tempest has a defined scope, and > neither of the current add-ons fit in that scope. > > Can someone clarify what the set of criteria is? I think it will help > this discussion. > > Another push back is the "scaling" issue - adding more tests will > overload the QA team.
In the past the QA team agreed to accept trademark-related tests from all projects in the tempest repo. Has that changed? > > Right now, DNS adds 10 tests, Orchestration adds 22, to a current suite > of 353. > > I do not think there is many other add-ons proposed yet, and the new > Vertical programs will probably mainly be re-using tests in the > openstack/tempest repos as is. > > This is not a big tent-esque influx of programs - the only projects > that can be added to the trademarks are programs in tc-approved-release > [4], so I do not see scaling as a big issue, especially as these tests > are such base concepts that if they need to be changed there is a > completely new API, so the only overhead will be ensuring that nothing > in tempest breaks the new tests (which is a good thing for trademark > tests). > > Personally, for me, I like option 3. I did not initially add it, as I > knew it would cause endless bikesheding, but I do think it fits both > a technical and social model. > > I see 2 immediate routes forward: > > - Option 1, and we start adding these tests asap > - Pseudo Option 2, were we delete the resolution at [2] as it clearly > does not apply anymore, and abandon the review at [1]. > > Finally - do not conflate my actions with those of the Designate team. > I have seen people talking about how this resolution was leverage the > team needed to move our tests in tree. This is definitely *not* true. > Having our tests in a plugin is useful to us, and if the above > resolution passed, I cannot see a situation where we would try to > move any tests that were not listed in the interop standards. > > This is something I have done as an individual in the community, not > something the designate team have pushed for. Thanks for pushing for a clear resolution to this, Graham. > > > [4] - > https://governance.openstack.org/tc/reference/tags/tc_approved-release.html > > > __________________________________________________________________________ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev