On Wed, May 03, 2017 at 11:56:56PM -0400, Matthew Treinish wrote: > On Wed, May 03, 2017 at 11:51:13AM +0000, Andrea Frittoli wrote: > > On Tue, May 2, 2017 at 5:33 PM Matthew Treinish <mtrein...@kortar.org> > > wrote: > > > > > On Tue, May 02, 2017 at 09:49:14AM +0530, Rabi Mishra wrote: > > > > On Fri, Apr 28, 2017 at 2:17 PM, Andrea Frittoli < > > > andrea.fritt...@gmail.com> > > > > wrote: > > > > > > > > > > > > > > > > > > > On Fri, Apr 28, 2017 at 10:29 AM Rabi Mishra <ramis...@redhat.com> > > > wrote: > > > > > > > > > >> On Thu, Apr 27, 2017 at 3:55 PM, Andrea Frittoli < > > > > >> andrea.fritt...@gmail.com> wrote: > > > > >> > > > > >>> Dear stackers, > > > > >>> > > > > >>> starting in the Liberty cycle Tempest has defined a set of projects > > > > >>> which are in scope for direct > > > > >>> testing in Tempest [0]. The current list includes keystone, nova, > > > > >>> glance, swift, cinder and neutron. > > > > >>> All other projects can use the same Tempest testing infrastructure > > > (or > > > > >>> parts of it) by taking advantage > > > > >>> the Tempest plugin and stable interfaces. > > > > >>> > > > > >>> Tempest currently hosts a set of API tests as well as a service > > > client > > > > >>> for the Heat project. > > > > >>> The Heat service client is used by the tests in Tempest, which run > > > > >>> in > > > > >>> Heat gate as part of the grenade > > > > >>> job, as well as in the Tempest gate (check pipeline) as part of the > > > > >>> layer4 job. > > > > >>> According to code search [3] the Heat service client is also used by > > > > >>> Murano and Daisycore. > > > > >>> > > > > >> > > > > >> For the heat grenade job, I've proposed two patches. > > > > >> > > > > >> 1. To run heat tree gabbi api tests as part of grenade 'post-upgrade' > > > > >> phase > > > > >> > > > > >> https://review.openstack.org/#/c/460542/ > > > > >> > > > > >> 2. To remove tempest tests from the grenade job > > > > >> > > > > >> https://review.openstack.org/#/c/460810/ > > > > >> > > > > >> > > > > >> > > > > >>> I proposed a patch to Tempest to start the deprecation counter for > > > Heat > > > > >>> / orchestration related > > > > >>> configuration items in Tempest [4], and I would like to make sure > > > that > > > > >>> all tests and the service client > > > > >>> either find a new home outside of Tempest, or are removed, by the > > > > >>> end > > > > >>> the Pike cycle at the latest. > > > > >>> > > > > >>> Heat has in-tree integration tests and Gabbi based API tests, but I > > > > >>> don't know if those provide > > > > >>> enough coverage to replace the tests on Tempest side. > > > > >>> > > > > >>> > > > > >> Yes, the heat gabbi api tests do not yet have the same coverage as > > > > >> the > > > > >> tempest tree api tests (lacks tests using nova, neutron and swift > > > > >> resources), but I think that should not stop us from *not* running > > > the > > > > >> tempest tests in the grenade job. > > > > >> > > > > >> I also don't know if the tempest tree heat tests are used by any > > > > >> other > > > > >> upstream/downstream jobs. We could surely add more tests to bridge > > > the gap. > > > > >> > > > > >> Also, It's possible to run the heat integration tests (we've enough > > > > >> coverage there) with tempest plugin after doing some initial setup, > > > as we > > > > >> do in all our dsvm gate jobs. > > > > >> > > > > >> It would propose to move tests and client to a Tempest plugin owned / > > > > >>> maintained by > > > > >>> the Heat team, so that the Heat team can have full flexibility in > > > > >>> consolidating their integration > > > > >>> tests. For Murano and Daisycloud - and any other team that may want > > > to > > > > >>> use the Heat service > > > > >>> client in their tests, even if the client is removed from Tempest, > > > > >>> it > > > > >>> would still be available via > > > > >>> the Heat Tempest plugin. As long as the plugin implements the > > > > >>> service > > > > >>> client interface, > > > > >>> the Heat service client will register automatically in the service > > > > >>> client manager and be available > > > > >>> for use as today. > > > > >>> > > > > >>> > > > > >> if I understand correctly, you're proposing moving the existing > > > tempest > > > > >> tests and service clients to a separate repo managed by heat team. > > > Though > > > > >> that would be collective decision, I'm not sure that's something I > > > would > > > > >> like to do. To start with we may look at adding some of the missing > > > pieces > > > > >> in heat tree itself. > > > > >> > > > > > > > > > > I'm proposing to move tests and the service client outside of tempest > > > to a > > > > > new home. > > > > > > > > > > I also suggested that the new home could be a dedicate repo, since > > > > > that > > > > > would allow you to maintain the > > > > > current branchless nature of those tests. A more detailed discussion > > > about > > > > > the topic can be found > > > > > in the corresponding proposed queens goal [5], > > > > > > > > > > Using a dedicated repo *is not* a precondition for moving tests and > > > > > service client out of Tempest. > > > > > > > > > > > > > > We probably are mixing two different things here. > > > > > > > > 1. Moving in-tree heat templest plugn and tests to a dedicated repo > > > > > > > > Though we don't have any plans for it now, we may have to do it when/if > > > > it's accepted as a community goal. > > > > > > > > 2. Moving tempest tree heat tests and heat service client to a new home > > > > and owner. > > > > > > > > I don't think that's something heat team would like to do given that we > > > > don't use these tests anywhere and would probably spend time improving > > > the > > > > coverage of the gabbi api tests we already have. > > > > > > > > > > > The test coverage provided by those tests is not available anywhere else at > > the moment, however the tests are not really used, so I'm confused about > > this. > > > > You could run the existing Tempest tests with very little extra effort > > until the > > corresponding coverage is available in Gabbi format. > > > > > > > > > > Ok, well if the heat team has no interest in maintaining these tests there > > > is > > > no point in keeping them around anymore. I've pushed up: > > > > > > https://review.openstack.org/461841 > > > > > > Thanks for this. > > We will need Heat PTL / QA Liaison +1 on the test removal before it goes > > through. > > > > > > > > > > to remove the tests. As for the clients we can just move those to > > > tempest.lib > > > to not break the plugins that are using them. > > > > > > > We could, but other projects maintain their own service clients, I'm not > > convinced we should > > make a special case for Heat. > > > > I'm not viewing it as a special case for heat, more treating it like any other > tempest interface that has a number of external consumers already but isn't in > lib yet. I don't think it's at all unreasonable to maintain the 321 lines of > heat client code in tempest.lib so that the plugins identified on this this > thread (and any others out there) don't lose an interface they rely on. The > maintenance burden for the clients is not high at all, especially since the > Rest API is supposed to be stable interface nothing should really need to > change in the client code often. We won't be adding new features just > preserving the existing interfaces. We just will need to document that they're > not self tested anymore because the tests no longer exist. > > Also, like it or not it this already is a special case. This is the first and > only time a project team didn't take ownership of their project's tempest > tests > since we made the decision to migrate higher level services into plugins back > at the Liberty summit. (it's also the last project left as part of that > migration)
FWIW I think it's unfair to say we didn't take ownership of tempest tests and I would have phrased the current status differently, basically the stuff that landed pre-dates the decision to migrate new services to a plugin based model, and that is also why our in-tree integration tests weren't based on that tempest framework. Thanks! Steve __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev