Re: [openstack-dev] [qa][heat][murano][daisycloud] Removing Heat support from Tempest

2017-05-04 Thread Matthew Treinish 
On Fri, May 05, 2017 at 09:29:40AM +1200, Steve Baker wrote:
> On Thu, May 4, 2017 at 3:56 PM, Matthew Treinish 
> wrote:
> 
> > On Wed, May 03, 2017 at 11:51:13AM +, Andrea Frittoli wrote:
> > > On Tue, May 2, 2017 at 5:33 PM Matthew Treinish 
> > > wrote:
> > >
> > > > On Tue, May 02, 2017 at 09:49:14AM +0530, Rabi Mishra wrote:
> > > > > On Fri, Apr 28, 2017 at 2:17 PM, Andrea Frittoli <
> > > > andrea.fritt...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, Apr 28, 2017 at 10:29 AM Rabi Mishra 
> > > > wrote:
> > > > > >
> > > > > >> On Thu, Apr 27, 2017 at 3:55 PM, Andrea Frittoli <
> > > > > >> andrea.fritt...@gmail.com> wrote:
> > > > > >>
> > > > > >>> Dear stackers,
> > > > > >>>
> > > > > >>> starting in the Liberty cycle Tempest has defined a set of
> > projects
> > > > > >>> which are in scope for direct
> > > > > >>> testing in Tempest [0]. The current list includes keystone, nova,
> > > > > >>> glance, swift, cinder and neutron.
> > > > > >>> All other projects can use the same Tempest testing
> > infrastructure
> > > > (or
> > > > > >>> parts of it) by taking advantage
> > > > > >>> the Tempest plugin and stable interfaces.
> > > > > >>>
> > > > > >>> Tempest currently hosts a set of API tests as well as a service
> > > > client
> > > > > >>> for the Heat project.
> > > > > >>> The Heat service client is used by the tests in Tempest, which
> > run in
> > > > > >>> Heat gate as part of the grenade
> > > > > >>> job, as well as in the Tempest gate (check pipeline) as part of
> > the
> > > > > >>> layer4 job.
> > > > > >>> According to code search [3] the Heat service client is also
> > used by
> > > > > >>> Murano and Daisycore.
> > > > > >>>
> > > > > >>
> > > > > >> For the heat grenade job, I've proposed two patches.
> > > > > >>
> > > > > >> 1. To run heat tree gabbi api tests as part of grenade
> > 'post-upgrade'
> > > > > >> phase
> > > > > >>
> > > > > >> https://review.openstack.org/#/c/460542/
> > > > > >>
> > > > > >> 2. To remove tempest tests from the grenade job
> > > > > >>
> > > > > >> https://review.openstack.org/#/c/460810/
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>> I proposed a patch to Tempest to start the deprecation counter
> > for
> > > > Heat
> > > > > >>> / orchestration related
> > > > > >>> configuration items in Tempest [4], and I would like to make sure
> > > > that
> > > > > >>> all tests and the service client
> > > > > >>> either find a new home outside of Tempest, or are removed, by
> > the end
> > > > > >>> the Pike cycle at the latest.
> > > > > >>>
> > > > > >>> Heat has in-tree integration tests and Gabbi based API tests,
> > but I
> > > > > >>> don't know if those provide
> > > > > >>> enough coverage to replace the tests on Tempest side.
> > > > > >>>
> > > > > >>>
> > > > > >> Yes, the heat gabbi api tests do not yet have the same coverage
> > as the
> > > > > >> tempest tree api tests (lacks tests using nova, neutron and swift
> > > > > >> resources),  but I think that should not stop us from *not*
> > running
> > > > the
> > > > > >> tempest tests in the grenade job.
> > > > > >>
> > > > > >> I also don't know if the tempest tree heat tests are used by any
> > other
> > > > > >> upstream/downstream jobs. We could surely add more tests to bridge
> > > > the gap.
> > > > > >>
> > > > > >> Also, It's possible to run the heat integration tests (we've
> > enough
> > > > > >> coverage there) with tempest plugin after doing some initial
> > setup,
> > > > as we
> > > > > >> do in all our dsvm gate jobs.
> > > > > >>
> > > > > >> It would propose to move tests and client to a Tempest plugin
> > owned /
> > > > > >>> maintained by
> > > > > >>> the Heat team, so that the Heat team can have full flexibility in
> > > > > >>> consolidating their integration
> > > > > >>> tests. For Murano and Daisycloud - and any other team that may
> > want
> > > > to
> > > > > >>> use the Heat service
> > > > > >>> client in their tests, even if the client is removed from
> > Tempest, it
> > > > > >>> would still be available via
> > > > > >>> the Heat Tempest plugin. As long as the plugin implements the
> > service
> > > > > >>> client interface,
> > > > > >>> the Heat service client will register automatically in the
> > service
> > > > > >>> client manager and be available
> > > > > >>> for use as today.
> > > > > >>>
> > > > > >>>
> > > > > >> if I understand correctly, you're proposing moving the existing
> > > > tempest
> > > > > >> tests and service clients to a separate repo managed by heat team.
> > > > Though
> > > > > >> that would be collective decision, I'm not sure that's something I
> > > > would
> > > > > >> like to do. To start with we may look at adding some of the
> > missing
> > > > pieces
> > > > > >> in heat tree itself.
> > > > > >>
> > > > > >
> > > > > > I'm proposing to move tests and the service client outside of
> > tempest
> > > > to a
> >

Re: [openstack-dev] [qa][heat][murano][daisycloud] Removing Heat support from Tempest

2017-05-04 Thread Steve Baker 
On Thu, May 4, 2017 at 3:56 PM, Matthew Treinish 
wrote:

> On Wed, May 03, 2017 at 11:51:13AM +, Andrea Frittoli wrote:
> > On Tue, May 2, 2017 at 5:33 PM Matthew Treinish 
> > wrote:
> >
> > > On Tue, May 02, 2017 at 09:49:14AM +0530, Rabi Mishra wrote:
> > > > On Fri, Apr 28, 2017 at 2:17 PM, Andrea Frittoli <
> > > andrea.fritt...@gmail.com>
> > > > wrote:
> > > >
> > > > >
> > > > >
> > > > > On Fri, Apr 28, 2017 at 10:29 AM Rabi Mishra 
> > > wrote:
> > > > >
> > > > >> On Thu, Apr 27, 2017 at 3:55 PM, Andrea Frittoli <
> > > > >> andrea.fritt...@gmail.com> wrote:
> > > > >>
> > > > >>> Dear stackers,
> > > > >>>
> > > > >>> starting in the Liberty cycle Tempest has defined a set of
> projects
> > > > >>> which are in scope for direct
> > > > >>> testing in Tempest [0]. The current list includes keystone, nova,
> > > > >>> glance, swift, cinder and neutron.
> > > > >>> All other projects can use the same Tempest testing
> infrastructure
> > > (or
> > > > >>> parts of it) by taking advantage
> > > > >>> the Tempest plugin and stable interfaces.
> > > > >>>
> > > > >>> Tempest currently hosts a set of API tests as well as a service
> > > client
> > > > >>> for the Heat project.
> > > > >>> The Heat service client is used by the tests in Tempest, which
> run in
> > > > >>> Heat gate as part of the grenade
> > > > >>> job, as well as in the Tempest gate (check pipeline) as part of
> the
> > > > >>> layer4 job.
> > > > >>> According to code search [3] the Heat service client is also
> used by
> > > > >>> Murano and Daisycore.
> > > > >>>
> > > > >>
> > > > >> For the heat grenade job, I've proposed two patches.
> > > > >>
> > > > >> 1. To run heat tree gabbi api tests as part of grenade
> 'post-upgrade'
> > > > >> phase
> > > > >>
> > > > >> https://review.openstack.org/#/c/460542/
> > > > >>
> > > > >> 2. To remove tempest tests from the grenade job
> > > > >>
> > > > >> https://review.openstack.org/#/c/460810/
> > > > >>
> > > > >>
> > > > >>
> > > > >>> I proposed a patch to Tempest to start the deprecation counter
> for
> > > Heat
> > > > >>> / orchestration related
> > > > >>> configuration items in Tempest [4], and I would like to make sure
> > > that
> > > > >>> all tests and the service client
> > > > >>> either find a new home outside of Tempest, or are removed, by
> the end
> > > > >>> the Pike cycle at the latest.
> > > > >>>
> > > > >>> Heat has in-tree integration tests and Gabbi based API tests,
> but I
> > > > >>> don't know if those provide
> > > > >>> enough coverage to replace the tests on Tempest side.
> > > > >>>
> > > > >>>
> > > > >> Yes, the heat gabbi api tests do not yet have the same coverage
> as the
> > > > >> tempest tree api tests (lacks tests using nova, neutron and swift
> > > > >> resources),  but I think that should not stop us from *not*
> running
> > > the
> > > > >> tempest tests in the grenade job.
> > > > >>
> > > > >> I also don't know if the tempest tree heat tests are used by any
> other
> > > > >> upstream/downstream jobs. We could surely add more tests to bridge
> > > the gap.
> > > > >>
> > > > >> Also, It's possible to run the heat integration tests (we've
> enough
> > > > >> coverage there) with tempest plugin after doing some initial
> setup,
> > > as we
> > > > >> do in all our dsvm gate jobs.
> > > > >>
> > > > >> It would propose to move tests and client to a Tempest plugin
> owned /
> > > > >>> maintained by
> > > > >>> the Heat team, so that the Heat team can have full flexibility in
> > > > >>> consolidating their integration
> > > > >>> tests. For Murano and Daisycloud - and any other team that may
> want
> > > to
> > > > >>> use the Heat service
> > > > >>> client in their tests, even if the client is removed from
> Tempest, it
> > > > >>> would still be available via
> > > > >>> the Heat Tempest plugin. As long as the plugin implements the
> service
> > > > >>> client interface,
> > > > >>> the Heat service client will register automatically in the
> service
> > > > >>> client manager and be available
> > > > >>> for use as today.
> > > > >>>
> > > > >>>
> > > > >> if I understand correctly, you're proposing moving the existing
> > > tempest
> > > > >> tests and service clients to a separate repo managed by heat team.
> > > Though
> > > > >> that would be collective decision, I'm not sure that's something I
> > > would
> > > > >> like to do. To start with we may look at adding some of the
> missing
> > > pieces
> > > > >> in heat tree itself.
> > > > >>
> > > > >
> > > > > I'm proposing to move tests and the service client outside of
> tempest
> > > to a
> > > > > new home.
> > > > >
> > > > > I also suggested that the new home could be a dedicate repo, since
> that
> > > > > would allow you to maintain the
> > > > > current branchless nature of those tests. A more detailed
> discussion
> > > about
> > > > > the topic can be found
> > > > > in the corresponding proposed queens

Re: [openstack-dev] [qa][heat][murano][daisycloud] Removing Heat support from Tempest

2017-05-04 Thread Steven Hardy 
On Wed, May 03, 2017 at 11:56:56PM -0400, Matthew Treinish wrote:
> On Wed, May 03, 2017 at 11:51:13AM +, Andrea Frittoli wrote:
> > On Tue, May 2, 2017 at 5:33 PM Matthew Treinish 
> > wrote:
> > 
> > > On Tue, May 02, 2017 at 09:49:14AM +0530, Rabi Mishra wrote:
> > > > On Fri, Apr 28, 2017 at 2:17 PM, Andrea Frittoli <
> > > andrea.fritt...@gmail.com>
> > > > wrote:
> > > >
> > > > >
> > > > >
> > > > > On Fri, Apr 28, 2017 at 10:29 AM Rabi Mishra 
> > > wrote:
> > > > >
> > > > >> On Thu, Apr 27, 2017 at 3:55 PM, Andrea Frittoli <
> > > > >> andrea.fritt...@gmail.com> wrote:
> > > > >>
> > > > >>> Dear stackers,
> > > > >>>
> > > > >>> starting in the Liberty cycle Tempest has defined a set of projects
> > > > >>> which are in scope for direct
> > > > >>> testing in Tempest [0]. The current list includes keystone, nova,
> > > > >>> glance, swift, cinder and neutron.
> > > > >>> All other projects can use the same Tempest testing infrastructure
> > > (or
> > > > >>> parts of it) by taking advantage
> > > > >>> the Tempest plugin and stable interfaces.
> > > > >>>
> > > > >>> Tempest currently hosts a set of API tests as well as a service
> > > client
> > > > >>> for the Heat project.
> > > > >>> The Heat service client is used by the tests in Tempest, which run 
> > > > >>> in
> > > > >>> Heat gate as part of the grenade
> > > > >>> job, as well as in the Tempest gate (check pipeline) as part of the
> > > > >>> layer4 job.
> > > > >>> According to code search [3] the Heat service client is also used by
> > > > >>> Murano and Daisycore.
> > > > >>>
> > > > >>
> > > > >> For the heat grenade job, I've proposed two patches.
> > > > >>
> > > > >> 1. To run heat tree gabbi api tests as part of grenade 'post-upgrade'
> > > > >> phase
> > > > >>
> > > > >> https://review.openstack.org/#/c/460542/
> > > > >>
> > > > >> 2. To remove tempest tests from the grenade job
> > > > >>
> > > > >> https://review.openstack.org/#/c/460810/
> > > > >>
> > > > >>
> > > > >>
> > > > >>> I proposed a patch to Tempest to start the deprecation counter for
> > > Heat
> > > > >>> / orchestration related
> > > > >>> configuration items in Tempest [4], and I would like to make sure
> > > that
> > > > >>> all tests and the service client
> > > > >>> either find a new home outside of Tempest, or are removed, by the 
> > > > >>> end
> > > > >>> the Pike cycle at the latest.
> > > > >>>
> > > > >>> Heat has in-tree integration tests and Gabbi based API tests, but I
> > > > >>> don't know if those provide
> > > > >>> enough coverage to replace the tests on Tempest side.
> > > > >>>
> > > > >>>
> > > > >> Yes, the heat gabbi api tests do not yet have the same coverage as 
> > > > >> the
> > > > >> tempest tree api tests (lacks tests using nova, neutron and swift
> > > > >> resources),  but I think that should not stop us from *not* running
> > > the
> > > > >> tempest tests in the grenade job.
> > > > >>
> > > > >> I also don't know if the tempest tree heat tests are used by any 
> > > > >> other
> > > > >> upstream/downstream jobs. We could surely add more tests to bridge
> > > the gap.
> > > > >>
> > > > >> Also, It's possible to run the heat integration tests (we've enough
> > > > >> coverage there) with tempest plugin after doing some initial setup,
> > > as we
> > > > >> do in all our dsvm gate jobs.
> > > > >>
> > > > >> It would propose to move tests and client to a Tempest plugin owned /
> > > > >>> maintained by
> > > > >>> the Heat team, so that the Heat team can have full flexibility in
> > > > >>> consolidating their integration
> > > > >>> tests. For Murano and Daisycloud - and any other team that may want
> > > to
> > > > >>> use the Heat service
> > > > >>> client in their tests, even if the client is removed from Tempest, 
> > > > >>> it
> > > > >>> would still be available via
> > > > >>> the Heat Tempest plugin. As long as the plugin implements the 
> > > > >>> service
> > > > >>> client interface,
> > > > >>> the Heat service client will register automatically in the service
> > > > >>> client manager and be available
> > > > >>> for use as today.
> > > > >>>
> > > > >>>
> > > > >> if I understand correctly, you're proposing moving the existing
> > > tempest
> > > > >> tests and service clients to a separate repo managed by heat team.
> > > Though
> > > > >> that would be collective decision, I'm not sure that's something I
> > > would
> > > > >> like to do. To start with we may look at adding some of the missing
> > > pieces
> > > > >> in heat tree itself.
> > > > >>
> > > > >
> > > > > I'm proposing to move tests and the service client outside of tempest
> > > to a
> > > > > new home.
> > > > >
> > > > > I also suggested that the new home could be a dedicate repo, since 
> > > > > that
> > > > > would allow you to maintain the
> > > > > current branchless nature of those tests. A more detailed discussion
> > > about
> > > > > the topic can be found
> > > > > in the

Re: [openstack-dev] [qa][heat][murano][daisycloud] Removing Heat support from Tempest

2017-05-03 Thread Matthew Treinish 
On Wed, May 03, 2017 at 11:51:13AM +, Andrea Frittoli wrote:
> On Tue, May 2, 2017 at 5:33 PM Matthew Treinish 
> wrote:
> 
> > On Tue, May 02, 2017 at 09:49:14AM +0530, Rabi Mishra wrote:
> > > On Fri, Apr 28, 2017 at 2:17 PM, Andrea Frittoli <
> > andrea.fritt...@gmail.com>
> > > wrote:
> > >
> > > >
> > > >
> > > > On Fri, Apr 28, 2017 at 10:29 AM Rabi Mishra 
> > wrote:
> > > >
> > > >> On Thu, Apr 27, 2017 at 3:55 PM, Andrea Frittoli <
> > > >> andrea.fritt...@gmail.com> wrote:
> > > >>
> > > >>> Dear stackers,
> > > >>>
> > > >>> starting in the Liberty cycle Tempest has defined a set of projects
> > > >>> which are in scope for direct
> > > >>> testing in Tempest [0]. The current list includes keystone, nova,
> > > >>> glance, swift, cinder and neutron.
> > > >>> All other projects can use the same Tempest testing infrastructure
> > (or
> > > >>> parts of it) by taking advantage
> > > >>> the Tempest plugin and stable interfaces.
> > > >>>
> > > >>> Tempest currently hosts a set of API tests as well as a service
> > client
> > > >>> for the Heat project.
> > > >>> The Heat service client is used by the tests in Tempest, which run in
> > > >>> Heat gate as part of the grenade
> > > >>> job, as well as in the Tempest gate (check pipeline) as part of the
> > > >>> layer4 job.
> > > >>> According to code search [3] the Heat service client is also used by
> > > >>> Murano and Daisycore.
> > > >>>
> > > >>
> > > >> For the heat grenade job, I've proposed two patches.
> > > >>
> > > >> 1. To run heat tree gabbi api tests as part of grenade 'post-upgrade'
> > > >> phase
> > > >>
> > > >> https://review.openstack.org/#/c/460542/
> > > >>
> > > >> 2. To remove tempest tests from the grenade job
> > > >>
> > > >> https://review.openstack.org/#/c/460810/
> > > >>
> > > >>
> > > >>
> > > >>> I proposed a patch to Tempest to start the deprecation counter for
> > Heat
> > > >>> / orchestration related
> > > >>> configuration items in Tempest [4], and I would like to make sure
> > that
> > > >>> all tests and the service client
> > > >>> either find a new home outside of Tempest, or are removed, by the end
> > > >>> the Pike cycle at the latest.
> > > >>>
> > > >>> Heat has in-tree integration tests and Gabbi based API tests, but I
> > > >>> don't know if those provide
> > > >>> enough coverage to replace the tests on Tempest side.
> > > >>>
> > > >>>
> > > >> Yes, the heat gabbi api tests do not yet have the same coverage as the
> > > >> tempest tree api tests (lacks tests using nova, neutron and swift
> > > >> resources),  but I think that should not stop us from *not* running
> > the
> > > >> tempest tests in the grenade job.
> > > >>
> > > >> I also don't know if the tempest tree heat tests are used by any other
> > > >> upstream/downstream jobs. We could surely add more tests to bridge
> > the gap.
> > > >>
> > > >> Also, It's possible to run the heat integration tests (we've enough
> > > >> coverage there) with tempest plugin after doing some initial setup,
> > as we
> > > >> do in all our dsvm gate jobs.
> > > >>
> > > >> It would propose to move tests and client to a Tempest plugin owned /
> > > >>> maintained by
> > > >>> the Heat team, so that the Heat team can have full flexibility in
> > > >>> consolidating their integration
> > > >>> tests. For Murano and Daisycloud - and any other team that may want
> > to
> > > >>> use the Heat service
> > > >>> client in their tests, even if the client is removed from Tempest, it
> > > >>> would still be available via
> > > >>> the Heat Tempest plugin. As long as the plugin implements the service
> > > >>> client interface,
> > > >>> the Heat service client will register automatically in the service
> > > >>> client manager and be available
> > > >>> for use as today.
> > > >>>
> > > >>>
> > > >> if I understand correctly, you're proposing moving the existing
> > tempest
> > > >> tests and service clients to a separate repo managed by heat team.
> > Though
> > > >> that would be collective decision, I'm not sure that's something I
> > would
> > > >> like to do. To start with we may look at adding some of the missing
> > pieces
> > > >> in heat tree itself.
> > > >>
> > > >
> > > > I'm proposing to move tests and the service client outside of tempest
> > to a
> > > > new home.
> > > >
> > > > I also suggested that the new home could be a dedicate repo, since that
> > > > would allow you to maintain the
> > > > current branchless nature of those tests. A more detailed discussion
> > about
> > > > the topic can be found
> > > > in the corresponding proposed queens goal [5],
> > > >
> > > > Using a dedicated repo *is not* a precondition for moving tests and
> > > > service client out of Tempest.
> > > >
> > > >
> > > We probably are mixing two different things here.
> > >
> > > 1. Moving in-tree heat templest plugn and tests to a dedicated repo
> > >
> > > Though we don't have any plans for it now, we may have to do it

Re: [openstack-dev] [qa][heat][murano][daisycloud] Removing Heat support from Tempest

2017-05-03 Thread Andrea Frittoli 
On Tue, May 2, 2017 at 5:33 PM Matthew Treinish 
wrote:

> On Tue, May 02, 2017 at 09:49:14AM +0530, Rabi Mishra wrote:
> > On Fri, Apr 28, 2017 at 2:17 PM, Andrea Frittoli <
> andrea.fritt...@gmail.com>
> > wrote:
> >
> > >
> > >
> > > On Fri, Apr 28, 2017 at 10:29 AM Rabi Mishra 
> wrote:
> > >
> > >> On Thu, Apr 27, 2017 at 3:55 PM, Andrea Frittoli <
> > >> andrea.fritt...@gmail.com> wrote:
> > >>
> > >>> Dear stackers,
> > >>>
> > >>> starting in the Liberty cycle Tempest has defined a set of projects
> > >>> which are in scope for direct
> > >>> testing in Tempest [0]. The current list includes keystone, nova,
> > >>> glance, swift, cinder and neutron.
> > >>> All other projects can use the same Tempest testing infrastructure
> (or
> > >>> parts of it) by taking advantage
> > >>> the Tempest plugin and stable interfaces.
> > >>>
> > >>> Tempest currently hosts a set of API tests as well as a service
> client
> > >>> for the Heat project.
> > >>> The Heat service client is used by the tests in Tempest, which run in
> > >>> Heat gate as part of the grenade
> > >>> job, as well as in the Tempest gate (check pipeline) as part of the
> > >>> layer4 job.
> > >>> According to code search [3] the Heat service client is also used by
> > >>> Murano and Daisycore.
> > >>>
> > >>
> > >> For the heat grenade job, I've proposed two patches.
> > >>
> > >> 1. To run heat tree gabbi api tests as part of grenade 'post-upgrade'
> > >> phase
> > >>
> > >> https://review.openstack.org/#/c/460542/
> > >>
> > >> 2. To remove tempest tests from the grenade job
> > >>
> > >> https://review.openstack.org/#/c/460810/
> > >>
> > >>
> > >>
> > >>> I proposed a patch to Tempest to start the deprecation counter for
> Heat
> > >>> / orchestration related
> > >>> configuration items in Tempest [4], and I would like to make sure
> that
> > >>> all tests and the service client
> > >>> either find a new home outside of Tempest, or are removed, by the end
> > >>> the Pike cycle at the latest.
> > >>>
> > >>> Heat has in-tree integration tests and Gabbi based API tests, but I
> > >>> don't know if those provide
> > >>> enough coverage to replace the tests on Tempest side.
> > >>>
> > >>>
> > >> Yes, the heat gabbi api tests do not yet have the same coverage as the
> > >> tempest tree api tests (lacks tests using nova, neutron and swift
> > >> resources),  but I think that should not stop us from *not* running
> the
> > >> tempest tests in the grenade job.
> > >>
> > >> I also don't know if the tempest tree heat tests are used by any other
> > >> upstream/downstream jobs. We could surely add more tests to bridge
> the gap.
> > >>
> > >> Also, It's possible to run the heat integration tests (we've enough
> > >> coverage there) with tempest plugin after doing some initial setup,
> as we
> > >> do in all our dsvm gate jobs.
> > >>
> > >> It would propose to move tests and client to a Tempest plugin owned /
> > >>> maintained by
> > >>> the Heat team, so that the Heat team can have full flexibility in
> > >>> consolidating their integration
> > >>> tests. For Murano and Daisycloud - and any other team that may want
> to
> > >>> use the Heat service
> > >>> client in their tests, even if the client is removed from Tempest, it
> > >>> would still be available via
> > >>> the Heat Tempest plugin. As long as the plugin implements the service
> > >>> client interface,
> > >>> the Heat service client will register automatically in the service
> > >>> client manager and be available
> > >>> for use as today.
> > >>>
> > >>>
> > >> if I understand correctly, you're proposing moving the existing
> tempest
> > >> tests and service clients to a separate repo managed by heat team.
> Though
> > >> that would be collective decision, I'm not sure that's something I
> would
> > >> like to do. To start with we may look at adding some of the missing
> pieces
> > >> in heat tree itself.
> > >>
> > >
> > > I'm proposing to move tests and the service client outside of tempest
> to a
> > > new home.
> > >
> > > I also suggested that the new home could be a dedicate repo, since that
> > > would allow you to maintain the
> > > current branchless nature of those tests. A more detailed discussion
> about
> > > the topic can be found
> > > in the corresponding proposed queens goal [5],
> > >
> > > Using a dedicated repo *is not* a precondition for moving tests and
> > > service client out of Tempest.
> > >
> > >
> > We probably are mixing two different things here.
> >
> > 1. Moving in-tree heat templest plugn and tests to a dedicated repo
> >
> > Though we don't have any plans for it now, we may have to do it when/if
> > it's accepted as a community goal.
> >
> > 2.  Moving tempest tree heat tests and heat service client to a new home
> > and owner.
> >
> > I don't think that's something heat team would like to do given that we
> > don't use these tests anywhere and would probably spend time improving
> the
> > coverage of

Re: [openstack-dev] [qa][heat][murano][daisycloud] Removing Heat support from Tempest

2017-05-02 Thread Matthew Treinish 
On Tue, May 02, 2017 at 09:49:14AM +0530, Rabi Mishra wrote:
> On Fri, Apr 28, 2017 at 2:17 PM, Andrea Frittoli 
> wrote:
> 
> >
> >
> > On Fri, Apr 28, 2017 at 10:29 AM Rabi Mishra  wrote:
> >
> >> On Thu, Apr 27, 2017 at 3:55 PM, Andrea Frittoli <
> >> andrea.fritt...@gmail.com> wrote:
> >>
> >>> Dear stackers,
> >>>
> >>> starting in the Liberty cycle Tempest has defined a set of projects
> >>> which are in scope for direct
> >>> testing in Tempest [0]. The current list includes keystone, nova,
> >>> glance, swift, cinder and neutron.
> >>> All other projects can use the same Tempest testing infrastructure (or
> >>> parts of it) by taking advantage
> >>> the Tempest plugin and stable interfaces.
> >>>
> >>> Tempest currently hosts a set of API tests as well as a service client
> >>> for the Heat project.
> >>> The Heat service client is used by the tests in Tempest, which run in
> >>> Heat gate as part of the grenade
> >>> job, as well as in the Tempest gate (check pipeline) as part of the
> >>> layer4 job.
> >>> According to code search [3] the Heat service client is also used by
> >>> Murano and Daisycore.
> >>>
> >>
> >> For the heat grenade job, I've proposed two patches.
> >>
> >> 1. To run heat tree gabbi api tests as part of grenade 'post-upgrade'
> >> phase
> >>
> >> https://review.openstack.org/#/c/460542/
> >>
> >> 2. To remove tempest tests from the grenade job
> >>
> >> https://review.openstack.org/#/c/460810/
> >>
> >>
> >>
> >>> I proposed a patch to Tempest to start the deprecation counter for Heat
> >>> / orchestration related
> >>> configuration items in Tempest [4], and I would like to make sure that
> >>> all tests and the service client
> >>> either find a new home outside of Tempest, or are removed, by the end
> >>> the Pike cycle at the latest.
> >>>
> >>> Heat has in-tree integration tests and Gabbi based API tests, but I
> >>> don't know if those provide
> >>> enough coverage to replace the tests on Tempest side.
> >>>
> >>>
> >> Yes, the heat gabbi api tests do not yet have the same coverage as the
> >> tempest tree api tests (lacks tests using nova, neutron and swift
> >> resources),  but I think that should not stop us from *not* running the
> >> tempest tests in the grenade job.
> >>
> >> I also don't know if the tempest tree heat tests are used by any other
> >> upstream/downstream jobs. We could surely add more tests to bridge the gap.
> >>
> >> Also, It's possible to run the heat integration tests (we've enough
> >> coverage there) with tempest plugin after doing some initial setup, as we
> >> do in all our dsvm gate jobs.
> >>
> >> It would propose to move tests and client to a Tempest plugin owned /
> >>> maintained by
> >>> the Heat team, so that the Heat team can have full flexibility in
> >>> consolidating their integration
> >>> tests. For Murano and Daisycloud - and any other team that may want to
> >>> use the Heat service
> >>> client in their tests, even if the client is removed from Tempest, it
> >>> would still be available via
> >>> the Heat Tempest plugin. As long as the plugin implements the service
> >>> client interface,
> >>> the Heat service client will register automatically in the service
> >>> client manager and be available
> >>> for use as today.
> >>>
> >>>
> >> if I understand correctly, you're proposing moving the existing tempest
> >> tests and service clients to a separate repo managed by heat team. Though
> >> that would be collective decision, I'm not sure that's something I would
> >> like to do. To start with we may look at adding some of the missing pieces
> >> in heat tree itself.
> >>
> >
> > I'm proposing to move tests and the service client outside of tempest to a
> > new home.
> >
> > I also suggested that the new home could be a dedicate repo, since that
> > would allow you to maintain the
> > current branchless nature of those tests. A more detailed discussion about
> > the topic can be found
> > in the corresponding proposed queens goal [5],
> >
> > Using a dedicated repo *is not* a precondition for moving tests and
> > service client out of Tempest.
> >
> >
> We probably are mixing two different things here.
> 
> 1. Moving in-tree heat templest plugn and tests to a dedicated repo
> 
> Though we don't have any plans for it now, we may have to do it when/if
> it's accepted as a community goal.
> 
> 2.  Moving tempest tree heat tests and heat service client to a new home
> and owner.
> 
> I don't think that's something heat team would like to do given that we
> don't use these tests anywhere and would probably spend time improving the
> coverage of the gabbi api tests we already have.
> 

Ok, well if the heat team has no interest in maintaining these tests there is
no point in keeping them around anymore. I've pushed up:

https://review.openstack.org/461841

to remove the tests. As for the clients we can just move those to tempest.lib
to not break the plugins that are using them.

Re: [openstack-dev] [qa][heat][murano][daisycloud] Removing Heat support from Tempest

2017-05-02 Thread Luigi Toscano 
- Original Message -
> 
> 
> On Tue, May 2, 2017 at 5:27 AM, MONTEIRO, FELIPE C < fm5...@att.com > wrote:
> 
> 
>> Murano currently uses the Tempest orchestration client for its scenario
>> Tempest tests [0], which are not turned on by default in the Murano Tempest
>> gate due to resource constraints.
>> 
>> However, I'm hesitant to switch to Heat's testing client, because it is not a
>> Tempest client, but rather the python-heatclient. I would like to know
>> whether there are plans to change this to a Tempest-based client?
> 
> There are no plans to switch the heat integration/functional tests to using
> the tempest based client. The heat tests will use heatclient for most tests,
> and gabbi for testing the REST API.
> 
> Since you're testing Murano rather than the Heat API, I think converting your
> tests to heatclient would be reasonable.

I think that a Tempest-based Heat client should live somewhere anyway, for the 
same reasons which lead to the creation of the Tempest clients.

Even if as a Heat team you are not directly interested in it, other consumers 
may be interested (like in this case). I think that a Heat Tempest client 
should live somewhere anyway.

-- 
Luigi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][heat][murano][daisycloud] Removing Heat support from Tempest

2017-05-01 Thread Rabi Mishra 
On Fri, Apr 28, 2017 at 2:17 PM, Andrea Frittoli 
wrote:

>
>
> On Fri, Apr 28, 2017 at 10:29 AM Rabi Mishra  wrote:
>
>> On Thu, Apr 27, 2017 at 3:55 PM, Andrea Frittoli <
>> andrea.fritt...@gmail.com> wrote:
>>
>>> Dear stackers,
>>>
>>> starting in the Liberty cycle Tempest has defined a set of projects
>>> which are in scope for direct
>>> testing in Tempest [0]. The current list includes keystone, nova,
>>> glance, swift, cinder and neutron.
>>> All other projects can use the same Tempest testing infrastructure (or
>>> parts of it) by taking advantage
>>> the Tempest plugin and stable interfaces.
>>>
>>> Tempest currently hosts a set of API tests as well as a service client
>>> for the Heat project.
>>> The Heat service client is used by the tests in Tempest, which run in
>>> Heat gate as part of the grenade
>>> job, as well as in the Tempest gate (check pipeline) as part of the
>>> layer4 job.
>>> According to code search [3] the Heat service client is also used by
>>> Murano and Daisycore.
>>>
>>
>> For the heat grenade job, I've proposed two patches.
>>
>> 1. To run heat tree gabbi api tests as part of grenade 'post-upgrade'
>> phase
>>
>> https://review.openstack.org/#/c/460542/
>>
>> 2. To remove tempest tests from the grenade job
>>
>> https://review.openstack.org/#/c/460810/
>>
>>
>>
>>> I proposed a patch to Tempest to start the deprecation counter for Heat
>>> / orchestration related
>>> configuration items in Tempest [4], and I would like to make sure that
>>> all tests and the service client
>>> either find a new home outside of Tempest, or are removed, by the end
>>> the Pike cycle at the latest.
>>>
>>> Heat has in-tree integration tests and Gabbi based API tests, but I
>>> don't know if those provide
>>> enough coverage to replace the tests on Tempest side.
>>>
>>>
>> Yes, the heat gabbi api tests do not yet have the same coverage as the
>> tempest tree api tests (lacks tests using nova, neutron and swift
>> resources),  but I think that should not stop us from *not* running the
>> tempest tests in the grenade job.
>>
>> I also don't know if the tempest tree heat tests are used by any other
>> upstream/downstream jobs. We could surely add more tests to bridge the gap.
>>
>> Also, It's possible to run the heat integration tests (we've enough
>> coverage there) with tempest plugin after doing some initial setup, as we
>> do in all our dsvm gate jobs.
>>
>> It would propose to move tests and client to a Tempest plugin owned /
>>> maintained by
>>> the Heat team, so that the Heat team can have full flexibility in
>>> consolidating their integration
>>> tests. For Murano and Daisycloud - and any other team that may want to
>>> use the Heat service
>>> client in their tests, even if the client is removed from Tempest, it
>>> would still be available via
>>> the Heat Tempest plugin. As long as the plugin implements the service
>>> client interface,
>>> the Heat service client will register automatically in the service
>>> client manager and be available
>>> for use as today.
>>>
>>>
>> if I understand correctly, you're proposing moving the existing tempest
>> tests and service clients to a separate repo managed by heat team. Though
>> that would be collective decision, I'm not sure that's something I would
>> like to do. To start with we may look at adding some of the missing pieces
>> in heat tree itself.
>>
>
> I'm proposing to move tests and the service client outside of tempest to a
> new home.
>
> I also suggested that the new home could be a dedicate repo, since that
> would allow you to maintain the
> current branchless nature of those tests. A more detailed discussion about
> the topic can be found
> in the corresponding proposed queens goal [5],
>
> Using a dedicated repo *is not* a precondition for moving tests and
> service client out of Tempest.
>
>
We probably are mixing two different things here.

1. Moving in-tree heat templest plugn and tests to a dedicated repo

Though we don't have any plans for it now, we may have to do it when/if
it's accepted as a community goal.

2.  Moving tempest tree heat tests and heat service client to a new home
and owner.

I don't think that's something heat team would like to do given that we
don't use these tests anywhere and would probably spend time improving the
coverage of the gabbi api tests we already have.


> andrea
>
> [5] https://review.openstack.org/#/c/369749/
>
>
>>
>> Andrea Frittoli (andreaf)
>>>
>>> [0] https://docs.openstack.org/developer/tempest/test_
>>> removal.html#tempest-scope
>>> [1] https://docs.openstack.org/developer/tempest/plugin.html
>>> [2] https://docs.openstack.org/developer/tempest/library.html
>>> [3] http://codesearch.openstack.org/?q=self.orchestration_client=nope;
>>> files==
>>> [4] https://review.openstack.org/#/c/456843/
>>>
>>> 
>>> __
>>> OpenStack Development Mailing List (not for

Re: [openstack-dev] [qa][heat][murano][daisycloud] Removing Heat support from Tempest

2017-05-01 Thread Steve Baker 
On Tue, May 2, 2017 at 5:27 AM, MONTEIRO, FELIPE C <fm5...@att.com> wrote:

> Murano currently uses the Tempest orchestration client for its scenario
> Tempest tests [0], which are not turned on by default in the Murano Tempest
> gate due to resource constraints.
>
> However, I'm hesitant to switch to Heat's testing client, because it is
> not a Tempest client, but rather the python-heatclient. I would like to
> know whether there are plans to change this to a Tempest-based client?
>

There are no plans to switch the heat integration/functional tests to using
the tempest based client. The heat tests will use heatclient for most
tests, and gabbi for testing the REST API.

Since you're testing Murano rather than the Heat API, I think converting
your tests to heatclient would be reasonable.


> [0] https://github.com/openstack/murano/blob/master/murano_
> tempest_tests/tests/scenario/application_catalog/base.py#L100
> [1] https://github.com/openstack/heat/blob/master/heat_
> integrationtests/common/clients.py#L120
>
> Felipe
>
> -Original Message-
> From: Ghanshyam Mann [mailto:ghanshyamm...@gmail.com]
> Sent: Sunday, April 30, 2017 1:53 AM
> To: OpenStack Development Mailing List (not for usage questions) <
> openstack-dev@lists.openstack.org>
> Subject: Re: [openstack-dev] [qa][heat][murano][daisycloud] Removing Heat
> support from Tempest
>
> On Fri, Apr 28, 2017 at 5:47 PM, Andrea Frittoli
> <andrea.fritt...@gmail.com> wrote:
> >
> >
> > On Fri, Apr 28, 2017 at 10:29 AM Rabi Mishra <ramis...@redhat.com>
> wrote:
> >>
> >> On Thu, Apr 27, 2017 at 3:55 PM, Andrea Frittoli
> >> <andrea.fritt...@gmail.com> wrote:
> >>>
> >>> Dear stackers,
> >>>
> >>> starting in the Liberty cycle Tempest has defined a set of projects
> which
> >>> are in scope for direct
> >>> testing in Tempest [0]. The current list includes keystone, nova,
> glance,
> >>> swift, cinder and neutron.
> >>> All other projects can use the same Tempest testing infrastructure (or
> >>> parts of it) by taking advantage
> >>> the Tempest plugin and stable interfaces.
> >>>
> >>> Tempest currently hosts a set of API tests as well as a service client
> >>> for the Heat project.
> >>> The Heat service client is used by the tests in Tempest, which run in
> >>> Heat gate as part of the grenade
> >>> job, as well as in the Tempest gate (check pipeline) as part of the
> >>> layer4 job.
> >>> According to code search [3] the Heat service client is also used by
> >>> Murano and Daisycore.
> >>
> >>
> >> For the heat grenade job, I've proposed two patches.
> >>
> >> 1. To run heat tree gabbi api tests as part of grenade 'post-upgrade'
> >> phase
> >>
> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__review.
> openstack.org_-23_c_460542_=DwIGaQ=LFYZ-o9_HUMeMTSQicvjIg=X4GwEru-
> SJ9DRnCxhze-aw=aN-OTm6qpDxNIXC86mUeowDuZe9O-NeCWHJdSvrVsYA=
> d2pZwZ8xKsFLHxQ0YNiM4itJjUHzgE0ibHNu7v28mXM=
> >>
> >> 2. To remove tempest tests from the grenade job
> >>
> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__review.
> openstack.org_-23_c_460810_=DwIGaQ=LFYZ-o9_HUMeMTSQicvjIg=X4GwEru-
> SJ9DRnCxhze-aw=aN-OTm6qpDxNIXC86mUeowDuZe9O-NeCWHJdSvrVsYA=07__
> zljUdvdtD_K5ltoKwdjaBwrs0fYJKaXSr93AAiU=
> >>
> >>
> >>>
> >>> I proposed a patch to Tempest to start the deprecation counter for
> Heat /
> >>> orchestration related
> >>> configuration items in Tempest [4], and I would like to make sure that
> >>> all tests and the service client
> >>> either find a new home outside of Tempest, or are removed, by the end
> the
> >>> Pike cycle at the latest.
> >>>
> >>> Heat has in-tree integration tests and Gabbi based API tests, but I
> don't
> >>> know if those provide
> >>> enough coverage to replace the tests on Tempest side.
> >>>
> >>
> >> Yes, the heat gabbi api tests do not yet have the same coverage as the
> >> tempest tree api tests (lacks tests using nova, neutron and swift
> >> resources),  but I think that should not stop us from *not* running the
> >> tempest tests in the grenade job.
> >>
> >> I also don't know if the tempest tree heat tests are used by any other
> >> upstream/downstream jobs. We could surely add more tests to bridge the
> gap.
> >>
> >> Also, It's possible to run t

Re: [openstack-dev] [qa][heat][murano][daisycloud] Removing Heat support from Tempest

2017-05-01 Thread MONTEIRO, FELIPE C 
Murano currently uses the Tempest orchestration client for its scenario Tempest 
tests [0], which are not turned on by default in the Murano Tempest gate due to 
resource constraints. 

However, I'm hesitant to switch to Heat's testing client, because it is not a 
Tempest client, but rather the python-heatclient. I would like to know whether 
there are plans to change this to a Tempest-based client? 

[0] 
https://github.com/openstack/murano/blob/master/murano_tempest_tests/tests/scenario/application_catalog/base.py#L100
[1] 
https://github.com/openstack/heat/blob/master/heat_integrationtests/common/clients.py#L120
 

Felipe

-Original Message-
From: Ghanshyam Mann [mailto:ghanshyamm...@gmail.com] 
Sent: Sunday, April 30, 2017 1:53 AM
To: OpenStack Development Mailing List (not for usage questions) 
<openstack-dev@lists.openstack.org>
Subject: Re: [openstack-dev] [qa][heat][murano][daisycloud] Removing Heat 
support from Tempest

On Fri, Apr 28, 2017 at 5:47 PM, Andrea Frittoli
<andrea.fritt...@gmail.com> wrote:
>
>
> On Fri, Apr 28, 2017 at 10:29 AM Rabi Mishra <ramis...@redhat.com> wrote:
>>
>> On Thu, Apr 27, 2017 at 3:55 PM, Andrea Frittoli
>> <andrea.fritt...@gmail.com> wrote:
>>>
>>> Dear stackers,
>>>
>>> starting in the Liberty cycle Tempest has defined a set of projects which
>>> are in scope for direct
>>> testing in Tempest [0]. The current list includes keystone, nova, glance,
>>> swift, cinder and neutron.
>>> All other projects can use the same Tempest testing infrastructure (or
>>> parts of it) by taking advantage
>>> the Tempest plugin and stable interfaces.
>>>
>>> Tempest currently hosts a set of API tests as well as a service client
>>> for the Heat project.
>>> The Heat service client is used by the tests in Tempest, which run in
>>> Heat gate as part of the grenade
>>> job, as well as in the Tempest gate (check pipeline) as part of the
>>> layer4 job.
>>> According to code search [3] the Heat service client is also used by
>>> Murano and Daisycore.
>>
>>
>> For the heat grenade job, I've proposed two patches.
>>
>> 1. To run heat tree gabbi api tests as part of grenade 'post-upgrade'
>> phase
>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__review.openstack.org_-23_c_460542_=DwIGaQ=LFYZ-o9_HUMeMTSQicvjIg=X4GwEru-SJ9DRnCxhze-aw=aN-OTm6qpDxNIXC86mUeowDuZe9O-NeCWHJdSvrVsYA=d2pZwZ8xKsFLHxQ0YNiM4itJjUHzgE0ibHNu7v28mXM=
>>  
>>
>> 2. To remove tempest tests from the grenade job
>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__review.openstack.org_-23_c_460810_=DwIGaQ=LFYZ-o9_HUMeMTSQicvjIg=X4GwEru-SJ9DRnCxhze-aw=aN-OTm6qpDxNIXC86mUeowDuZe9O-NeCWHJdSvrVsYA=07__zljUdvdtD_K5ltoKwdjaBwrs0fYJKaXSr93AAiU=
>>  
>>
>>
>>>
>>> I proposed a patch to Tempest to start the deprecation counter for Heat /
>>> orchestration related
>>> configuration items in Tempest [4], and I would like to make sure that
>>> all tests and the service client
>>> either find a new home outside of Tempest, or are removed, by the end the
>>> Pike cycle at the latest.
>>>
>>> Heat has in-tree integration tests and Gabbi based API tests, but I don't
>>> know if those provide
>>> enough coverage to replace the tests on Tempest side.
>>>
>>
>> Yes, the heat gabbi api tests do not yet have the same coverage as the
>> tempest tree api tests (lacks tests using nova, neutron and swift
>> resources),  but I think that should not stop us from *not* running the
>> tempest tests in the grenade job.
>>
>> I also don't know if the tempest tree heat tests are used by any other
>> upstream/downstream jobs. We could surely add more tests to bridge the gap.
>>
>> Also, It's possible to run the heat integration tests (we've enough
>> coverage there) with tempest plugin after doing some initial setup, as we do
>> in all our dsvm gate jobs.
>>
>>> It would propose to move tests and client to a Tempest plugin owned /
>>> maintained by
>>> the Heat team, so that the Heat team can have full flexibility in
>>> consolidating their integration
>>> tests. For Murano and Daisycloud - and any other team that may want to
>>> use the Heat service
>>> client in their tests, even if the client is removed from Tempest, it
>>> would still be available via
>>> the Heat Tempest plugin. As long as the plugin implements the service
>>> client interface,
>>> the Heat service client will register automatically in the service client
>>> ma

Re: [openstack-dev] [qa][heat][murano][daisycloud] Removing Heat support from Tempest

2017-04-29 Thread Ghanshyam Mann 
On Fri, Apr 28, 2017 at 5:47 PM, Andrea Frittoli
 wrote:
>
>
> On Fri, Apr 28, 2017 at 10:29 AM Rabi Mishra  wrote:
>>
>> On Thu, Apr 27, 2017 at 3:55 PM, Andrea Frittoli
>>  wrote:
>>>
>>> Dear stackers,
>>>
>>> starting in the Liberty cycle Tempest has defined a set of projects which
>>> are in scope for direct
>>> testing in Tempest [0]. The current list includes keystone, nova, glance,
>>> swift, cinder and neutron.
>>> All other projects can use the same Tempest testing infrastructure (or
>>> parts of it) by taking advantage
>>> the Tempest plugin and stable interfaces.
>>>
>>> Tempest currently hosts a set of API tests as well as a service client
>>> for the Heat project.
>>> The Heat service client is used by the tests in Tempest, which run in
>>> Heat gate as part of the grenade
>>> job, as well as in the Tempest gate (check pipeline) as part of the
>>> layer4 job.
>>> According to code search [3] the Heat service client is also used by
>>> Murano and Daisycore.
>>
>>
>> For the heat grenade job, I've proposed two patches.
>>
>> 1. To run heat tree gabbi api tests as part of grenade 'post-upgrade'
>> phase
>>
>> https://review.openstack.org/#/c/460542/
>>
>> 2. To remove tempest tests from the grenade job
>>
>> https://review.openstack.org/#/c/460810/
>>
>>
>>>
>>> I proposed a patch to Tempest to start the deprecation counter for Heat /
>>> orchestration related
>>> configuration items in Tempest [4], and I would like to make sure that
>>> all tests and the service client
>>> either find a new home outside of Tempest, or are removed, by the end the
>>> Pike cycle at the latest.
>>>
>>> Heat has in-tree integration tests and Gabbi based API tests, but I don't
>>> know if those provide
>>> enough coverage to replace the tests on Tempest side.
>>>
>>
>> Yes, the heat gabbi api tests do not yet have the same coverage as the
>> tempest tree api tests (lacks tests using nova, neutron and swift
>> resources),  but I think that should not stop us from *not* running the
>> tempest tests in the grenade job.
>>
>> I also don't know if the tempest tree heat tests are used by any other
>> upstream/downstream jobs. We could surely add more tests to bridge the gap.
>>
>> Also, It's possible to run the heat integration tests (we've enough
>> coverage there) with tempest plugin after doing some initial setup, as we do
>> in all our dsvm gate jobs.
>>
>>> It would propose to move tests and client to a Tempest plugin owned /
>>> maintained by
>>> the Heat team, so that the Heat team can have full flexibility in
>>> consolidating their integration
>>> tests. For Murano and Daisycloud - and any other team that may want to
>>> use the Heat service
>>> client in their tests, even if the client is removed from Tempest, it
>>> would still be available via
>>> the Heat Tempest plugin. As long as the plugin implements the service
>>> client interface,
>>> the Heat service client will register automatically in the service client
>>> manager and be available
>>> for use as today.
>>>
>>
>> if I understand correctly, you're proposing moving the existing tempest
>> tests and service clients to a separate repo managed by heat team. Though
>> that would be collective decision, I'm not sure that's something I would
>> like to do. To start with we may look at adding some of the missing pieces
>> in heat tree itself.
>
>
> I'm proposing to move tests and the service client outside of tempest to a
> new home.
>
> I also suggested that the new home could be a dedicate repo, since that
> would allow you to maintain the
> current branchless nature of those tests. A more detailed discussion about
> the topic can be found
> in the corresponding proposed queens goal [5],
>
> Using a dedicated repo *is not* a precondition for moving tests and service
> client out of Tempest.
>
> andrea
>

Other than grenade and layer4 job on Tempest, do we run heat tests
anywhere else? With Rabi's proposed changes on grenade, If tempest
tests are not going to run on gate any more and heat team is going to
improve the heat in-tree integration tests coverage, then we can just
remove tests and corresponding config options as Andrea proposed.

Otherwise, IMO it should be done with separate repo to avoid more work
once proposed TC goal is approved. But we should make decision on
tempest heat tests now as its been long time those are maintained in
Tempest.

> [5] https://review.openstack.org/#/c/369749/
>
>>
>>
>>> Andrea Frittoli (andreaf)
>>>
>>> [0]
>>> https://docs.openstack.org/developer/tempest/test_removal.html#tempest-scope
>>> [1] https://docs.openstack.org/developer/tempest/plugin.html
>>> [2] https://docs.openstack.org/developer/tempest/library.html
>>> [3]
>>> http://codesearch.openstack.org/?q=self.orchestration_client=nope==
>>> [4] https://review.openstack.org/#/c/456843/
>>>
>>>
>>> __
>>> OpenStack

Re: [openstack-dev] [qa][heat][murano][daisycloud] Removing Heat support from Tempest

2017-04-28 Thread Andrea Frittoli 
On Fri, Apr 28, 2017 at 10:29 AM Rabi Mishra  wrote:

> On Thu, Apr 27, 2017 at 3:55 PM, Andrea Frittoli <
> andrea.fritt...@gmail.com> wrote:
>
>> Dear stackers,
>>
>> starting in the Liberty cycle Tempest has defined a set of projects which
>> are in scope for direct
>> testing in Tempest [0]. The current list includes keystone, nova, glance,
>> swift, cinder and neutron.
>> All other projects can use the same Tempest testing infrastructure (or
>> parts of it) by taking advantage
>> the Tempest plugin and stable interfaces.
>>
>> Tempest currently hosts a set of API tests as well as a service client
>> for the Heat project.
>> The Heat service client is used by the tests in Tempest, which run in
>> Heat gate as part of the grenade
>> job, as well as in the Tempest gate (check pipeline) as part of the
>> layer4 job.
>> According to code search [3] the Heat service client is also used by
>> Murano and Daisycore.
>>
>
> For the heat grenade job, I've proposed two patches.
>
> 1. To run heat tree gabbi api tests as part of grenade 'post-upgrade' phase
>
> https://review.openstack.org/#/c/460542/
>
> 2. To remove tempest tests from the grenade job
>
> https://review.openstack.org/#/c/460810/
>
>
>
>> I proposed a patch to Tempest to start the deprecation counter for Heat /
>> orchestration related
>> configuration items in Tempest [4], and I would like to make sure that
>> all tests and the service client
>> either find a new home outside of Tempest, or are removed, by the end the
>> Pike cycle at the latest.
>>
>> Heat has in-tree integration tests and Gabbi based API tests, but I don't
>> know if those provide
>> enough coverage to replace the tests on Tempest side.
>>
>>
> Yes, the heat gabbi api tests do not yet have the same coverage as the
> tempest tree api tests (lacks tests using nova, neutron and swift
> resources),  but I think that should not stop us from *not* running the
> tempest tests in the grenade job.
>
> I also don't know if the tempest tree heat tests are used by any other
> upstream/downstream jobs. We could surely add more tests to bridge the gap.
>
> Also, It's possible to run the heat integration tests (we've enough
> coverage there) with tempest plugin after doing some initial setup, as we
> do in all our dsvm gate jobs.
>
> It would propose to move tests and client to a Tempest plugin owned /
>> maintained by
>> the Heat team, so that the Heat team can have full flexibility in
>> consolidating their integration
>> tests. For Murano and Daisycloud - and any other team that may want to
>> use the Heat service
>> client in their tests, even if the client is removed from Tempest, it
>> would still be available via
>> the Heat Tempest plugin. As long as the plugin implements the service
>> client interface,
>> the Heat service client will register automatically in the service client
>> manager and be available
>> for use as today.
>>
>>
> if I understand correctly, you're proposing moving the existing tempest
> tests and service clients to a separate repo managed by heat team. Though
> that would be collective decision, I'm not sure that's something I would
> like to do. To start with we may look at adding some of the missing pieces
> in heat tree itself.
>

I'm proposing to move tests and the service client outside of tempest to a
new home.

I also suggested that the new home could be a dedicate repo, since that
would allow you to maintain the
current branchless nature of those tests. A more detailed discussion about
the topic can be found
in the corresponding proposed queens goal [5],

Using a dedicated repo *is not* a precondition for moving tests and service
client out of Tempest.

andrea

[5] https://review.openstack.org/#/c/369749/


>
> Andrea Frittoli (andreaf)
>>
>> [0]
>> https://docs.openstack.org/developer/tempest/test_removal.html#tempest-scope
>> [1] https://docs.openstack.org/developer/tempest/plugin.html
>> [2] https://docs.openstack.org/developer/tempest/library.html
>> [3]
>> http://codesearch.openstack.org/?q=self.orchestration_client=nope==
>>
>> [4] https://review.openstack.org/#/c/456843/
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
>
> --
> Regards,
> Rabi Mishra
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

Re: [openstack-dev] [qa][heat][murano][daisycloud] Removing Heat support from Tempest

2017-04-28 Thread Rabi Mishra 
On Thu, Apr 27, 2017 at 3:55 PM, Andrea Frittoli 
wrote:

> Dear stackers,
>
> starting in the Liberty cycle Tempest has defined a set of projects which
> are in scope for direct
> testing in Tempest [0]. The current list includes keystone, nova, glance,
> swift, cinder and neutron.
> All other projects can use the same Tempest testing infrastructure (or
> parts of it) by taking advantage
> the Tempest plugin and stable interfaces.
>
> Tempest currently hosts a set of API tests as well as a service client for
> the Heat project.
> The Heat service client is used by the tests in Tempest, which run in Heat
> gate as part of the grenade
> job, as well as in the Tempest gate (check pipeline) as part of the layer4
> job.
> According to code search [3] the Heat service client is also used by
> Murano and Daisycore.
>

For the heat grenade job, I've proposed two patches.

1. To run heat tree gabbi api tests as part of grenade 'post-upgrade' phase

https://review.openstack.org/#/c/460542/

2. To remove tempest tests from the grenade job

https://review.openstack.org/#/c/460810/



> I proposed a patch to Tempest to start the deprecation counter for Heat /
> orchestration related
> configuration items in Tempest [4], and I would like to make sure that all
> tests and the service client
> either find a new home outside of Tempest, or are removed, by the end the
> Pike cycle at the latest.
>
> Heat has in-tree integration tests and Gabbi based API tests, but I don't
> know if those provide
> enough coverage to replace the tests on Tempest side.
>
>
Yes, the heat gabbi api tests do not yet have the same coverage as the
tempest tree api tests (lacks tests using nova, neutron and swift
resources),  but I think that should not stop us from *not* running the
tempest tests in the grenade job.

I also don't know if the tempest tree heat tests are used by any other
upstream/downstream jobs. We could surely add more tests to bridge the gap.

Also, It's possible to run the heat integration tests (we've enough
coverage there) with tempest plugin after doing some initial setup, as we
do in all our dsvm gate jobs.

It would propose to move tests and client to a Tempest plugin owned /
> maintained by
> the Heat team, so that the Heat team can have full flexibility in
> consolidating their integration
> tests. For Murano and Daisycloud - and any other team that may want to use
> the Heat service
> client in their tests, even if the client is removed from Tempest, it
> would still be available via
> the Heat Tempest plugin. As long as the plugin implements the service
> client interface,
> the Heat service client will register automatically in the service client
> manager and be available
> for use as today.
>
>
if I understand correctly, you're proposing moving the existing tempest
tests and service clients to a separate repo managed by heat team. Though
that would be collective decision, I'm not sure that's something I would
like to do. To start with we may look at adding some of the missing pieces
in heat tree itself.

Andrea Frittoli (andreaf)
>
> [0] https://docs.openstack.org/developer/tempest/test_remova
> l.html#tempest-scope
> [1] https://docs.openstack.org/developer/tempest/plugin.html
> [2] https://docs.openstack.org/developer/tempest/library.html
> [3] http://codesearch.openstack.org/?q=self.orchestration_
> client=nope==
> [4] https://review.openstack.org/#/c/456843/
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Regards,
Rabi Mishra
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [qa][heat][murano][daisycloud] Removing Heat support from Tempest

2017-04-27 Thread Andrea Frittoli 
Dear stackers,

starting in the Liberty cycle Tempest has defined a set of projects which
are in scope for direct
testing in Tempest [0]. The current list includes keystone, nova, glance,
swift, cinder and neutron.
All other projects can use the same Tempest testing infrastructure (or
parts of it) by taking advantage
the Tempest plugin and stable interfaces.

Tempest currently hosts a set of API tests as well as a service client for
the Heat project.
The Heat service client is used by the tests in Tempest, which run in Heat
gate as part of the grenade
job, as well as in the Tempest gate (check pipeline) as part of the layer4
job.
According to code search [3] the Heat service client is also used by Murano
and Daisycore.

I proposed a patch to Tempest to start the deprecation counter for Heat /
orchestration related
configuration items in Tempest [4], and I would like to make sure that all
tests and the service client
either find a new home outside of Tempest, or are removed, by the end the
Pike cycle at the latest.

Heat has in-tree integration tests and Gabbi based API tests, but I don't
know if those provide
enough coverage to replace the tests on Tempest side.

It would propose to move tests and client to a Tempest plugin owned /
maintained by
the Heat team, so that the Heat team can have full flexibility in
consolidating their integration
tests. For Murano and Daisycloud - and any other team that may want to use
the Heat service
client in their tests, even if the client is removed from Tempest, it would
still be available via
the Heat Tempest plugin. As long as the plugin implements the service
client interface,
the Heat service client will register automatically in the service client
manager and be available
for use as today.

Andrea Frittoli (andreaf)

[0]
https://docs.openstack.org/developer/tempest/test_removal.html#tempest-scope
[1] https://docs.openstack.org/developer/tempest/plugin.html
[2] https://docs.openstack.org/developer/tempest/library.html
[3]
http://codesearch.openstack.org/?q=self.orchestration_client=nope==

[4] https://review.openstack.org/#/c/456843/
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev