Re: [openstack-dev] [neutron] Skip-only tests

[John Davidge]

John Schwarz wrote:

>Hi all,
>
>The Neutron community has recently started contributing Tempest
>scenario tests in the Neutron tree and I'd like to discuss a general
>issue we're hitting. Mainly, we're talking about required hardware for
>some features and/or VM images that support more features, which makes
>it hard (or impossible) to test certain features at the gate
>end-to-end. Specifically, tests that require SR-IOV or OVS-DPDK, and
>tests relating to VLAN Aware VMs.
>[Š]

Resurrecting this thread with another example where this would be useful:

IPv6 Prefix Delegation

This feature still lacks end-to-end testing because it depends upon having
a compatible DHCPv6 server installed and running with an appropriate
config, and specifically dibbler[1] to act as the DHCPv6 client.

I¹d be interested in assisting with this effort.

John

[1] https://github.com/tomaszmrugalski/dibbler




Rackspace Limited is a company registered in England & Wales (company 
registered number 03897010) whose registered office is at 5 Millington Road, 
Hyde Park Hayes, Middlesex UB3 4AZ. Rackspace Limited privacy policy can be 
viewed at www.rackspace.co.uk/legal/privacy-policy - This e-mail message may 
contain confidential or privileged information intended for the recipient. Any 
dissemination, distribution or copying of the enclosed material is prohibited. 
If you receive this transmission in error, please notify us immediately by 
e-mail at ab...@rackspace.com and delete the original message. Your cooperation 
is appreciated.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Skip-only tests

[Rodrigo Duarte]

On Thu, Sep 8, 2016 at 12:44 PM, Joe Hakim Rahme 
wrote:

> On Thu, 2016-09-08 at 10:36 +0200, Andreas Jaeger wrote:
> >
> > Toni, sorry, I wasn't clear. I'm not advocating for this, I wanted to
> > bring Joe into this discussion to see whether the solution that John
> > suggested will help them as well - I wanted to broaden the scope of
> > this since it's not only neutron that faces this.
> >
>
> Hi,
>
> Just to be clear, my intention with the repo I'm trying to create is
> to hold tests that are not acceptable in upstream Tempest by design
> (mainly whitebox testing). I think what John was referring to (correct
> me if I'm wrong) are tests that are acceptable Tempest tests but we
> lack the required infrastrcuture (hardware, images, ...) to run them
> in the gate.
>
> I realize now that the name I chose "nova-tempest-staging" might be
> misleading and I think it should be changed. Maybe something like
> "tempest-whitebox-plugin" might be more descriptive?
>

This sounds like a generic repo, not specific or tied to service. Not being
specific sounds much better to me.

I liked the idea and would be willing to help out in the effort, but agree
it needs more thorough discussion.


>
> As for the topic of skip-only tests, I think that it'd be a shame to
> reject valid and valuable tests simply because the gate is not equipped
> to run them. There are use cases for Tempest outside of the upstream
> gate, so marking certain tests as valid-but-skipped-in-the-gate seems
> like an acceptable solution to me.
>
> (I'm not a maintainer of the Tempest repo so please take my opinion
> with a grain of salt).
>
> Cheers,
> Joe
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Rodrigo Duarte Sousa
Senior Quality Engineer @ Red Hat
MSc in Computer Science
http://rodrigods.com
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Skip-only tests

[Joe Hakim Rahme]

On Thu, 2016-09-08 at 10:36 +0200, Andreas Jaeger wrote:
> 
> Toni, sorry, I wasn't clear. I'm not advocating for this, I wanted to
> bring Joe into this discussion to see whether the solution that John
> suggested will help them as well - I wanted to broaden the scope of
> this since it's not only neutron that faces this.
> 

Hi,

Just to be clear, my intention with the repo I'm trying to create is
to hold tests that are not acceptable in upstream Tempest by design
(mainly whitebox testing). I think what John was referring to (correct
me if I'm wrong) are tests that are acceptable Tempest tests but we
lack the required infrastrcuture (hardware, images, ...) to run them
in the gate.

I realize now that the name I chose "nova-tempest-staging" might be
misleading and I think it should be changed. Maybe something like
"tempest-whitebox-plugin" might be more descriptive?

As for the topic of skip-only tests, I think that it'd be a shame to
reject valid and valuable tests simply because the gate is not equipped
to run them. There are use cases for Tempest outside of the upstream
gate, so marking certain tests as valid-but-skipped-in-the-gate seems
like an acceptable solution to me.

(I'm not a maintainer of the Tempest repo so please take my opinion
with a grain of salt).

Cheers,
Joe

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Skip-only tests

[Andreas Jaeger]

On 2016-09-08 10:23, Toni Freger wrote:
> On Thu, Sep 8, 2016 at 10:52 AM, Andreas Jaeger  wrote:
>>
>> This reminds me of this review for a nova-tempest-staging repository:
>> https://review.openstack.org/366091
>>
>> Joe seems to face similar challenges that you articulate, John,
>>
>> Andreas
>>
> 
> Hi Andreas,
> 
> I understand other projects are going in the direction of using a
> different repo for these skip-only tests. This seems
> counter-productive for us for the following reasons:

Toni, sorry, I wasn't clear. I'm not advocating for this, I wanted to
bring Joe into this discussion to see whether the solution that John
suggested will help them as well - I wanted to broaden the scope of this
since it's not only neutron that faces this.

If you review my comments on that linked review, you see that I'm not in
favor of a separate repository at all.

> 1. We believe that eventually some of these pre-conditions John
> mentioned will be solved, thus requiring even more patches to migrate
> tests back to the Neutron repos.
> 2. Since the infra is still maturing (and since new features are
> written all the time), contributing tests will eventually require
> multiple patches in multiple repos. For example, adding a test for
> VLAN Aware VMs will probably require making changes to the current
> tempest infrastructure (which resides in the Neutron repo), and then
> additional "Depends-On" patches in the new repo.
> 
> These will only slow down the process which can be made fluent if
> tests will be all be done in the same repo as the actual features and
> infra.

Andreas
-- 
 Andreas Jaeger aj@{suse.com,opensuse.org} Twitter: jaegerandi
  SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
   GF: Felix Imendörffer, Jane Smithard, Graham Norton,
   HRB 21284 (AG Nürnberg)
GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Skip-only tests

[Toni Freger]

On Thu, Sep 8, 2016 at 10:52 AM, Andreas Jaeger  wrote:
>
> This reminds me of this review for a nova-tempest-staging repository:
> https://review.openstack.org/366091
>
> Joe seems to face similar challenges that you articulate, John,
>
> Andreas
>

Hi Andreas,

I understand other projects are going in the direction of using a
different repo for these skip-only tests. This seems
counter-productive for us for the following reasons:
1. We believe that eventually some of these pre-conditions John
mentioned will be solved, thus requiring even more patches to migrate
tests back to the Neutron repos.
2. Since the infra is still maturing (and since new features are
written all the time), contributing tests will eventually require
multiple patches in multiple repos. For example, adding a test for
VLAN Aware VMs will probably require making changes to the current
tempest infrastructure (which resides in the Neutron repo), and then
additional "Depends-On" patches in the new repo.

These will only slow down the process which can be made fluent if
tests will be all be done in the same repo as the actual features and
infra.

Thanks,
Toni.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Skip-only tests

[Andreas Jaeger]

On 2016-09-08 09:38, John Schwarz wrote:
> Hi all,
> 
> The Neutron community has recently started contributing Tempest
> scenario tests in the Neutron tree and I'd like to discuss a general
> issue we're hitting. Mainly, we're talking about required hardware for
> some features and/or VM images that support more features, which makes
> it hard (or impossible) to test certain features at the gate
> end-to-end. Specifically, tests that require SR-IOV or OVS-DPDK, and
> tests relating to VLAN Aware VMs.
> 
> For this reason, we would like to bring up the idea of introducing in
> a new concept to the "testing arena" - skip-only tests. These will be
> tests that will be merged upstream and skipped unless some
> pre-conditions are met (example: "@skip_unless(has_sriov=True)"). The
> key idea here though, is that we don't expect upstream gates to add
> support for these pre-conditions any time soon. Instead, the tests
> will be merged upstream and used on the various downstream gates most
> of us have (with specific hardware/deployment that match our
> respective interests).
> 
> Another way to look at it is: "I want to contribute tests that require
> specific hardware/software/deployments, and I want it to be robust and
> shared amongst the community as per the Open Source way, even though
> it won't be actually run on the upstream gates". Obviously, we would
> continue to investigate ways to "unskip" these kind of tests where
> possible.
> [...]

This reminds me of this review for a nova-tempest-staging repository:
https://review.openstack.org/366091

Joe seems to face similar challenges that you articulate, John,

Andreas
-- 
 Andreas Jaeger aj@{suse.com,opensuse.org} Twitter: jaegerandi
  SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
   GF: Felix Imendörffer, Jane Smithard, Graham Norton,
   HRB 21284 (AG Nürnberg)
GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev