Hi all, The Neutron community has recently started contributing Tempest scenario tests in the Neutron tree and I'd like to discuss a general issue we're hitting. Mainly, we're talking about required hardware for some features and/or VM images that support more features, which makes it hard (or impossible) to test certain features at the gate end-to-end. Specifically, tests that require SR-IOV or OVS-DPDK, and tests relating to VLAN Aware VMs.
For this reason, we would like to bring up the idea of introducing in a new concept to the "testing arena" - skip-only tests. These will be tests that will be merged upstream and skipped unless some pre-conditions are met (example: "@skip_unless(has_sriov=True)"). The key idea here though, is that we don't expect upstream gates to add support for these pre-conditions any time soon. Instead, the tests will be merged upstream and used on the various downstream gates most of us have (with specific hardware/deployment that match our respective interests). Another way to look at it is: "I want to contribute tests that require specific hardware/software/deployments, and I want it to be robust and shared amongst the community as per the Open Source way, even though it won't be actually run on the upstream gates". Obviously, we would continue to investigate ways to "unskip" these kind of tests where possible. Examples where this would be extremely helpful include (but aren't limited to): 1. SR-IOV - Sure, the Mellanox CI currently tests SR-IOV in VF mode, however this isn't enough; SR-IOV in PF mode isn't checked and additional interesting use-cases (such as QoS + SR-IOV) aren't tested in Mellanox' CI (let alone any other upstream gate). Also, considering the current QoS test - in order to test it specifically with SR-IOV, one must be able to create a port with vnic_type='direct'. 2. VLAN Aware VMs - while tested (or planned to be tested) upstream in the unit/functional/api/fullstack levels, it isn't tested (nor is it planned to be, afaik) in a tempest scenario. The reason is simple: VMs require 802.1Q [1] in order to support VLAN Aware VMs, and it isn't currently in the Cirros image. While we're currently blocking on these tests until the Cirros images are updated, we could easily test the feature using our respective images downstream. This is also a good example where eventually these tests can be un-skipped when the upstream Cirros images are updated. 3. SCTP tests for security groups - also blocked because Cirros doesn't carry a netcat which supports this. ...And so on and so forth. Here's a brief QA section that should include some common questions/complaints: Q: The VLAN Aware VMs issue can be solved by adding Cirros support for 802.1Q - why should we in the mean time merge tests that need to be maintained? A: No one has guarantees when the new Cirros will be released and that it will work with all the other gate jobs and tests. Though we have already started pushing for a new version with support, we have (downstream-wise) other VMs that do carry the support required - we are just missing the tests. Q: So write the tests downstream - why should I care? A: We want to collaborate on the tests upstream so we (and you) can gain the standard benefits of improved and better maintained code. This will also (hopefully) help removing duplication of efforts in the testing scene. Q: So how will we know if a downstream test fails? A: This is a currently open question. One suggestion we had was that any time these skip-only tests are explicitly changed, the author has to show logs of a successful run. While we don't see how this can be relevant to certain trivial project-wide changes (and such changes can be exempt from such proofs), explicit modification of tests, logic or infra that involve these tests will need to prove that they actually work downstream. We are now open for ideas about this idea. [1]: https://en.wikipedia.org/wiki/IEEE_802.1Q -- John Schwarz, Senior Software Engineer, Red Hat. __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev