The tempest-full job used to run API and scenario tests concurrently,
and if you go back far enough I think it also ran slow tests.
Sometime in the last year or so, the full job was changed to run the
scenario tests in serial and exclude the slow tests altogether. So the
API tests run concurrently first, and then the scenario tests run in
serial. During that change, some other tests were identified as 'slow'
and marked as such, meaning they don't get run in the normal
tempest-full job.
There are some valuable scenario tests marked as slow, however, like the
only encrypted volume testing we have in tempest is marked slow so it
doesn't get run on every change for at least nova.
There is only one job that can be run against nova changes which runs
the slow tests but it's in the experimental queue so people forget to
run it.
As a test, I've proposed a nova-slow job [1] which only runs the slow
tests and only the compute API and scenario tests. Since there currently
no compute API tests marked as slow, it's really just running slow
scenario tests. Results show it runs 37 tests in about 37 minutes [2].
The overall job runtime was 1 hour and 9 minutes, which is on average
less than the tempest-full job. The nova-slow job is also running
scenarios that nova patches don't actually care about, like the neutron
IPv6 scenario tests.
My question is, should we make this a generic tempest-slow job which can
be run either in the integrated-gate or at least in nova/neutron/cinder
consistently (I'm not sure if there are slow tests for just keystone or
glance)? I don't know if the other projects already have something like
this that they gate on. If so, a nova-specific job for nova changes is
fine for me.
[1] https://review.openstack.org/#/c/567697/
[2]
http://logs.openstack.org/97/567697/1/check/nova-slow/bedfafb/job-output.txt.gz#_2018-05-10_23_46_47_588138
--
Thanks,
Matt
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
