On Wed, Aug 10, 2016 at 09:52:55AM -0700, Clay Gerrard wrote: > On Wed, Aug 10, 2016 at 7:42 AM, Ben Swartzlander <b...@swartzlander.org> > wrote: > > > > > A big source of problems IMO is that tempest doesn't have stable branches. > > We use the master branch of tempest to test stable branches of other > > projects, and tempest regularly adds new features. > > > > How come not this +1000 just fix this?
Well, mostly because it's actually not a problem and ignores the history on why tempest is branchless. We actually used to do this pre-icehouse and it actually made things much worse. What was happening back then was we didn't have enough activity to keep the stable branches working at all. So we'd go very long periods where nothing actually could land. We also often wedged ourselves where master tempest changed to a point where we couldn't sanely backport a fix to the stable branch. This would often mean that up until right before a stable release things just couldn't land until someone was actually motivated to try and dig us out. But, what more often happened was we had to just disable tempest on the branch, because we didn't have another option. It also turns out that having different tests across a release boundary meant we weren't actually validating that the OpenStack APIs were consistent and worked the same. We had many instances where a projects API just changed between release boundaries, which violates our API consistency and backwards compatibility guidelines. Tempest is about verifying the API and just like an other API client it should work against any OpenStack release. Doing this has been a huge benefit for making things actually work on the stable branches. (in fact just thinking back about how broken everything was all the time back then makes me appreciate it even more) We also test every incoming tempest change on all the stable branches, and nothing can land unless it works on all supported branches. It means we have a consistent and stable api across releases. We do have occasional bugs where a new test or change in tempest triggers a new race in a project's stable branch. But, that's a real bug and normally a fix can be backported.(which is the whole point of doing stable branches) If it can't and the race is bad enough to actively interfere with things, we have a mechanism to skip the test. (but that's normally a last resort) Although, these issues tend to come up pretty infrequently in practice, especially as we slowly ramp up the stability of things over time. FWIW, a lot of these details are covered in the original spec for implementing this: (although it's definitely assumes a bit of prior knowledge about the state of things going on when it was written) http://specs.openstack.org/openstack/qa-specs/specs/tempest/implemented/branchless-tempest.html -Matt Treinish
signature.asc
Description: PGP signature
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev