On 08/10/2016 11:33 AM, Luigi Toscano wrote:
On Wednesday, 10 August 2016 17:00:36 CEST Ihar Hrachyshka wrote:
Luigi Toscano <ltosc...@redhat.com> wrote:
On Wednesday, 10 August 2016 10:42:41 CEST Ben Swartzlander wrote:
On 08/10/2016 04:33 AM, Duncan Thomas wrote:
So I tried to get into helping with the cinder stable tree for a while,
and while I wasn't very successful (lack of time and an inability to
convince my employer it should be a priority), one thing I did notice it
that much of the breakage seemed to come from outside cinder - many of
the libraries we depend on make backwards incompatible changes by
accident, for example. Would it be possible to have a long-term-support
branch where we pinned the max version of everything for the gate, pips
and devtstack? I'd have thought (and I'm very willing to be corrected)
that would make the stable gate, well, stable, such that it required far
less work to keep it able to run a basic devstack test plus unit tests.
Does that sound at all sane?
A big source of problems IMO is that tempest doesn't have stable
branches. We use the master branch of tempest to test stable branches of
other projects, and tempest regularly adds new features. This guarantees
instability if you rely on tempest anywhere in your gate (and cinder
does).
Orthogonal to the discussion, but: this is not due to the lack of stable
branch, but that part of the Tempest API are not stable yet. This is being
addressed right now (in scope for Newton).
Once the Tempest stable API are used, no breakages should happen.
Well, it’s only partially true. But what happens when you add a new test to
tempest/master? It gets executed on all branches, and maybe some of them
are failing it. We can argue that it’s probably a bug revealed, but it
nevertheless requires attention from stable maintainers to solve.
The new test should work on all support branches. As tester I find a lot of
advantages of maintaining a unified set of tests than fighting with backports
of tests.
I'm sure it makes YOUR life easier to not have to deal with backports.
The rest of us who do maintain stable branches though don't appreciate
it when tempest adds a new feature or a new dependency which breaks the
stable gate jobs. This happens a few times each release, and tends to
get fixed within a day or two, which is barely tolerable.
The fact that it happens at all though says that we're "doing it wrong"
w.r.t. testing of stable branches, and completely explains why we find
it so challenging to support stuff more than 12 months old. If
everything we used had stable branches and proper dependency management,
then it would easy to keep gate job running for years.
There are mechanisms to skip tests based on the cloud capabilities.
So this should not be an issue, and if a bug is found that should definitely
be viewed as a good thing.
It's not new tests that cause the problem, because those would be easy
to skip. It's changes to tempest core which force changes elsewhere.
-Ben
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
