On Wednesday, 10 August 2016 12:00:41 CEST Ben Swartzlander wrote: > On 08/10/2016 11:33 AM, Luigi Toscano wrote: > > On Wednesday, 10 August 2016 17:00:36 CEST Ihar Hrachyshka wrote: > >> Luigi Toscano <ltosc...@redhat.com> wrote: > >>> On Wednesday, 10 August 2016 10:42:41 CEST Ben Swartzlander wrote: > >>>> On 08/10/2016 04:33 AM, Duncan Thomas wrote: > >>>>> So I tried to get into helping with the cinder stable tree for a > >>>>> while, > >>>>> and while I wasn't very successful (lack of time and an inability to > >>>>> convince my employer it should be a priority), one thing I did notice > >>>>> it > >>>>> that much of the breakage seemed to come from outside cinder - many of > >>>>> the libraries we depend on make backwards incompatible changes by > >>>>> accident, for example. Would it be possible to have a > >>>>> long-term-support > >>>>> branch where we pinned the max version of everything for the gate, > >>>>> pips > >>>>> and devtstack? I'd have thought (and I'm very willing to be corrected) > >>>>> that would make the stable gate, well, stable, such that it required > >>>>> far > >>>>> less work to keep it able to run a basic devstack test plus unit > >>>>> tests. > >>>>> > >>>>> Does that sound at all sane? > >>>> > >>>> A big source of problems IMO is that tempest doesn't have stable > >>>> branches. We use the master branch of tempest to test stable branches > >>>> of > >>>> other projects, and tempest regularly adds new features. This > >>>> guarantees > >>>> instability if you rely on tempest anywhere in your gate (and cinder > >>>> does). > >>> > >>> Orthogonal to the discussion, but: this is not due to the lack of stable > >>> branch, but that part of the Tempest API are not stable yet. This is > >>> being > >>> addressed right now (in scope for Newton). > >>> Once the Tempest stable API are used, no breakages should happen. > >> > >> Well, it’s only partially true. But what happens when you add a new test > >> to > >> tempest/master? It gets executed on all branches, and maybe some of them > >> are failing it. We can argue that it’s probably a bug revealed, but it > >> nevertheless requires attention from stable maintainers to solve. > > > > The new test should work on all support branches. As tester I find a lot > > of > > advantages of maintaining a unified set of tests than fighting with > > backports of tests. > > I'm sure it makes YOUR life easier to not have to deal with backports. > The rest of us who do maintain stable branches though don't appreciate > it when tempest adds a new feature or a new dependency which breaks the > stable gate jobs. This happens a few times each release, and tends to > get fixed within a day or two, which is barely tolerable.
Again, if a new test uncover the bug, why blame the test and not the bug? If you don't want to fix bugs in stable branches, why keep them? I don't understand... > > The fact that it happens at all though says that we're "doing it wrong" > w.r.t. testing of stable branches, and completely explains why we find > it so challenging to support stuff more than 12 months old. If > everything we used had stable branches and proper dependency management, > then it would easy to keep gate job running for years. Again, if a test uncover a bugs, let fix the bug. > > > There are mechanisms to skip tests based on the cloud capabilities. > > So this should not be an issue, and if a bug is found that should > > definitely be viewed as a good thing. > > It's not new tests that cause the problem, because those would be easy > to skip. It's changes to tempest core which force changes elsewhere. As I mentioned before, that's something temporary, due to the change of scope of Tempest (the core six, the plugin systems) and the stabilization of that's ongoing for Newton, so *right now*. We will forget quickly about all of this once its fixed (more or less like the old complains about Neutron - rememeber?). Patches always welcome to speed up the Tempest API stabilization. -- Luigi __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev