On Thu, Apr 27, 2017 at 5:15 PM, Cleber Rosa <cr...@redhat.com> wrote: > > > On 04/27/2017 04:04 AM, Amador Segundo wrote: >> On Thu, Apr 27, 2017 at 1:07 AM, Cleber Rosa <cr...@redhat.com> wrote: >>> ========================== >>> RFC: Long Term Stability >>> ========================== >>> >>> This RFC contains proposals and clarifications regarding the >>> maintenance and release processes of Avocado. >>> >>> We understand there are multiple teams currently depending on the >>> stability of Avocado and we don't want their work to be disrupted by >>> incompatibilities nor instabilities in new releases. >>> >>> This version is a minor update to previous versions of the same RFC >>> (see `Changelog`_) which drove the release of Avocado 36.0 LTS. The >>> Avocado team has plans for a new LTS release in the near future, so >>> please consider reading and providing feedback on the proposals here. >>> >>> TL;DR >>> ===== >>> >>> We plan to keep the current approach of sprint releases every 3-4 >>> weeks, but we're introducing "Long Term Stability" releases which >>> should be adopted in production environments where users can't keep up >>> with frequent upgrades. >>> >>> Changelog >>> ========= >>> >>> Changes from `Version 3`_: >>> >>> * Converted formatting to REStructuredText >>> * Replaced "me" mentions on version 1 changelog with proper name >>> (Ademar Reis) >>> * Renamed section "Misc Details" to `Deployment Details`_ >>> * Renamed "avocado-vt" to "Avocado-VT" >>> * Start the timeline example with version 36.0 >>> * Be explicit on timeline example that a minor bug did not generate >>> an immediate release >>> >>> Changes from `Version 2`_: >>> >>> * Wording changes on second paragraph ("... nor instabilities...") >>> * Clarified on "Introduction" that change of behavior is introduced >>> between regular releases >>> * Updated distro versions for which official packages are built >>> * Add more clear explanation on official packages on the various >>> hardware platforms >>> * Used more recent version numbers as examples, and the planned >>> new LTS version too >>> * Explain how users can get the LTS version when using tools such as >>> pip >>> * Simplified the timeline example, with examples that will possibly >>> match the future versions and releases >>> * Documented current status of Avocado-VT releases and packages >>> >>> Changes from `Version 1`_: >>> >>> * Changed "Support" to "Stability" and "supported" to "maintained" >>> [Jeff Nelson] >>> * Misc improvements and clarifications in the >>> supportability/stability statements [Jeff Nelson, Ademar Reis] >>> * Fixed a few typos [Jeff Nelson, Ademar Reis] >>> >>> Introduction >>> ============ >>> >>> We make new releases of Avocado every 3-4 weeks on average. In theory >>> at least, we're very careful with backwards compatibility. We test >>> Avocado for regressions and we try to document any issues, so >>> upgrading to a new version should be (again, in theory) safe. >>> >>> But in practice both intended and unintended changes are introduced >>> during development, and both can be frustrating for conservative >>> users. We also understand it's not feasible for users to upgrade >>> Avocado very frequently in a production environment. >>> >>> The objective of this RFC is to clarify our maintenance practices and >>> introduce Long Term Stability (LTS) releases, which are intended to >>> solve, or at least mitigate, these problems. >>> >>> >>> Our definition of maintained, or stable >>> ======================================= >>> >>> First of all, Avocado and its sub-projects are provided 'AS IS' and >>> WITHOUT ANY WARRANTY, as described in the LICENSE file. >>> >>> The process described here doesn't imply any commitments or >>> promises. It's just a set of best practices and recommendations. >>> >>> When something is identified as "stable" or "maintained", it means the >>> development community makes a conscious effort to keep it working and >>> consider reports of bugs and issues as high priorities. Fixes >>> submitted for these issues will also be considered high priorities, >>> although they will be accepted only if they pass the general >>> acceptance criteria for new contributions (design, quality, >>> documentation, testing, etc), at the development team discretion. >>> >>> >>> Maintained projects and platforms >>> ================================= >>> >>> The only maintained project as of today is the Avocado Test Runner, >>> including its APIs and core plugins (the contents of the main avocado >>> git repository). >>> >>> Other projects kept under the "Avocado Umbrella" in github may be >>> maintained by different teams (e.g.: Avocado-VT) or be considered >>> experimental (e.g.: avocado-server and avocado-virt). >>> >>> More about Avocado-VT in its own section further down. >>> >>> As a general rule, fixes and bug reports for Avocado when running in >>> any modern Linux distribution are welcome. >>> >>> But given the limited capacity of the development team, packaged >>> versions of Avocado will be tested and maintained only for the >>> following Linux distributions: >>> >>> * RHEL 7.x (latest) >>> * Fedora (stable releases from the Fedora projects) >>> >>> Currently all packages produced by the Avocado projects are "noarch". >>> That means that they could be installable on any hardware platform. >>> Still, the development team will currently attempt to provide versions >>> that are stable for the following platforms: >>> >>> * x86 >>> * ppc64 >>> * ppc64le >>> >>> Contributions from the community to maintain other platforms and >>> operating systems are very welcome. >>> >>> The lists above may change without prior notice. >>> >>> Avocado Releases >>> ================ >>> >>> The proposal is to have two different types of Avocado releases: >>> >>> Sprint Releases >>> --------------- >>> >>> (This is the model we currently adopt in Avocado) >>> >>> They happen every 3-4 weeks (the schedule is not fixed) and >>> their versions are numbered serially, with decimal digits in >>> the format <major>.<minor>. Examples: 47.0, 48.0, 49.0. Minor >>> releases are rare, but necessary to correct some major issue >>> with the original release (47.1, 47.2, etc). >>> >>> Only the latest Sprint Release is maintained. >>> >>> In Sprint Releases we make a conscious effort to keep backwards >>> compatibility with the previous version (APIs and behavior) and >>> as a general rule and best practice, incompatible changes in >>> Sprint Releases should be documented in the release notes and >>> if possible deprecated slowly, to give users time to adapt >>> their environments. >>> >>> But we understand changes are inevitable as the software >>> evolves and therefore there's no absolute promise for API and >>> behavioral stability. >>> >>> Long Term Stability (LTS) Releases >>> ---------------------------------- >>> >>> LTS releases should happen whenever the team feels the code is >>> stable enough to be maintained for a longer period of time, ideally >>> once or twice per year (no fixed schedule). >>> >>> They should be maintained for 18 months, receiving fixes for major >>> bugs in the form of minor (sub-)releases. With the exception of >>> these fixes, no API or behavior should change in a minor LTS >>> release. >>> >>> They will be versioned just like Sprint Releases, so looking at the >>> version number alone will not reveal the differentiate release >>> process and stability characteristics. >>> >>> In practice each major LTS release will imply in the creation of a >>> git branch where only serious issues affecting users will be >>> fixed. The code in a LTS branch is stable, frozen for new features. >> >> I'd improve the sentence "only serious issues affecting users will be >> fixed". In fact, we "backport" (meaning upstream first) "any issues" >> (not only the serious ones) that are possible to be fixed *and* are >> affecting users. >> > > OK, good point. My suggestion is to use "important issues", because the > definition of a "bug" and a "serious issue" is too relative. We may > identify a bug in LTS that is mostly harmless, and decide to not fix it > because it could require major changes. Anyway, my proposal: > > "In practice each major LTS release will imply in the creation of a git > branch where only important issues affecting users will be fixed, > usually as a backport of a fix initially applied upstream. The code in > a LTS branch is stable, frozen for new features." > > How does that sound?
Yup, looks great. > >> Not sure if it is also worth it to define 'backport'. Depending on the >> code distance between the LTS and the master branch, it can be a >> simple 'cherry-pick' or a completely different fix in terms of code, >> as long as the bug gets fixed. >> > > I'd expect readers to be familiar with the concept of backport. I'm > reluctant about introducing the concept in this document. > >>> >>> Notice that although within a LTS release there's a expectation >>> of stability because the code is frozen, different (major) LTS >>> releases may include changes in behavior, API incompatibilities >>> and new features. The development team will make a considerable >>> effort to minimize and properly document these changes (changes >>> when comparing it to the last major LTS release). >>> >>> Sprint Releases are replaced by LTS releases. I.e., in the cycle >>> when 52.0 (LTS) is released, that's also the version used as a >>> Sprint Release (there's no 52.0 -- non LTS -- in this case). >>> >>> New LTS releases should be done carefully, with ample time for >>> announcements, testing and documentation. It's recommended >>> that one or two sprints are dedicated as preparations for a LTS >>> release, with a Sprint Release serving as a "LTS beta" release. >>> >>> Similarly, there should be announcements about the end-of-life >>> (EOL) of a LTS release once it approaches its 18 months of >>> life. >>> >>> >>> Deployment details >>> ================== >>> >>> Sprint and LTS releases, when packaged, whenever possible, will be >>> preferably distributed through different package channels >>> (repositories). >>> >>> This is possible for repository types such as `YUM/DNF repos`_. In >>> such cases, users can disable the regular channel, and enable the LTS >>> version. A request for the installation of Avocado packages will >>> fetch the latest version available in the enabled repository. If the >>> LTS repository channel is enabled, the packages will receive minor >>> updates (bugfixes only), until a new LTS version is released (roughly >>> every 12 months). >>> >>> If the non-LTS channel is enabled, users will receive updates every >>> 3-4 weeks. >>> >>> On other types of repos such as `PyPI`_ which have no concept of >>> "sub-repos" or "channels", users can request a version smaller than >>> the version that succeeds the current LTS to get the latest LTS >>> (including minor releases). Suppose the current LTS major version is >>> 52, but there have been minor releases 52.1 and 52.2. By running:: >>> >>> pip install 'avocado-framework<53.0' >>> >>> pip provide LTS version 52.2. If 52.3 gets released, they will be >>> automatically deployed instead. When a new LTS is released, users >>> would still get the latest minor release from the 52.0 series, unless >>> they update the version specification. >>> >>> The existence of LTS releases should never be used as an excuse >>> to break a Sprint Release or to introduce gratuitous >>> incompatibilities there. In other words, Sprint Releases should >>> still be taken seriously, just as they are today. >>> >>> >>> Timeline example >>> ================ >>> >>> Assume each sprint is taking 3 weeks. Notice how multiple LTS >>> releases can co-exist before EOL: >>> >>> 36.0 >>> * **LTS** release (the only LTS release available at the time of >>> writing) >>> >>> 37.0 .. 49.0 >>> * sprint releases >>> >>> 50.0 >>> * sprint release >>> * start preparing a LTS release, so 51.0 will be a **beta LTS** >>> >>> 51.0 >>> * sprint release >>> * **beta LTS** release >>> >>> 52.0 >>> * **LTS** release >>> * 52lts branch is created >>> * packages go into LTS repo >>> * both **36.x LTS** and **52.x LTS** maintained from this point on >>> >>> 53.0 >>> * sprint release >>> * bug that also affects 52.0 is found, fix gets added to master and >>> 52lts branches >> >> Fix gets added to master and backported to supported LTS branches (at >> this point, 52lts and 36lts). >> > > Yeah, good point (assuming it also applies to 36lts, as you mentioned on > the other reply). Yes, I think it makes sense to mention that here to make clear that all supported branches will receive the fix, as long as there are affected users. My proposal: 53.0 * sprint release * bug that also affects users of 52.0 and 36.0 is found, fix gets added to master and is backported to 52lts and 36lts branches. 54.0 * sprint release 54.0 **AND** LTS 52.1 release **AND** LTS 36.5 release. * minor bug that also affects users of 52.1 is found, fix gets added to master and is backported to 52lts branch. 55.0 * sprint release 55.0 **AND** LTS 52.2 release * critical bug that affects 52.2 *only* is found, fix gets added to 52lts and 52.3 LTS is immediately released > >>> >>> 54.0 >>> * sprint release 54.0 **AND** LTS 52.1 release >>> * minor bug that also affects 52.0 is found, fix gets added to >>> master and 52lts branches >> >> Fix gets added to master and backported to supported LTS branches (at >> this point, 52lts and 36lts). >> >>> >>> 55.0 >>> * sprint release >>> * critical bug that affects 52.1 *only* is found, fix gets added to >>> 52lts and 52.2 LTS is immediately released >>> >>> 56.0 >>> * sprint release >>> >>> 57.0 >>> * sprint release >>> >>> 58.0 >>> * sprint release >>> >>> 59.0 >>> * sprint release >>> * EOL for **36.x LTS** (18 months since the release of 36.0) >>> >>> Avocado-VT >>> ========== >>> >>> Avocado-VT is an Avocado plugin that allows "VT tests" to be run >>> inside Avocado. It's a third-party project maintained mostly by >>> Engineers from Red Hat QE with assistance from the Avocado team >>> and other community members. >>> >>> It's a general consensus that QE teams use Avocado-VT directly >>> from git, usually following the master branch, which they >>> control. >>> >>> There's no official maintenance or stability statement for >>> Avocado-VT. Even though the upstream community is quite >>> friendly and open to both contributions and bug reports, >>> Avocado-VT is made available without any promises for >>> compatibility or supportability. >>> >>> When packaged and versioned, Avocado-VT rpms should be considered >>> just snapshots, available in packaged form as a convenience to >>> users outside of the Avocado-VT development community. Again, >>> they are made available without any promises of compatibility or >>> stability. >>> >>> * Which Avocado version should be used by Avocado-VT? >>> >>> This is up to the Avocado-VT community to decide, but the >>> current consensus is that to guarantee some stability in >>> production environments, Avocado-VT should stick to a specific >>> LTS release of Avocado. In other words, the Avocado team >>> recommends production users of Avocado-VT not to install Avocado >>> from its master branch or upgrade it from Sprint Releases. >>> >>> Given each LTS release will be maintained for 18 months, it >>> should be reasonable to expect Avocado-VT to upgrade to a new >>> LTS release once a year or so. This process will be done with >>> support from the Avocado team to avoid disruptions, with proper >>> coordination via the avocado mailing lists. >>> >>> In practice the Avocado development team will keep watching >>> Avocado-VT to detect and document incompatibilities, so when >>> the time comes to do an upgrade in production, it's expected >>> that it should happen smoothly. >>> >>> * Will it be possible to use the latest Avocado and Avocado-VT >>> together? >>> >>> Users are welcome to *try* this combination. The Avocado >>> development team itself will do it internally as a way to monitor >>> incompatibilities and regressions. >>> >>> Whenever Avocado is released, a matching versioned snapshot of >>> Avocado-VT will be made. Packages containing those Avocado-VT >>> snapshots, for convenience only, will be made available in the >>> regular Avocado repository. >>> >>> .. _Version 1: >>> https://www.redhat.com/archives/avocado-devel/2016-April/msg00006.html >>> .. _Version 2: >>> https://www.redhat.com/archives/avocado-devel/2016-April/msg00038.html >>> .. _Version 3: >>> https://www.redhat.com/archives/avocado-devel/2017-April/msg00032.html >>> .. _YUM/DNF repos: >>> http://avocado-framework.readthedocs.io/en/latest/GetStartedGuide.html#fedora-from-avocado-s-own-repo >>> .. _PyPI: https://pypi.python.org/pypi >>> >>> >>> -- >>> Cleber Rosa >>> [ Sr Software Engineer - Virtualization Team - Red Hat ] >>> [ Avocado Test Framework - avocado-framework.github.io ] >>> [ 7ABB 96EB 8B46 B94D 5E0F E9BB 657E 8D33 A5F2 09F3 ] >>> > > -- > Cleber Rosa > [ Sr Software Engineer - Virtualization Team - Red Hat ] > [ Avocado Test Framework - avocado-framework.github.io ] > [ 7ABB 96EB 8B46 B94D 5E0F E9BB 657E 8D33 A5F2 09F3 ] >