On Wed, 16 Sep 2020 15:02:58 -0400 Cleber Rosa <cr...@redhat.com> wrote:
> Hi Avocado community, > > This is an invitation to a conversation about Avocado's present and > future. > > If you missed Avocado's latest LTS release (82.0 \o/), these are the > two documents you may want to look at: > > * https://avocado-framework.readthedocs.io/en/82.0/releases/lts/82_0.html > > * https://avocado-framework.readthedocs.io/en/82.0/releases/82_0.html > > There are many co-related topics, so instead of firing multiple > e-mails, I'll try to address all of them here, while using section > names to keep individual topics in check. Feel free to comment on the > section that matters to you and ignore others. > > 69.x LTS > ======== > > The general rule that the previous LTS version will be maintained for > a minimum of 6 months following another LTS release. Following that > rule, we would be maintaining the 69.x LTS series until March 2021. > > One important characteristic of the 69.x series is that it supports > Python 2, which has been declared EOL on Dec 2019. So users stuck > with Python 2 can only use 69.x LTS. In theory, the March 2021 date > far exceeds the Python 2 EOL date, so we suppose that it's a safe > date. +1 > Avocado-VT and Avocado Compatibility > ==================================== > > Avocado-VT is Avocado's primary "customer" and the two share a common > origin, and one retro feeds improvements into the other. Having said > that, a lot of energy has been spent into keeping Avocado-VT compatible > with multiple Avocado of versions. At any given moment, Avocado-VT is > supposed to be compatible with: > > 1. Avocado 69.x LTS > 2. Avocado Latest release (currently 82.0) > 3. Avocado development branch > > With Avocado 82.0 LTS, there's an opportunity to improve this > situation. The first drastic change I'll propose is to drop as many > of the compatibility requirements above, and pin Avocado-VT for the > time being to require compatibility Avocado 82.0 LTS *only*. This > would greatly simplify Avocado-VT's development, as there would be > one and only way to deal with Avocado. Reducing the sync points between avocado and avocado-vt is a desirable goal. At the same time, I don't think this should give developers free reign to intentionally break compatibility. It should always be done intentionally and with a plan to address it somehow. > On the other hand, this could cause: > > 1. Bitrot, where Avocado-VT would at a given point not work with > newer Avocado, say the next LTS version. > > 2. A "lock down", where improvements to "avocado.utils" libraries > would not be easily available to Avocado-VT tests. > > Avoiding bitrot > --------------- > > Let's assume that the next LTS release is version 95.0. The worst > possible scenario would be "waking up" and finding out that making > Avocado-VT compatible Avocado 95.0 would be overwhelming and > prohibitive mission. > > One mitigation would be to extend 82.0 LTS life time, but this would > only push the problem further away and not really solve it. > > To avoid such a situation, I propose that a proactive effort is > made every few Avocado releases, with the goal of: > > * evaluate the current compatibility > * plan and develop compatibility tasks +1 > Those should be done not only in light of the *current* Avocado state, > but also in sync with Avocado's planned features. For example, > consider the following release time line: > > * Avocado 83.0 released > * Avocado 84.0 released > * Avocado 85.0 released > * Avocado-VT compatibility evaluated with Avocado 85.0 > - Compatibility with newer Avocado "runner API" identified and > issue created > - Compatibility with newer Avocado "result API" identified and > issue created > * Avocado 86.0 is released > - Compatibility with "runner API" resolved > * Avocado 87.0 is released > - Compatibility with "result API" resolved > * Avocado 88.0 released > * Avocado-VT compatibility evaluated with Avocado 88.0 > ... > * Avocado 94.0 released (pre LTS release) > - All pending compatibility issues are addressed > * Avocado 95.0 released > - Avocado-VT should be compatible with both current 95.0 LTS and 82.x > LTS > > Integration jobs on CI can also keep running with Avocado master, but > in non-gating state ("allow failures" in Travis CI terminology). This > can serve as a "temperature check" for the level of compatibility and > tasks to records as issues and address in the near future. > > Library lock downs > ------------------ > > Users running tests can usually wait a reasonable long time for new > features to land in the Avocado test runner. In contrast, fixes or > new features in utility libraries are usually needed to write a test > and can not wait for too long. > > With the proposal above to bind Avocado-VT to Avocado LTS versions > only, utility libraries present within Avocado that is, > "avocado.utils", would also be limited to the LTS version. This is > not desirable, as it can: > > * Slow down test development > > * Create duplicate copies of utilities: > - within different tests > - across Avocado and Avocado-VT > > Moving all utility libraries into Avocado-VT (such as the one currently > available in Avocado) is also not an option, as users of other tests > (for example avocado-misc-tests) and even Avocado itself need them. > > The most obvious solution is to move the utility libraries to its own > repository. Let's suppose this new repository/module is called > "greenmatter" (my imagination ran wild, please do the same ;). This > would allow for scenarios such as: > > 1. a) Avocado-VT master > b) Avocado LTS (with its builtin current version of "avocado.utils") > c) new tp-qemu tests using newer "greenmatter" utility libraries > > 2. a) avocado-misc-tests master > b) Avocado (either LTS or latest) > c) new tests using newer "greenmatter" utility libraries Adding another repo of utilities creates its own set of problems: now you have to sync across avocado, avocado-vt and "greenmatter". This creates a matrix of combinations that need to be supported and tested, new kinds of defects to be triaged, and new support issues to consider; for example, will you now need to tag the "greenmatter" repo to indicate what it supports? Will the tag be relative to what avocado wants, avocado-vt wants, or (ugh) both? As suggested by Plamen Dimitrov <pdimit...@pevogam.com>, another option would be to move the utilities into avocado itself. This collapses the matrix back to just two, avocado and avocado-vt, and is a much more straightforward configuration to reason about. > Goals of "greenmatter" libraries > -------------------------------- > > "tried, tested and trusted" > ~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > The current utility libraries suffer a big problem. They are either: > > * tested by superficial tests with lots of mocking > * "tested" by the tests that actually have another goal and simply > use the libraries > * are not tested at all > > It's challenging to test the kind of libraries in "virttest.utils_*" > or in "avocado.utils" because they require specific environments, and > can be destructive. For instance, not many people will trust > (rightfully so) running tests on their local machines. > > On the other hand, Avocado is in the business of: > > 1) Running tests isolated (see the nrunner spanwers) > 2) Checking, fulfilling and caching test requirements (see BP002) > > The idea is to leverage and extend Avocado's features so that every > single new utility library in this new repository gets real test > coverage. Examples: > > 1. a) "avocado.utils.lv_utils" becomes "greenmatter.lv" > b) "greenmater.lv" gets a "tests/test_lv.py" module > c) "tests/test_lv.py" describes its requirements, such as: > - A software package (from the distribution): lvm2 > - A specific isolation level for the spawner: vm > - An empty block device of around 10G > > 2. a) "avocado.utils.distro" becomes "greenmatter.distro" > b) "tests/test_distro.py:Distro.test_fedora" describe its requirements: > - Operating system: fedora:32 > c) "tests/test_distro.py:Distro.test_centos" describe its requirements: > - Operating system: centos:8 > > For example #2, the requirement for "fedora:32" could be fulfilled by > the current environment, and the spawner selected could end up being > "process". In that case, the requirement for "centos:8" would need > to be fulfilled by another spawner, such as a container or vm based > spawner. > > The resulting test coverage for each module should be generated and > presented automatically, such as it's currently being done with the > (per release) coverage for the "avocado.utils.vmimage" library: > > > https://avocado-framework.readthedocs.io/en/82.0/guides/writer/libs/vmimage.html#supported-images > > It should be possible for a user to "shop" for a library, such as > "lv", and find out that in which environments it has been tested on. I see a huge explosion of a test and support matrix, of which I am not fond. -Jeff > Common API "look and feel" > ~~~~~~~~~~~~~~~~~~~~~~~~~~ > > Getting the design of APIs right is pretty hard, but here we have a > unique change of declaring a common set of guidelines from the start > and enforcing as libraries are added. > > It's expected that a "BluePrint" would be written defining, among other > things: > > * how modules should be named > * how classes should be named > * how functions should be named if they return information > * how functions should be named if they check or assert some condition > * how functions should behave if running on a platform that doesn't > apply (say "distro" is found running on a BSD or Windows system) > > Concurrent usage and migration plan > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > It should be possible to keep using the libraries available at > Avocado-VT's "virtest.utils_*" and Avocado's "avocado.utils" > concurrently with the libraries migrated to the new repository. > > Suppose your test looks like: > > from avocado.utils import cpu > from avocado.utils import lv_utils > from avocado.utils import distro > from virttest import utils_net > > Once "greenmatter 1.0" gets release, let's say with nothing but "lv" and > "distro" libraries. You project can add a new requirement: > > greenmatter==1.0 > > And the next version of a test can look like: > > from avocado.utils import cpu > from virttest import utils_net > from greenmatter import lv > from greenmatter import distro > > Gradually, as more modules are migrated, the requirement can eventually be: > > greenmatter==12.0 > > And the test can look like: > > from greenmatter import cpu > from greenmatter import net > from greenmatter import lv > from greenmatter import distro > > > That's all for now. All feedback is appreciated. > > Thanks! > -- > Cleber Rosa > [ Sr Software Engineer - Virtualization Team - Red Hat ] > [ Avocado Test Framework - avocado-framework.github.io ] > [ 7ABB 96EB 8B46 B94D 5E0F E9BB 657E 8D33 A5F2 09F3 ]
