Intro ===== The N(ext) Runner is an experiment started within the Avocado project, with the overall goal of solving some fundamental problems with the current architecture, and opening up the possibility of smoother implementation of a number of advanced features and use case.
For a complete list of the limitations and use cases, please refer to:: https://avocado-framework.readthedocs.io/en/79.0/future/core/nrunner.html#motivation On the last Avocado release (79.0), a comparison of the *current* and *N(ext) Runner* components currently co-existing in Avocado was posted. It's a suggested reading for people interested in following this discussion and contributing to this RFC:: https://avocado-framework.readthedocs.io/en/79.0/future/core/nrunner.html#current-and-n-ext-runner-components-of-avocado In addition the explanation linked above, it's also a good idea to explain to go over the various development efforts in place, which to some extent, contribute or influence or require some parts of the N(ext) Runner architecture. Job API ======= Avocado, as a "test automation framework", provides a native API that users can use to write tests from scratch. These are, quite simply, the "Test APIs":: https://avocado-framework.readthedocs.io/en/79.0/api/test/avocado.html#test-apis But, we have noticed that a large part of the test automation process in the real world also happens outside of individual tests. To facilitate the integration of the environment and the execution of a number of tests, it was natural to look at the layer that encompasses more responsibilities. In Avocado terminology, a "Job" is an unique execution of a number of tests. An Avocado Job is traditionally created/executed by means of the ``avocado run`` command. The goal of the "Job API" work is to augment the expressiveness of jobs, by removing the obligatory command line interface, and giving users the possibility of using more flexible job descriptions. With the "Job API", users should be able to write advanced jobs in a number ofcomplementary ways, listed here from least complex to most complex: * Providing a Python Dictionary configuration to the Job() class, which can be easily loaded, say, from a JSON file * Writing Python code that creates the Python dictionary configuration at run time, and gives that to the Job() class * Subclassing the core Job() implementation, customizing specific phases of the job workflow, such as the creation of the "test suite", that is the definition of the tests to be executed by the job. Because Avocado is not interested in creating a new programming language, most of that should be done with standard programming languages, such as Avocado's primary language: pure Python. Currently ongoing development ----------------------------- There has been three major types of developments around the Job API: 1. Setting some examples "job files", and making use of custom jobs on Avocado's own code. Users can find example job files in "examples/jobs" directory in the source code:: https://github.com/avocado-framework/avocado/tree/master/examples/jobs And jobs that are used during the Avocado release process:: https://github.com/avocado-framework/avocado/tree/master/jobs 2. Introducing and porting Avocado's code to new combined settings module. This should allow the same experience for users of the command line tool giving option as command line arguments, users of the Job API providing the configuration as a dictionary, or users of both command line and Job API settings options with configuration files. 3. Making sure that features that work while running jobs on the command line (via "avocado run"), also work the same way in custom jobs. Recently a number of issues with output not being created when running custom jobs were fixed, but there are certainly many more to be found. Relationship with N(ext) Runner ------------------------------- The N(ext) Runner will only be considered "feature complete" when it is completely integrated into an Avocado Job. This means that the job files discussed on the previous section item #1 should behave the same with the current runner, or the "nrunner" implementation. The items #2 and #3, should, as much as possible be finished *before* that, given that not doing so will cast a number of questions on who the culprit for a bug is. Any bug will lead to the question: is it a Job API bug or a N(ext) Runner bug? The mechanism that a Job uses to select a runner implementation already is in place. Once can see the runner implementations available by running:: $ avocado plugins ... Plugins that run test suites on a job (runners): docker *DEPRECATED* Runs on a Docker (or compatible) container nrunner *EXPERIMENTAL* nrunner based implementation of job compliant runner remote *DEPRECATED* Runs on a remote machine using SSH runner The conventional test runner vm *DEPRECATED* Runs on a VM using SSH And one can set a custom runner by setting the "test_runner" key in an Avocado, as it can be seen in "examples/jobs/nrunner.py":: #!/usr/bin/env python3 import sys from avocado.core.job import Job config = { 'test_runner': 'nrunner', 'run.references': [ 'selftests/unit/test_resolver.py', 'selftests/functional/test_argument_parsing.py', '/bin/true', '/bin/false', ], } with Job(config) as j: sys.exit(j.run()) The ability to set the runner as part of the command line, has also been proposed (and be merged by the time one read this):: https://github.com/avocado-framework/avocado/pull/3816 But setting the configuration in a configuration file and running all tests should give a quick indication of the number of issues with the N(ext) Runner. Requirements Resolver ===================== The requirements resolver is an abstraction and evolution of the current "asset fetcher" utility. Currently ongoing development ----------------------------- There's a blue print document describing the motivation, specification, etc:: https://avocado-framework.readthedocs.io/en/79.0/blueprints/BP002.html Relationship with N(ext) Runner ------------------------------- While the ideas and features of the Requirements Resolver are not bound to the N(ext) Runner, its implementation must be based on one architecture. Given the current plans to phase out and remove the current runner implementation, it makes sense to implement it based on the N(ext) Runner architecture, but because of the lack of a complete implementation or specification, this can prove to be challenging. One of the goals of this RFC, is to look at the proposal for the N(ext) Runner pending development tasks, and discuss an architecture that facilitates the implementation of the Requirements Resolver. Some early brainstorming about how some of the N(ext) Runner bits may be (or may NOT be) useful in the context of the Requirements Resolver implementation happened here: https://github.com/avocado-framework/avocado/pull/3756 But as noticed there, a lot of the N(ext) Runner missing bits themselves have to be defined. Avocado-VT ========== Avocado-VT, in its previous Virt-Test and Autotest names incarnations and names, were the projects that gave birth to Avocado, so we'll always be fond of them. Having said that, we have to be careful with the promise of compatibility for the N(ext) Runner and current Avocado-VT, specially about most of the Avocado-VT workflows and expectations. For instance, Avocado-VT tests will, as a general rule, access guest image files in an installation wide location, say "~/avocado/data/avocado-vt/images/f30-x86_64.qcow2". There's even a special plugin, "vt-joblock" whose goal is to prevent multiple jobs for running simultaneously and prevent corruption of such images. Typical Avocado-VT usage with the libvirt backend, will usually require or suggest a fixed set of tests. For instance, the first test in a job may be one that "defines" a libvirt guest domain, the following tests may be tests that operate on the guest domain defined by the first test, and the final test is usually one that cleans up the libvirt domain guest definition. Now, thinking of N(ext) Runner, one of its main selling points is the ability to run tests in parallel. While this is a benefit to most users, it's actually a problem to current Avocado-VT tests and workflows. Corruption and general test errors and failures would be almost guaranteed to occur. I can think of two action items to manage Avocado-VT, which are complimentary: 1. Have an LTS release, say version 82.0, with the current runner implementation still supported. This LTS version can be maintained for twice the time of previous LTS releases (~3 years), and the possibility to extend it even further. 2. Attempt to migrate Avocado-VT infrascture code and tests to behave well within the assumptions of the N(ext) Runner architecture. Other? ====== If you're an Avocado user or developer, and you think there are other topics that may be impacted by the N(ext) Runner architecture and the deprecation and removal of the current runner architecture, please join the discussion and raise your point. Next ==== Following this, I'll be posting an architectural blue print proposal for the missing pieces of the N(ext) Runner. Feedback on the proposal is much appreciated, and it's worth reminding everyone that there's no unworthy suggestion or question. Thanks! - Cleber.
signature.asc
Description: PGP signature
