On 2017-05-18 20:18, Cleber Rosa wrote:
On 05/18/2017 12:22 AM, Narasimhan V wrote:
Hi Cleber,
I agree the proposal.
Please find some of my comments in-line.
Hi Narasimhan,
Thanks for the feedback. My responses are inline.
On 2017-05-18 03:19, Cleber Rosa wrote:
Introduction
============
Avocado allows users to select tests to run based on free form
"tags".
These tags are given as "docstring directives", that is, special
entries on a class or function docstring.
I like this kind of a proposal :-)
I'm not sure if it was clear enough, but this functionality already
exists in Avocado:
http://avocado-framework.readthedocs.io/en/50.0/WritingTests.html#categorizing-tests
This proposal is about proposing a set of guidelines for using that
feature.
Thanks for pointing this out.
As a user of an Avocado based test suite, I'd see value in not
**having**
to look at all the test tags before realizing that to not run tests
that
require "super user" permissions I should run::
$ avocado run test.py --filter-by-tags=-root
Instead of::
$ avocado run test.py --filter-by-tags=-privileged
Not even that, by having different tests as part of the same job,
the following very odd sequence of command line options may be
needed::
$ avocado run test.py test2.py --filter-by-tags=-root,-privileged
So the goal here is to let users familiar with a given Avocado based
test, to have fair expectations when running another Avocado based
tests.
This was initially going to be a documentation update, but I felt
that
it was not fair to make a formal proposal without without some
initial
brainstorming.
Proposal
========
To set the tone for my proposal, I'd like to make most things simple
and easy, while allowing for "everything else" to be doable.
My general impression is that higher level information about the test
itself and its requirements are going to be the most commonly used
tags, so they must be easily set. Some examples follow.
Simple (standalone) tags
------------------------
Tags by functional area:
* cpu - Exercises a system's CPU
* net - Exercises a system's network devices or networking stack
* storage - Exercises a system's local storage
* fs - Exercises a system's file system
Tags by architecture:
* x86_64 - Requires a x86_64 architecture
* ppc64 - Requries a ppc64
Tags by access privileges:
* privileged - requires the test to be run with the most privileged,
unrestricted privileges. For Linux systems, this usually means
the
root account
How are these tags going to be differentiated, ie, tag names that can
be
common across types/areas.
Examples, if already on your mind, can help here.
Well, that's exactly the point of this proposal. If users have access
to two different test repos, and both are related to hardware/software
testing, they can expect that by choosing only tests with "cpu" tags,
they will be exercising their Central Processing Units.
Conversely, if a test repo is concerned about testing Hospital
procedures (this is very hypothetical), they may choose to use "cpu" as
a shorthand for "Care of Patients with Urgency". They would be going
against the general Avocado guidelines, users may expect those tests to
be about "Central Processing Units", and be frustrated.
Does this example make sense?
Are we looking at tree structure for tags, as this gets more use cases ?
Composed (key:value) tags
-------------------------
The more specific tags can be achieved by composing a predefined key
with a value. For instance, to tag a test as needing a specific
CPU flag:
* cpu_flag:vmx
Or a specific PCI device:
* pci_id:8086:08b2
Or even a software package:
* package:gcc
Or a package group altogether:
* package_group:development-tools
Some examples
-------------
* ``cpu,x86_64`` - The test exercises the CPU and requires a
``x86_64`` based platform
* ``net,privileged,pci_id:14e4:1657`` - The test exercises either a
network device or the network stack, needs super user privileges
and a "Broadcom Limited NetXtreme BCM5719 Gigabit Ethernet PCIe
(rev 01)" device.
I would be interested in seeing how we can implement this.
We are actually free to start using this right away. Actually, it'd be
better to start doing this once we settle on a set of guidelines.
The point is that tags are already accepted by docstring statements,
and
recognized by `--filter-by-tags`. The next evolutionary step is on the
"Do users have to provide all the ``--filter-by-tags`` themselves?"
section bellow.
So, are we going to tag a test as privileged based on the tag
``superuser`` ?
And, how do we use the key value pairs (pci_id14e4:1657) to run / skip
the tests ?
Some examples there would help as well.
- Cleber.
Looking at test tags
====================
Currently, there's no way to actually list the test tags from the
command line, alongside the test name themselves. An RFE for fixing
this has been filed at https://trello.com/c/NXrbKEJC .
Do users have to provide all the ``--filter-by-tags`` themselves?
=================================================================
The test runner can certainly help here, getting system information
when the job starts, and feeding them to the filtering. This is yet
another reason why coming up with a good set of guidelines for
tagging
tests is important.
In some ways, this can be seen similar to a dependency resolution
mechanism for tests, only that at this point it will not resolve the
requirements. It will only filter out tests that can't (or
shouldn't)
be loaded on the current system.
Effectively, instead of many in-tests checks, and many SKIPs/CANCELs,
the system information can be loaded once, and the only relevant
tests
will be part of the tests suite.
A list of the tests that were filtered out from the job test suite
can
certainly be a useful part of the job results.
As in every RFC, feedback is extremely welcome!
--
Regards
Narasimhan V
