Re: [Avocado-devel] RFC: Guidelines for categorizing tests
On Fri, 2017-05-19 at 10:28 -0400, Cleber Rosa wrote: > > On 05/18/2017 01:07 AM, Satheesh Rajendran wrote: > > > > On Wed, 2017-05-17 at 17:49 -0400, Cleber Rosa wrote: > > > > > > Introduction > > > > > > > > > Avocado allows users to select tests to run based on free form > > > "tags". > > > These tags are given as "docstring directives", that is, special > > > entries on a class or function docstring. > > > > > This would be very helpful and becomes easy way to > > create a dynamic testsuite based on needs. > > +1 > I'm not sure if it was clear enough, but this functionality already > exists in Avocado: > > http://avocado-framework.readthedocs.io/en/50.0/WritingTests.html#cat > egorizing-tests > > This proposal is about proposing a set of guidelines for using that > feature. > Oh Sure, need to explore on that, Thanks :-) > > > > > > > > As a user of an Avocado based test suite, I'd see value in not > > > **having** > > > to look at all the test tags before realizing that to not run > > > tests > > > that > > > require "super user" permissions I should run:: > > > > > > $ avocado run test.py --filter-by-tags=-root > > > > > > Instead of:: > > > > > > $ avocado run test.py --filter-by-tags=-privileged > > > > > > Not even that, by having different tests as part of the same job, > > > the following very odd sequence of command line options may be > > > needed:: > > > > > > $ avocado run test.py test2.py --filter-by-tags=-root,- > > > privileged > > > > > > So the goal here is to let users familiar with a given Avocado > > > based > > > test, to have fair expectations when running another Avocado > > > based > > > tests. > > > > > > This was initially going to be a documentation update, but I felt > > > that > > > it was not fair to make a formal proposal without without some > > > initial > > > brainstorming. > > > > > > Proposal > > > > > > > > > To set the tone for my proposal, I'd like to make most things > > > simple > > > and easy, while allowing for "everything else" to be doable. > > > > > > My general impression is that higher level information about the > > > test > > > itself and its requirements are going to be the most commonly > > > used > > > tags, so they must be easily set. Some examples follow. > > > > > > Simple (standalone) tags > > > > > > > > > Tags by functional area: > > > > > > * cpu - Exercises a system's CPU > > > * net - Exercises a system's network devices or networking stack > > > * storage - Exercises a system's local storage > > > * fs - Exercises a system's file system > > > > > > Tags by architecture: > > > > > > * x86_64 - Requires a x86_64 architecture > > > * ppc64 - Requries a ppc64 > > > > > > Tags by access privileges: > > > > > > * privileged - requires the test to be run with the most > > > privileged, > > > unrestricted privileges. For Linux systems, this usually > > > means > > > the > > > root account > > > > > > Composed (key:value) tags > > > - > > > > > > The more specific tags can be achieved by composing a predefined > > > key > > > with a value. For instance, to tag a test as needing a specific > > > CPU flag: > > > > > > * cpu_flag:vmx > > > > > > Or a specific PCI device: > > > > > > * pci_id:8086:08b2 > > > > > > Or even a software package: > > > > > > * package:gcc > > > > > > Or a package group altogether: > > > > > > * package_group:development-tools > > > > > > Some examples > > > - > > > > > > * ``cpu,x86_64`` - The test exercises the CPU and requires a > > > ``x86_64`` based platform > > > > > > * ``net,privileged,pci_id:14e4:1657`` - The test exercises > > > either a > > > network device or the network stack, needs super user > > > privileges > > > and a "Broadcom Limited NetXtreme BCM5719 Gigabit Ethernet > > > PCIe > > > (rev 01)" device. > > > > > > Looking at test tags > > > > > > > > > Currently, there's no way to actually list the test tags from the > > > command line, alongside the test name themselves. An RFE for > > > fixing > > > this has been filed at https://trello.com/c/NXrbKEJC . > > > > > > Do users have to provide all the ``--filter-by-tags`` themselves? > > > = > > > > > > The test runner can certainly help here, getting system > > > information > > > when the job starts, and feeding them to the filtering. This is > > > yet > > > another reason why coming up with a good set of guidelines for > > > tagging > > > tests is important. > > > > > > In some ways, this can be seen similar to a dependency resolution > > > mechanism for tests, only that at this point it will not resolve > > > the > > > requirements. It will only filter out tests that can't (or > > > shouldn't) > > > be loaded on the current system. > > > > > > Effectively, instead of many in-tests checks, and many > > >
Re: [Avocado-devel] RFC: Guidelines for categorizing tests
This feature looks to be great and I acknowledge this RFC, we have to consider two things, 1. with respect to avocado-vt, whether tags mentioned should filter the tests based on host or guest. 2. Negative test scenarios should not be filtered based on this tags. by considering these in design for implementation helps us to achieve this feature well. Regards, Bala On Wed, May 17, 2017 at 05:49:36PM -0400, Cleber Rosa wrote: > Introduction > > > Avocado allows users to select tests to run based on free form "tags". > These tags are given as "docstring directives", that is, special > entries on a class or function docstring. > > As a user of an Avocado based test suite, I'd see value in not **having** > to look at all the test tags before realizing that to not run tests that > require "super user" permissions I should run:: > > $ avocado run test.py --filter-by-tags=-root > > Instead of:: > > $ avocado run test.py --filter-by-tags=-privileged > > Not even that, by having different tests as part of the same job, > the following very odd sequence of command line options may be > needed:: > > $ avocado run test.py test2.py --filter-by-tags=-root,-privileged > > So the goal here is to let users familiar with a given Avocado based > test, to have fair expectations when running another Avocado based > tests. > > This was initially going to be a documentation update, but I felt that > it was not fair to make a formal proposal without without some initial > brainstorming. > > Proposal > > > To set the tone for my proposal, I'd like to make most things simple > and easy, while allowing for "everything else" to be doable. > > My general impression is that higher level information about the test > itself and its requirements are going to be the most commonly used > tags, so they must be easily set. Some examples follow. > > Simple (standalone) tags > > > Tags by functional area: > > * cpu - Exercises a system's CPU > * net - Exercises a system's network devices or networking stack > * storage - Exercises a system's local storage > * fs - Exercises a system's file system > > Tags by architecture: > > * x86_64 - Requires a x86_64 architecture > * ppc64 - Requries a ppc64 > > Tags by access privileges: > > * privileged - requires the test to be run with the most privileged, >unrestricted privileges. For Linux systems, this usually means the >root account > > Composed (key:value) tags > - > > The more specific tags can be achieved by composing a predefined key > with a value. For instance, to tag a test as needing a specific > CPU flag: > > * cpu_flag:vmx > > Or a specific PCI device: > > * pci_id:8086:08b2 > > Or even a software package: > > * package:gcc > > Or a package group altogether: > > * package_group:development-tools > > Some examples > - > > * ``cpu,x86_64`` - The test exercises the CPU and requires a >``x86_64`` based platform > > * ``net,privileged,pci_id:14e4:1657`` - The test exercises either a >network device or the network stack, needs super user privileges >and a "Broadcom Limited NetXtreme BCM5719 Gigabit Ethernet PCIe >(rev 01)" device. > > Looking at test tags > > > Currently, there's no way to actually list the test tags from the > command line, alongside the test name themselves. An RFE for fixing > this has been filed at https://trello.com/c/NXrbKEJC . > > Do users have to provide all the ``--filter-by-tags`` themselves? > = > > The test runner can certainly help here, getting system information > when the job starts, and feeding them to the filtering. This is yet > another reason why coming up with a good set of guidelines for tagging > tests is important. > > In some ways, this can be seen similar to a dependency resolution > mechanism for tests, only that at this point it will not resolve the > requirements. It will only filter out tests that can't (or shouldn't) > be loaded on the current system. > > Effectively, instead of many in-tests checks, and many SKIPs/CANCELs, > the system information can be loaded once, and the only relevant tests > will be part of the tests suite. > > A list of the tests that were filtered out from the job test suite can > certainly be a useful part of the job results. > > As in every RFC, feedback is extremely welcome! > > > -- > Cleber Rosa > [ Sr Software Engineer - Virtualization Team - Red Hat ] > [ Avocado Test Framework - avocado-framework.github.io ] > [ 7ABB 96EB 8B46 B94D 5E0F E9BB 657E 8D33 A5F2 09F3 ] >
Re: [Avocado-devel] RFC: Guidelines for categorizing tests
Hi Cleber, I agree the proposal. Please find some of my comments in-line. On 2017-05-18 03:19, Cleber Rosa wrote: Introduction Avocado allows users to select tests to run based on free form "tags". These tags are given as "docstring directives", that is, special entries on a class or function docstring. I like this kind of a proposal :-) As a user of an Avocado based test suite, I'd see value in not **having** to look at all the test tags before realizing that to not run tests that require "super user" permissions I should run:: $ avocado run test.py --filter-by-tags=-root Instead of:: $ avocado run test.py --filter-by-tags=-privileged Not even that, by having different tests as part of the same job, the following very odd sequence of command line options may be needed:: $ avocado run test.py test2.py --filter-by-tags=-root,-privileged So the goal here is to let users familiar with a given Avocado based test, to have fair expectations when running another Avocado based tests. This was initially going to be a documentation update, but I felt that it was not fair to make a formal proposal without without some initial brainstorming. Proposal To set the tone for my proposal, I'd like to make most things simple and easy, while allowing for "everything else" to be doable. My general impression is that higher level information about the test itself and its requirements are going to be the most commonly used tags, so they must be easily set. Some examples follow. Simple (standalone) tags Tags by functional area: * cpu - Exercises a system's CPU * net - Exercises a system's network devices or networking stack * storage - Exercises a system's local storage * fs - Exercises a system's file system Tags by architecture: * x86_64 - Requires a x86_64 architecture * ppc64 - Requries a ppc64 Tags by access privileges: * privileged - requires the test to be run with the most privileged, unrestricted privileges. For Linux systems, this usually means the root account How are these tags going to be differentiated, ie, tag names that can be common across types/areas. Examples, if already on your mind, can help here. Composed (key:value) tags - The more specific tags can be achieved by composing a predefined key with a value. For instance, to tag a test as needing a specific CPU flag: * cpu_flag:vmx Or a specific PCI device: * pci_id:8086:08b2 Or even a software package: * package:gcc Or a package group altogether: * package_group:development-tools Some examples - * ``cpu,x86_64`` - The test exercises the CPU and requires a ``x86_64`` based platform * ``net,privileged,pci_id:14e4:1657`` - The test exercises either a network device or the network stack, needs super user privileges and a "Broadcom Limited NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)" device. I would be interested in seeing how we can implement this. Looking at test tags Currently, there's no way to actually list the test tags from the command line, alongside the test name themselves. An RFE for fixing this has been filed at https://trello.com/c/NXrbKEJC . Do users have to provide all the ``--filter-by-tags`` themselves? = The test runner can certainly help here, getting system information when the job starts, and feeding them to the filtering. This is yet another reason why coming up with a good set of guidelines for tagging tests is important. In some ways, this can be seen similar to a dependency resolution mechanism for tests, only that at this point it will not resolve the requirements. It will only filter out tests that can't (or shouldn't) be loaded on the current system. Effectively, instead of many in-tests checks, and many SKIPs/CANCELs, the system information can be loaded once, and the only relevant tests will be part of the tests suite. A list of the tests that were filtered out from the job test suite can certainly be a useful part of the job results. As in every RFC, feedback is extremely welcome! -- Regards Narasimhan V
[Avocado-devel] RFC: Guidelines for categorizing tests
Introduction Avocado allows users to select tests to run based on free form "tags". These tags are given as "docstring directives", that is, special entries on a class or function docstring. As a user of an Avocado based test suite, I'd see value in not **having** to look at all the test tags before realizing that to not run tests that require "super user" permissions I should run:: $ avocado run test.py --filter-by-tags=-root Instead of:: $ avocado run test.py --filter-by-tags=-privileged Not even that, by having different tests as part of the same job, the following very odd sequence of command line options may be needed:: $ avocado run test.py test2.py --filter-by-tags=-root,-privileged So the goal here is to let users familiar with a given Avocado based test, to have fair expectations when running another Avocado based tests. This was initially going to be a documentation update, but I felt that it was not fair to make a formal proposal without without some initial brainstorming. Proposal To set the tone for my proposal, I'd like to make most things simple and easy, while allowing for "everything else" to be doable. My general impression is that higher level information about the test itself and its requirements are going to be the most commonly used tags, so they must be easily set. Some examples follow. Simple (standalone) tags Tags by functional area: * cpu - Exercises a system's CPU * net - Exercises a system's network devices or networking stack * storage - Exercises a system's local storage * fs - Exercises a system's file system Tags by architecture: * x86_64 - Requires a x86_64 architecture * ppc64 - Requries a ppc64 Tags by access privileges: * privileged - requires the test to be run with the most privileged, unrestricted privileges. For Linux systems, this usually means the root account Composed (key:value) tags - The more specific tags can be achieved by composing a predefined key with a value. For instance, to tag a test as needing a specific CPU flag: * cpu_flag:vmx Or a specific PCI device: * pci_id:8086:08b2 Or even a software package: * package:gcc Or a package group altogether: * package_group:development-tools Some examples - * ``cpu,x86_64`` - The test exercises the CPU and requires a ``x86_64`` based platform * ``net,privileged,pci_id:14e4:1657`` - The test exercises either a network device or the network stack, needs super user privileges and a "Broadcom Limited NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)" device. Looking at test tags Currently, there's no way to actually list the test tags from the command line, alongside the test name themselves. An RFE for fixing this has been filed at https://trello.com/c/NXrbKEJC . Do users have to provide all the ``--filter-by-tags`` themselves? = The test runner can certainly help here, getting system information when the job starts, and feeding them to the filtering. This is yet another reason why coming up with a good set of guidelines for tagging tests is important. In some ways, this can be seen similar to a dependency resolution mechanism for tests, only that at this point it will not resolve the requirements. It will only filter out tests that can't (or shouldn't) be loaded on the current system. Effectively, instead of many in-tests checks, and many SKIPs/CANCELs, the system information can be loaded once, and the only relevant tests will be part of the tests suite. A list of the tests that were filtered out from the job test suite can certainly be a useful part of the job results. As in every RFC, feedback is extremely welcome! -- Cleber Rosa [ Sr Software Engineer - Virtualization Team - Red Hat ] [ Avocado Test Framework - avocado-framework.github.io ] [ 7ABB 96EB 8B46 B94D 5E0F E9BB 657E 8D33 A5F2 09F3 ] signature.asc Description: OpenPGP digital signature
