On Wed, Feb 14, 2018 at 2:09 PM, Brenda J. Butler <b...@mojatatu.com> wrote: > Signed-off-by: Brenda J. Butler <b...@mojatatu.com>
Acked-by: Lucas Bates <luc...@mojatatu.com> > --- > tools/testing/selftests/tc-testing/README | 173 > +++++++++++++++++++++++++--- > tools/testing/selftests/tc-testing/TODO.txt | 25 +++- > 2 files changed, 179 insertions(+), 19 deletions(-) > > diff --git a/tools/testing/selftests/tc-testing/README > b/tools/testing/selftests/tc-testing/README > index 970ff294fec8..3a0336782d2d 100644 > --- a/tools/testing/selftests/tc-testing/README > +++ b/tools/testing/selftests/tc-testing/README > @@ -14,11 +14,11 @@ REQUIREMENTS > > * The kernel must have network namespace support > > -* The kernel must have veth support available, as a veth pair is created > +* The kernel must have veth support available, as a veth pair is created > prior to running the tests. > > -* All tc-related features must be built in or available as modules. > - To check what is required in current setup run: > +* All tc-related features being tested must be built in or available as > + modules. To check what is required in current setup run: > ./tdc.py -c > > Note: > @@ -44,10 +44,13 @@ using the -p option when running tdc: > RUNNING TDC > ----------- > > -To use tdc, root privileges are required. tdc will not run otherwise. > +To use tdc, root privileges are required. This is because the > +commands being tested must be run as root. The code that enforces > +execution by root uid has been moved into a plugin (see PLUGIN > +ARCHITECTURE, below). > > -All tests are executed inside a network namespace to prevent conflicts > -within the host. > +If nsPlugin is linked, all tests are executed inside a network > +namespace to prevent conflicts within the host. > > Running tdc without any arguments will run all tests. Refer to the section > on command line arguments for more information, or run: > @@ -59,6 +62,33 @@ output captured from the failing test will be printed > immediately following > the failed test in the TAP output. > > > +OVERVIEW OF TDC EXECUTION > +------------------------- > + > +One run of tests is considered a "test suite" (this will be refined in the > +future). A test suite has one or more test cases in it. > + > +A test case has four stages: > + > + - setup > + - execute > + - verify > + - teardown > + > +The setup and teardown stages can run zero or more commands. The setup > +stage does some setup if the test needs it. The teardown stage undoes > +the setup and returns the system to a "neutral" state so any other test > +can be run next. These two stages require any commands run to return > +success, but do not otherwise verify the results. > + > +The execute and verify stages each run one command. The execute stage > +tests the return code against one or more acceptable values. The > +verify stage checks the return code for success, and also compares > +the stdout with a regular expression. > + > +Each of the commands in any stage will run in a shell instance. > + > + > USER-DEFINED CONSTANTS > ---------------------- > > @@ -70,23 +100,132 @@ executed as part of the test. More will be added as > test cases require. > Example: > $TC qdisc add dev $DEV1 ingress > > +The NAMES values are used to substitute into the commands in the test cases. > + > > COMMAND LINE ARGUMENTS > ---------------------- > > Run tdc.py -h to see the full list of available arguments. > > --p PATH Specify the tc executable located at PATH to be used on > this > - test run > --c Show the available test case categories in this test file > --c CATEGORY Run only tests that belong to CATEGORY > --f FILE Read test cases from the JSON file named FILE > --l [CATEGORY] List all test cases in the JSON file. If CATEGORY is > - specified, list test cases matching that category. > --s ID Show the test case matching ID > --e ID Execute the test case identified by ID > --i Generate unique ID numbers for test cases with no existing > - ID number > +usage: tdc.py [-h] [-p PATH] [-D DIR [DIR ...]] [-f FILE [FILE ...]] > + [-c [CATG [CATG ...]]] [-e ID [ID ...]] [-l] [-s] [-i] [-v] > + [-d DEVICE] [-n NS] [-V] > + > +Linux TC unit tests > + > +optional arguments: > + -h, --help show this help message and exit > + -p PATH, --path PATH The full path to the tc executable to use > + -v, --verbose Show the commands that are being run > + -d DEVICE, --device DEVICE > + Execute the test case in flower category > + > +selection: > + select which test cases: files plus directories; filtered by categories > + plus testids > + > + -D DIR [DIR ...], --directory DIR [DIR ...] > + Collect tests from the specified directory(ies) > + (default [tc-tests]) > + -f FILE [FILE ...], --file FILE [FILE ...] > + Run tests from the specified file(s) > + -c [CATG [CATG ...]], --category [CATG [CATG ...]] > + Run tests only from the specified category/ies, or if > + no category/ies is/are specified, list known > + categories. > + -e ID [ID ...], --execute ID [ID ...] > + Execute the specified test cases with specified IDs > + > +action: > + select action to perform on selected test cases > + > + -l, --list List all test cases, or those only within the > + specified category > + -s, --show Display the selected test cases > + -i, --id Generate ID numbers for new test cases > + > +netns: > + options for nsPlugin(run commands in net namespace) > + > + -n NS, --namespace NS > + Run commands in namespace NS > + > +valgrind: > + options for valgrindPlugin (run command under test under Valgrind) > + > + -V, --valgrind Run commands under valgrind > + > + > +PLUGIN ARCHITECTURE > +------------------- > + > +There is now a plugin architecture, and some of the functionality that > +was in the tdc.py script has been moved into the plugins. > + > +The plugins are in the directory plugin-lib. The are executed from > +directory plugins. Put symbolic links from plugins to plugin-lib, > +and name them according to the order you want them to run. > + > +Example: > + > +bjb@bee:~/work/tc-testing$ ls -l plugins > +total 4 > +lrwxrwxrwx 1 bjb bjb 27 Oct 4 16:12 10-rootPlugin.py -> > ../plugin-lib/rootPlugin.py > +lrwxrwxrwx 1 bjb bjb 25 Oct 12 17:55 20-nsPlugin.py -> > ../plugin-lib/nsPlugin.py > +-rwxr-xr-x 1 bjb bjb 0 Sep 29 15:56 __init__.py > + > +The plugins are a subclass of TdcPlugin, defined in TdcPlugin.py and > +must be called "SubPlugin" so tdc can find them. They are > +distinguished from each other in the python program by their module > +name. > + > +This base class supplies "hooks" to run extra functions. These hooks are as > follows: > + > +pre- and post-suite > +pre- and post-case > +pre- and post-execute stage > +adjust-command (runs in all stages and receives the stage name) > + > +The pre-suite hook receives the number of tests and an array of test ids. > +This allows you to dump out the list of skipped tests in the event of a > +failure during setup or teardown stage. > + > +The pre-case hook receives the ordinal number and test id of the current > test. > + > +The adjust-command hook receives the stage id (see list below) and the > +full command to be executed. This allows for last-minute adjustment > +of the command. > + > +The stages are identified by the following strings: > + > + - pre (pre-suite) > + - setup > + - command > + - verify > + - teardown > + - post (post-suite) > + > + > +To write a plugin, you need to inherit from TdcPlugin in > +TdcPlugin.py. To use the plugin, you have to put the > +implementation file in plugin-lib, and add a symbolic link to it from > +plugins. It will be detected at run time and invoked at the > +appropriate times. There are a few examples in the plugin-lib > +directory: > + > + - rootPlugin.py: > + implements the enforcement of running as root > + - nsPlugin.py: > + sets up a network namespace and runs all commands in that namespace > + - valgrindPlugin.py > + runs each command in the execute stage under valgrind, > + and checks for leaks. > + This plugin will output an extra test for each test in the test file, > + one is the existing output as to whether the test passed or failed, > + and the other is a test whether the command leaked memory or not. > + (This one is a preliminary version, it may not work quite right yet, > + but the overall template is there and it should only need tweaks.) > > > ACKNOWLEDGEMENTS > diff --git a/tools/testing/selftests/tc-testing/TODO.txt > b/tools/testing/selftests/tc-testing/TODO.txt > index 6a266d811a78..c40698557e2f 100644 > --- a/tools/testing/selftests/tc-testing/TODO.txt > +++ b/tools/testing/selftests/tc-testing/TODO.txt > @@ -5,6 +5,27 @@ tc Testing Suite To-Do list: > > - Add support for multiple versions of tc to run successively > > -- Improve error messages when tdc aborts its run > +- Improve error messages when tdc aborts its run. Partially done - still > + need to better handle problems in pre- and post-suite. > > -- Allow tdc to write its results to file > +- Use python logger module for debug/verbose output > + > +- Allow tdc to write its results to file. > + Maybe use python logger module for this too. > + > +- A better implementation of the "hooks". Currently, every plugin > + will attempt to run a function at every hook point. Could be > + changed so that plugin __init__ methods will register functions to > + be run in the various predefined times. Then if a plugin does not > + require action at a specific point, no penalty will be paid for > + trying to run a function that will do nothing. > + > +- Proper exception handling - make an exception class and use it > + > +- a TestCase class, for easier testcase handling, searching, comparison > + > +- a TestSuite class > + and a way to configure a test suite, > + to automate running multiple "test suites" with different requirements > + > +- super simple test case example using ls, touch, etc > -- > 2.15.1 >
