Module Name: othersrc Committed By: agc Date: Sun May 29 20:47:56 UTC 2016
Added Files: othersrc/external/bsd/human2atf: Makefile README Testspec.template human2atf.1 human2atf.sh Log Message: Add human2atf, a means of generating ATF test scripts through a short test specification file. human2atf is a frontend for the ATF test harness. The idea is to provide an easy/simple way to sepcify tests, and to provide the correct environment for running these test cases. In order to accomplish this, a specfile, (by default Testspec) is given to human2atf, which is a shell script which produces the test Makefile, and t_*.sh shell script used by atf. The Testspec file is itself a shell file fragment. It specifies the names of the files which are to be used in testing, which are typically input files to the command being tested, and expected output files. Care should be taken in naming test sets so that a name that is a shell variable is used. It is advised that individual test cases are grouped into functionally-related test sets. This is not enforced, but is good practice since the tests are used together as a regression test suite, and so changes to utilities may show up in certain parts of the utility being tested, but not in others. Grouping indivual functionally-related test cases into a test set for that functionality is thus a good idea. Some hand-rolled tests in the tree use shell "here" documents, which can interfere with control flow unless used carefully. To avoid this, human2atf encodes its input and expected output files using base64 uuencoding, so that the contents of such files do not affect the way that the human2atf shell script itself works. For a test set, all that need to be specified are: + the testset name + any input and expected output files + the test cases themselves To use a worked example, to enable atf tests for a (possibly fictional) utility called b2e, a Testspec file was drawn up: TESTNAME=b2e TESTSET_1_NAME=basics TESTSET_1_FILES=' 1.in 1.expected 2.in 2.expected ' TESTSET_1_CASE_1="-s eq:0 -o file:1.expected -e empty b2e < 1.in" TESTSET_1_CASE_2="-s eq:0 -o file:2.expected -e empty b2e < 2.in" TESTSET_2_NAME=extended TESTSET_2_FILES=' 3.in 3.expected 4.in 4.expected 5.in 5.expected ' TESTSET_2_CASE_1="-s eq:0 -o file:3.expected -e empty b2e < 3.in" TESTSET_2_CASE_2="-s eq:0 -o file:4.expected -e empty b2e < 4.in" DISABLE_TESTSET_2_CASE_3="-s eq:0 -o file:5.expected -e empty b2e < 5.in" This specifies 2 test sets, the first set containing 2 test cases, and the second set containing 3 test cases. The first test set is for basic test cases, the second set is for extended tests (see the TESTSET_n_NAME definition). Test cases must be numbered in increasing order from 1, and may not miss numbers. (When the human2atf test script finds that a test case is not defined, it stops processing). The same numbering restrictions are in place for test sets. To provide the tests a useful environment in which to function, input and expected output files are specified. These are given in the TESTSET_n_FILES definition. Individual test cases are specified using the TESTSET_n_CASE_m style of definition. Occasionally, we will want to add a test and its supporting files, but leave it disabled (until appropriate functionality is written or debugged fully, for example) - for that case, the DISABLE_TESTSET_n_CASE_m definition is used. I opted to leave the atf-style of command invocation in there as the expected output file is specified in the test case itself but not as part of the command, so it seemed simpler to do it that way. Test execution (the directory and Makefile in tests/usr.bin had already been set up): [12:34:14] agc@netbsd-002 ~/local/human2atf-20160528 [3068] > ./human2atf -f Testspec2 Generating Makefile for b2e tests as tests-Makefile Generating test harness t_b2e.sh for b2e [12:34:19] agc@netbsd-002 ~/local/human2atf-20160528 [3069] > sudo cp t_b2e.sh =1 [12:34:30] agc@netbsd-002 ~/local/human2atf-20160528 [3070] > pushd +1 /usr/tests/usr.bin/b2e ~/local/human2atf-20160528 [12:34:33] agc@netbsd-002 ...tests/usr.bin/b2e [3071] > l total 20 drwxr-xr-x 2 root wheel 512 May 28 23:23 . drwxr-xr-x 35 root wheel 1024 May 28 19:17 .. -rw-r--r-- 1 root wheel 136 May 28 19:18 Atffile -rw-r--r-- 1 root wheel 112 May 28 19:18 Makefile -rw-r--r-- 1 root wheel 2184 May 28 23:23 atf-run.log -rwxr-xr-x 1 root wheel 3368 May 28 23:23 t_b2e -rw-r--r-- 1 root wheel 3751 May 29 12:34 t_b2e.sh [12:34:34] agc@netbsd-002 ...tests/usr.bin/b2e [3072] > sudo make # build b2e/t_b2e echo '#! /usr/bin/atf-sh' >t_b2e.tmp cat t_b2e.sh >>t_b2e.tmp chmod +x t_b2e.tmp mv t_b2e.tmp t_b2e [12:34:39] agc@netbsd-002 ...tests/usr.bin/b2e [3073] > sudo make test *** WARNING: make test is experimental *** *** Using this test does not preclude you from running the tests *** installed in /usr/tests. This test run may raise false *** positives and/or false negatives. Tests root: /usr/tests/usr.bin/b2e t_b2e (1/1): 2 test cases b2e_testset_1_basics: [0.031023s] Passed. b2e_testset_2_extended: [0.026240s] Passed. [0.060611s] Summary for 1 test programs: 2 passed test cases. 0 failed test cases. 0 expected failed test cases. 0 skipped test cases. *** The verbatim output of atf-run has been saved to /usr/tests/usr.bin/b2e/atf-run.log *** Once again, note that make test is unsupported. [12:34:43] agc@netbsd-002 ...tests/usr.bin/b2e [3074] > Alistair Crooks Sun May 29 12:41:19 PDT 2016 To generate a diff of this commit: cvs rdiff -u -r0 -r1.1 othersrc/external/bsd/human2atf/Makefile \ othersrc/external/bsd/human2atf/README \ othersrc/external/bsd/human2atf/Testspec.template \ othersrc/external/bsd/human2atf/human2atf.1 \ othersrc/external/bsd/human2atf/human2atf.sh Please note that diffs are not public domain; they are subject to the copyright notices on the relevant files.