> On Jan 13, 2017, at 4:44 PM, Patrick Sanan <[email protected]> wrote: > > This is really great! If they don't already do so, maybe it would be > simple now to modify the new example-walking scripts to ensure that > the test suite covers all the examples/tutorials (maybe even nightly), > noting any qualifying files that don't have a valid /*TEST .. TEST*/ > block.
This will be added, thanks for the suggestion. > > On Fri, Jan 13, 2017 at 11:16 PM, Barry Smith <[email protected]> wrote: >> >> PETSc developers, >> >> Scott Kruger has been implementing a new "test harness" for PETSc. >> Rather than having tests implemented as bash lines in the makefiles, the >> tests will be defined within the source code for each example with a simple >> format and the actual scripts that run the tests will be automatically >> generated. This offers three advantages over the current system >> >> 1) test definitions can be given in a very concise format that is >> independent of how the test is run >> 2) multiple approaches to actually running the tests can be implemented >> without requiring any changes to the test definitions for each example. >> 3) the tests can be compiled and run in parallel (eventually we hope to see >> much faster times for testing). >> >> Here is an example of the format that defines tests >> >> test: >> >> test: >> suffix: yaml >> requires: yaml !complex >> args: -options_file_yaml bag.yml -options_view >> filter: grep -v saws_port_auto_select >> localrunfiles: bag.yml >> >> The first is a test that takes no command line arguments and does no >> filtering of output. The second takes command line arguments, does some >> filtering of the output and this example also requires a data file >> associated with that example. The expected output of tests continues to >> reside in the output directory with the previously naming convention of >> XXX_1.out XXX_2.out where XXX is the name of the example or XXX_suffix.out >> where suffix is defined in the test definition. >> >> So far in the master branch we have only converted over two directories of >> examples sys/examples/tutorials and sys/examples/tests so you can look in >> those directories for other examples. Over the next few weeks we hope to >> convert over all the rest of the directories. >> >> Special thanks to Scott, Satish, and Jed for all the work they have done to >> set up the new test harness. >> >> Barry >> >> Technical details: the new python script ./config/gmakegentest.py which is >> normally automatically run at configure walks through the examples >> directories looking for the formatted test definitions, for each test it >> generates in the directory $PETSC_ARCH/test/src/XXX/examples/tests or >> tutorials/ a shell script that can run that test. In addition >> config/gmakegentest.py adds dependencies to the $PETSC_DIR/gmakefile for >> all the tests that it has found. The rule make alltests then executes all >> those dependencies. >> >> We will be refining both the testing harness and the test definitions as >> we convert the rest of the examples over to the new approach. >> >> >> >> >>
