Hello everyone, These four patches implement parallel execution of TESTS in Automake, adapted from the check.mk file Akim Demaille posted earlier. I would be delighted about review, comments, bug reports, any feedback. There are a couple of questions at the end.
Design considerations: 1) should be similar to current TESTS interface, and not enabled without developer consent (because of inevitable semantic differences) 2) should allow running of a subset of tests easily, preferably as make check TESTS="..." 3) should work with different test naming styles: - *.test (with log files named s/\.test$/.log/) - no particular suffix (log files will be s/$/.log/) - should work with executables (e.g., TESTS = $(check_PROGRAMS)) - arbitrary other suffixes 4) should work with @substituted@ values in TESTS, (here a limitation upon nonempty suffixes is ok) 5) parallel test execution, 6) lazy test completion (do not rerun already-run tests), 7) portable make (of course!), portable shell, etc., 8) should allow for hard errors that stop test suite execution, 9) should allow for dependency relations (ordering) between tests, 10) test names may have directory components, ------ 11) allow for extensibility like multiple test suites, each with their own set of logs. 12) allow for additional output on stdout/stderr? example: test was skipped because of $reason. 13) allow for post-test action in the rule (like: clean foo.dir in Automake's test suite)? 14) let subdir tests depend on dirstamp instead of each one calling MKDIR_P? The last items have not been addressed. Implementation notes: - (1) is addressed by using a new Automake option 'parallel-tests', and checking that both old and new driver pass all common tests (each one is run twice). - (2) precludes that we reuse the _SOURCES interface, unfortunately. - (2) and (3) preclude that the user can specify something like TEST_LOGS = $(TESTS:.test=.log) Instead, we let her specify TEST_SUFFIXES = .test We could also have tried to gather all kinds of suffixes from $(TESTS) but that may lead to more ambiguities in suffix rules that desirable - if there is a nonempty test suffix, we can use suffix rules and avoid a per-target rule for each test. That means @substituted@ values only work if the test suffix is nonempty (or we would have to use the hack from check.mk that restarts make with overwritten variables). - (5), (6), (8) are provided already by the check.mk code, except that (8) didn't work. Changes over the check.mk code: - Replace $(cmd) by `cmd` (for Solaris) - stick to portable awk (for Solaris) - fix awk script to not drop output - no white space at beginning or end of summary output - sed: use literal TAB instead of \t in regex - no leading underscore in make macro names - use $(AM_MAKEFLAGS) whereever $(MAKE) is used - also clean *.log-t intermediate files - clean rules merged into automake proper (%clean_files) - transform pattern rules into suffix or per-target rules rules, for non-GNU make (and avoid $< in the non-suffix rules) - avoid Solaris make VPATH rewriting bugs - output `PASS: foo.test' not `PASS: foo.log' but respect subdirs - some optimization (avoid lots of `sed 1q') - make hard errors work; use exit status 99 as indicator (values >128 are not portable, and should not be close to 128); variable used it DISABLE_HARD_ERRORS, not ENABLE_HARD_ERRORS, so that the default can be unset (which is better for portably transporting a user override across recursive makes). - Fix summary not to count SKIPs as PASSes - Fix singular/plural in summary Questions, and issues I'm not sure about yet: - improvements on the naming scheme would be good; some of the new variables seem quite ad-hoc - shouldn't test-suite.log also be created if all tests passed? - should skipped tests show up in test-suite.log? - is everybody ok with the following authorship for patch 1/4? 2008-10-XX Akim Demaille <[EMAIL PROTECTED]> Jim Meyering <[EMAIL PROTECTED]> Benoit Sigoure <[EMAIL PROTECTED]> Ralf Wildenhues <[EMAIL PROTECTED]> (as git allows only one author, I will put the first name in --author) - how should I best acknowledge The Vaucanson Group? Something like this in lib/am/check.am ok? ## This code is adapted from check.mk which came from: ## ## Vaucanson, a generic library for finite state machines. ## Copyright (C) 2006, 2007 The Vaucanson Group. The only reason I haven't put this in yet is that it would require a copyright disclaimer from Vaucanson. What do you think? Cheers, and thanks for reading this far, Ralf 1/4 Parallel test execution: new option `parallel-tests'. 2/4 parallel-tests: Ensure backward-compatible semantics. 3/4 New tests for `parallel-tests'. 4/4 Documentation for the parallel-tests driver.