Re: Parallel tests execution [0/4]
* Ralf Wildenhues wrote on Wed, Oct 15, 2008 at 11:35:07PM CEST: > * Jim Meyering wrote on Wed, Oct 15, 2008 at 09:13:54PM CEST: > > Ralf Wildenhues <[EMAIL PROTECTED]> wrote: > > > 12) allow for additional output on stdout/stderr? > > > example: test was skipped because of $reason. > > > > This would be nice. > Hmm. Maybe let another fd dup stderr? Would be trivial, by adding '3>&2' first thing to am__check_post. Several things that I am wary about: - the file descriptor choice is a bit arbitrary, and might cause work for users, - parallel runs will still intermingle output. (Of course, for any non-ancient shell, it could be had by putting '3>&2' in TESTS_ENVIRONMENT.) > > It'd would be useful also to mark as "surprising" or > > "highly undesirable" the results of certain tests. > > Better than a simple "yes" or "no". I don't really know what to do about this. Why not let them FAIL? More generally, if there are to be more result values, then we should define a set of generally usable semantics for them. But as a first approximation, I guess you could output some details on the extra fd, and let "surprising" results PASS or SKIP, and "highly undesirable" ones FAIL. WDYT? Cheers, Ralf
Re: Documentation for the parallel-tests driver. [4/4]
* Akim Demaille wrote on Thu, Oct 16, 2008 at 05:16:20PM CEST:
> >>> "RW" == Ralf Wildenhues <[EMAIL PROTECTED]> writes:
>
> > [EMAIL PROTECTED] Simple tests using @samp{parallel-tests}
> > [EMAIL PROTECTED] @option{parallel-tests}, Using
> > +The option @option{parallel-tests} (@pxref{Options}) enables a test
> > +suite driver that is mostly compatible to the simple test driver
> > +described above, but provides a few more features and slightly different
> > +semantics. It features concurrent execution of tests with @code{make -j},
> > +allows to specify inter-test dependencies, lazy reruns of tests that
> > +have not completed in a prior run,
>
> That's one use, but more importantly, this is perfect for unit-tests.
> Unit-tests are often self-contained, in which case if none of their
> dependencies changed, there is no point in running the test again, the
> results will be the same.
Yes. The first paragraph is intended as a feature overview only. Unit
tests are mentioned later:
[EMAIL PROTECTED] Unit tests
+The combination of lazy test execution and correct dependencies between
+tests and their sources may be exploited for efficient unit testing
+during development. To further speed up the edit-compile-test cycle, it
+may even be useful to specify compiled programs in @code{EXTRA_PROGRAMS}
+instead of with @code{check_PROGRAMS}, as the former allows intertwined
+compilation and test execution (but note that @code{EXTRA_PROGRAMS} are
+not cleaned automatically, @pxref{Uniform}).
I think this is special enough that it doesn't need to be mentioned
early.
> > [EMAIL PROTECTED] VERBOSE
> > +As with the simple driver above, by default one status line is printed
> > +per completed test, and a short summary after the suite has completed.
> > +If the variable @samp{VERBOSE} is set, the @file{test-suite.log} file is
> > +appended after the summary.
>
> IMHO, you should emphasize that tests should be verbose by default.
> VERBOSE is no longer an envvar that test should read to decide whether
> displaying something or not: they should be verbose by default, it
> goes into the log file. Now VERBOSE is meant for "make check" itself.
Good point; thanks.
> I think that you should detail how to write a .test->.log rule,
> showing the default value for instance (I use GNU Make syntax).
>
> # From a test file to a log file.
> # Do not use a regular `.test.log:' rule here, since in that case the
> # following rule (without incoming extension) will mask this one.
> %.log: %.test $(check_programs)
> @$(am__check_pre) $${dir}$< $(am__check_post)
>
> because that's where the user can explain how the test is run.
Erm, the framework in Automake is _not_ supposed to allow the user to
write such rules herself. That would likely be problematic, the least
of which being that it's difficult to make Automake version-agnostic.
Consequently, there is little point in showing users how to write such
a rule.
> Something I dislike very much in the current framework for tests in
> Automake is that each test must be an executable, so it is very
> inconvenient to test a given program foo with many different inputs.
> In that case I use
>
>TESTS = 1.foo 2.foo 3.foo ...
>
> and
>
> %.log: %.test $(check_programs)
> @$(am__check_pre) $(top_builddir)/src/foo $${dir}$< $(am__check_post)
>
> i.e., the testee is foo, and TESTS are plain files, not a bazillion of
> 1.sh that simple "exec $(top_builddir)/src/foo 1.foo".
Try
TESTS_ENVIRONMENT = $(top_builddir)/src/foo
and adding $(top_builddir)/src/foo to check_SCRIPTS.
If you have only some files that are to be run with foo, and some not,
consider looking at coreutils generalization of this:
.
Cheers,
Ralf
Re: Parallel tests execution [0/4]
Hi Akim, thanks for the feedback! * Akim Demaille wrote on Thu, Oct 16, 2008 at 04:05:47PM CEST: > >>> "RW" == Ralf Wildenhues <[EMAIL PROTECTED]> writes: > > > These four patches implement parallel execution of TESTS in Automake, > > adapted from the check.mk file Akim Demaille posted earlier. > > For the records, I attached the version I'm currently using. Thanks. FWIW, it still has some of the portability issues that I mentioned. > > 6) lazy test completion (do not rerun already-run tests), > > This one must be optional, but it provides huge savings when it > applies. Agreed on both accounts. It is optional in the version I have. > > - (5), (6), (8) are provided already by the check.mk code, except that > > (8) didn't work. > > Actually I never meant to have hard error stop the whole test suite. > The point of hard-errors as they were defined in check.mk was to make > them *non* ignorable. For instance our test suite raises a hard-error > if the program make a segmentation fault, which we never want to > tolerate. I don't understand. What is the difference to a normail FAIL then, i.e., to the process exiting with 1? > > - output `PASS: foo.test' not `PASS: foo.log' > > This was actually a feature :) We use an Emacs mode that opens the > (log) file when we click on it. But it's not the log file that fails. I found this very non-intuitive. I might be talked into a compromise, though; for example like this: FAIL: sub/foo.test (see sub/foo.log) WDYT? > > - is everybody ok with the following authorship for patch 1/4? > > 2008-10-XX Akim Demaille <[EMAIL PROTECTED]> > > Jim Meyering <[EMAIL PROTECTED]> > > Benoit Sigoure <[EMAIL PROTECTED]> > > Ralf Wildenhues <[EMAIL PROTECTED]> > > > (as git allows only one author, I will put the first name in --author) > > Thanks :) I think that some of the parts about tput were from Bob > Proulx, but I'm not sure. I ripped out all the tput parts, because in my tests they were far less portable than escape sequences. If there are other things from Bob then I'll happily add him. > > - how should I best acknowledge The Vaucanson Group? Something like > > this in lib/am/check.am ok? > > > ## This code is adapted from check.mk which came from: > > ## > > ## Vaucanson, a generic library for finite state machines. > > ## Copyright (C) 2006, 2007 The Vaucanson Group. > > Actually it would be more fair to thanks EPITA and Gostai, both worked > on it, and Vaucanson was just the initial impetus to develop this. OK, I will use this: ## This code is adapted from check.mk which was originally ## written by The Vaucanson Group, further developer at ## EPITA and Gostai, then made its way from GNU coreutils ## to end up, largely rewritten, in Automake. > > The only reason I haven't put this in yet is that it would require a > > copyright disclaimer from Vaucanson. What do you think? > > A mere thank, or whatever you feel is most appropriate will be > perfect. OK, thanks. Cheers, Ralf
Re: Documentation for the parallel-tests driver. [4/4]
>>> "RW" == Ralf Wildenhues <[EMAIL PROTECTED]> writes:
> [EMAIL PROTECTED] Simple tests using @samp{parallel-tests}
> [EMAIL PROTECTED] @option{parallel-tests}, Using
> +The option @option{parallel-tests} (@pxref{Options}) enables a test
> +suite driver that is mostly compatible to the simple test driver
> +described above, but provides a few more features and slightly different
> +semantics. It features concurrent execution of tests with @code{make -j},
> +allows to specify inter-test dependencies, lazy reruns of tests that
> +have not completed in a prior run,
That's one use, but more importantly, this is perfect for unit-tests.
Unit-tests are often self-contained, in which case if none of their
dependencies changed, there is no point in running the test again, the
results will be the same.
> +summary and verbose output in @samp{RST} (reStructuredText) and
> [EMAIL PROTECTED] format, and hard errors for early abort of test runs.
> +Similar to the simple test driver, @code{TESTS_ENVIRONMENT},
> [EMAIL PROTECTED], @code{XFAIL_TESTS}, and the @code{check_*}
> +variables are honored, and the environment variable @env{srcdir}
> +is set during test execution.
> [EMAIL PROTECTED] VERBOSE
> +As with the simple driver above, by default one status line is printed
> +per completed test, and a short summary after the suite has completed.
> +If the variable @samp{VERBOSE} is set, the @file{test-suite.log} file is
> +appended after the summary.
IMHO, you should emphasize that tests should be verbose by default.
VERBOSE is no longer an envvar that test should read to decide whether
displaying something or not: they should be verbose by default, it
goes into the log file. Now VERBOSE is meant for "make check" itself.
I think that you should detail how to write a .test->.log rule,
showing the default value for instance (I use GNU Make syntax).
# From a test file to a log file.
# Do not use a regular `.test.log:' rule here, since in that case the
# following rule (without incoming extension) will mask this one.
%.log: %.test $(check_programs)
@$(am__check_pre) $${dir}$< $(am__check_post)
because that's where the user can explain how the test is run.
Something I dislike very much in the current framework for tests in
Automake is that each test must be an executable, so it is very
inconvenient to test a given program foo with many different inputs.
In that case I use
TESTS = 1.foo 2.foo 3.foo ...
and
%.log: %.test $(check_programs)
@$(am__check_pre) $(top_builddir)/src/foo $${dir}$< $(am__check_post)
i.e., the testee is foo, and TESTS are plain files, not a bazillion of
1.sh that simple "exec $(top_builddir)/src/foo 1.foo".
Re: New tests for `parallel-tests'. [3/4]
>>> "RW" == Ralf Wildenhues <[EMAIL PROTECTED]> writes: + # Try the variants that are tried in check.am. + while :; do + for r2h in $RST2HTML rst2html rst2html.py; do + echo "$me: running $r2h --version" + $r2h --version && break 2 + done + exit 77 + done + while-for-break-2, nice idiom!
Re: Parallel test execution: new option `parallel-tests': [1/4]
>>> "RW" == Ralf Wildenhues <[EMAIL PROTECTED]> writes: > +case fail=$$fail:xpass=$$xpass:xfail=$$xfail in \ > + fail=0:xpass=0:xfail=0) \ > +msg="$$All$$all $$tests passed. "; \ > +exit=true;; \ > + fail=0:xpass=0:xfail=*) \ > +msg="$$All$$all $$tests behaved as expected"; \ > +if test "$$xfail" -eq 1; then xfailures=failure;\ > +else xfailures=failures; fi;\ > +msg="$$msg ($$xfail expected $$xfailures). "; \ > +exit=true;; \ > + fail=*:xpass=0:xfail=*) \ > +msg="$$fail of $$all $$tests failed. ";\ > +exit=false;;\ > + fail=*:xpass=*:xfail=*) \ > +msg="$$failures of $$all $$tests did not behave as expected"; \ > +if test "$$xpass" -eq 1; then xpasses=pass; \ > +else xpasses=passes; fi;\ > +msg="$$msg ($$xpass unexpected $$xpasses). "; \ > +exit=false;;\ > + *)\ > +echo >&2 "incorrect case"; exit 4;; \ > +esac; \ > +if test "$$skip" -ne 0; then\ > + if test "$$skip" -eq 1; then \ > +msg="$$msg($$skip test was not run). ";\ > + else \ > +msg="$$msg($$skip tests were not run). "; \ > + fi; \ > +fi; \ The logic to create this kind of message is really complex, and in retrospect, I don't think that the message delivered is much easier to read than something more regular. So currently check.mk does as follows: case fail=$$fail:xpass=$$xpass in \ fail=0:xpass=0) \ msg="The test suite passed. "; \ exit=true;; \ *)\ msg="The test suite failed. "; \ exit=false;;\ esac; \ msg="$$msg - $$pass/$$all tests passed. "; \ test $$fail -eq 0 ||\ msg="$$msg - $$fail failures. "; \ test $$xpass -eq 0 || \ msg="$$msg - $$xpass unexpected pass. "; \ test $$xfail -eq 0 || \ msg="$$msg - $$xfail expected failures. "; \ test $$tfail -eq 0 || \ msg="$$msg - $$tfail expected temporary failures. "; \ test $$skip -eq 0 ||\ msg="$$msg - $$skip skipped tests. "; \ It's simpler to create the message, and I think, the message is also simpler. We introduced TFAIL, temporary failures: it's sort of short-term XFAIL. When a test starts to fail, it might be because someone uncovered a bug elsewhere, and that person might not be competent to address the real bug. Yet, leaving the test as is, failing, is a problem for everybody else who might waste a lot of time to discover that they have not introduced the problem, it was already there. So we make it TFAIL, which means "should be processed soon".
Re: Parallel tests execution [0/4]
While talking about test suites, there is a simple change that would make several Makefile.ams much shorter. Often tests are simple one-compilation-unit programs, in which case the convention that foo_SOURCES defaults to foo.c saves lot of dummy typing (especially if there are many foos). But this does not apply to C++ :( I wish I could define AM_DEFAULT_SOURCE_EXT = .cc or whatever, so that I wouldn't have to define all the foo_SOURCES.
Re: Parallel tests execution [0/4]
>>> "RW" == Ralf Wildenhues <[EMAIL PROTECTED]> writes: > Hello everyone, Hi Ralf, I'm sorry for not being more responsive :( And many, many thanks for carrying this work to completion. > These four patches implement parallel execution of TESTS in Automake, > adapted from the check.mk file Akim Demaille posted earlier. For the records, I attached the version I'm currently using. check.mk Description: Binary data > 3) should work with different test naming styles: > - *.test (with log files named s/\.test$/.log/) > - no particular suffix (log files will be s/$/.log/) > - should work with executables (e.g., TESTS = $(check_PROGRAMS)) > - arbitrary other suffixes Yep. > 6) lazy test completion (do not rerun already-run tests), This one must be optional, but it provides huge savings when it applies. > - (5), (6), (8) are provided already by the check.mk code, except that > (8) didn't work. Actually I never meant to have hard error stop the whole test suite. The point of hard-errors as they were defined in check.mk was to make them *non* ignorable. For instance our test suite raises a hard-error if the program make a segmentation fault, which we never want to tolerate. > - output `PASS: foo.test' not `PASS: foo.log' This was actually a feature :) We use an Emacs mode that opens the (log) file when we click on it. > - shouldn't test-suite.log also be created if all tests passed? Why not, in which case it would be mostly empty and containing some like "all the tests passed as expected"? > - should skipped tests show up in test-suite.log? Can be useful, indeed. > - is everybody ok with the following authorship for patch 1/4? > 2008-10-XX Akim Demaille <[EMAIL PROTECTED]> > Jim Meyering <[EMAIL PROTECTED]> > Benoit Sigoure <[EMAIL PROTECTED]> > Ralf Wildenhues <[EMAIL PROTECTED]> > (as git allows only one author, I will put the first name in --author) Thanks :) I think that some of the parts about tput were from Bob Proulx, but I'm not sure. > - how should I best acknowledge The Vaucanson Group? Something like > this in lib/am/check.am ok? > ## This code is adapted from check.mk which came from: > ## > ## Vaucanson, a generic library for finite state machines. > ## Copyright (C) 2006, 2007 The Vaucanson Group. Actually it would be more fair to thanks EPITA and Gostai, both worked on it, and Vaucanson was just the initial impetus to develop this. > The only reason I haven't put this in yet is that it would require a > copyright disclaimer from Vaucanson. What do you think? A mere thank, or whatever you feel is most appropriate will be perfect.
