On 08/03/2014 10:57 PM, Bernhard Voelker wrote: > On 08/02/2014 12:49 AM, Assaf Gordon wrote: >> Attached is a patch that adds a 'makefile' target to test a program. >> >> Examples: >> >> make check-prog T=sort >> make check-very-expensive-prog T=tail > > Thanks for picking this up again. > >> diff --git a/HACKING b/HACKING >> index 393a623..9acc3cf 100644 >> --- a/HACKING >> +++ b/HACKING >> @@ -434,6 +434,22 @@ and moving into existing blocks. This avoids making >> unnecessary >> work for translators. >> >> >> +Testing >> +======== >> +To run all existing tests for a specific program (e.g. 'seq'), run: >> + >> + make check-prog T=seq >> + >> +Similarly, to run expensive or very expensive tests, run: >> + >> + make check-expensive-prog T=tail >> + make check-very-expensive-prog T=sort >> + >> +T=[PROG] takes a basic regular expression, allowing: >> + >> + make check-prog T='sha.*' >> + >> + >> Add tests >> ========== >> Nearly every significant change must be accompanied by a test suite > > IMO the "Testing" paragraph should go after "Add tests", > as it's logically after adding a test for the actual commit. > ... or even after or go into > 'Run "make syntax-check", or even "make distcheck"'. > >> diff --git a/Makefile.am b/Makefile.am >> index fb4af27..e238904 100644 >> --- a/Makefile.am >> +++ b/Makefile.am >> @@ -85,6 +85,22 @@ check-expensive: >> check-very-expensive: >> $(MAKE) check-expensive RUN_VERY_EXPENSIVE_TESTS=yes >> >> +# Shortcut to run all tests of a specific program >> +# Typical usage: >> +# make check-prog PROG=sort >> +check-prog: >> + $(AM_V_GEN)( test -n "$$T" || \ >> + { echo missing T=[program] parameter >&2 ; exit 1 ; } ; \ >> + TESTS=$$(find ./tests/ \( -name '*.sh' -o -name '*.pl' \) -print | \ >> + grep -w -- "$$T" | paste -s -d' ' -) ; \ >> + test -z "$$TESTS" && \ >> + { echo no tests found for program "$$T" >&2 ; exit 1 ; } ; \ >> + $(MAKE) check TESTS="$$TESTS" SUBDIRS=. VERBOSE=yes ) > > First of all, is there a specific reason why you added VERBOSE=yes, here? > > > I somehow have the impression that the approach using 'find | grep' > has some drawbacks ... and indeed: > > * It doesn't find the one "*.xpl" test we have: > tests/rm/fail-eperm.xpl > See $(TEST_EXTENSIONS) in tests/local.mk. > > * For T=factor, it would additionally run the helper program > tests/factor/create-test.sh - which then fails. > > * For T=factor it wouldn't find the $(factor_tests) if they > don't already exist. > > Therefore, I'd rather go with something based on $(built_programs). > What about the following? > > diff --git a/Makefile.am b/Makefile.am > index fb4af27..2910083 100644 > --- a/Makefile.am > +++ b/Makefile.am > @@ -85,6 +85,25 @@ check-expensive: > check-very-expensive: > $(MAKE) check-expensive RUN_VERY_EXPENSIVE_TESTS=yes > > +# Shortcut to run all tests of a specific program > +# Typical usage: > +# make check-prog PROG=sort > +check-prog: > + $(AM_V_GEN)( test -n "$$T" || \ > + { echo "missing T=[program] parameter" >&2 ; exit 1 ; } ; \ > + case " $(built_programs) " in \ > + *" $$T "*) ;; \ > + *) { echo "program not built: '$$T'" >&2 ; exit 1 ; } ;; \ > + esac; \ > + tests=$$(grep -lw -- "$$T" $(TESTS) | paste -s -d' ' -) ; \ > + test -z "$$tests" && \ > + { echo no tests found for program "$$T" >&2 ; exit 1 ; } ; \ > + $(MAKE) check TESTS="$$tests" SUBDIRS=. ) > +check-expensive-prog: > + $(MAKE) check-prog RUN_EXPENSIVE_TESTS=yes > +check-very-expensive-prog: > + $(MAKE) check-expensive-prog RUN_VERY_EXPENSIVE_TESTS=yes > + > # Just prior to distribution, ... > # transform the automake-generated rule that runs 'rm -f rm'. > # On some systems, that command would fail with a diagnostic like > > Note that check-prog would now not only run primary tests of > the given T=program (we don't have that information anywhere), > but all tests where 'program' is used. I consider this is quite > useful, isn't it? Admittedly, it's problematic with 'T=test'.
FWIW here's the --test bit of my ./make script, which can test a program, directory or list of tests. The program tests are identified from the print_ver_ tagging: if [ "$1" = '--test' ]; then shift if test -d "$1"; then export tests_dir=$1 tests="$(shopt -s nullglob; echo tests/$tests_dir/*.{sh,pl})" echo $tests elif ! printf '%s\n' "$1" | grep -qF .; then # No .suffix then look for program name prog="$1" [ "$prog" = 'install' ] && prog='ginstall' tests=$(find tests/ -name '*.sh' | xargs grep -El "print_ver_.* $prog" | paste -s -d' ') tests="$tests $(find tests/ -name "$prog*.pl" | paste -s -d' ')" else tests="$*" fi # SUBDIRS avoids gnulib tests make check TESTS="$tests" SUBDIRS=. VERBOSE=yes RUN_VERY_EXPENSIVE_TESTS=yes RUN_EXPENSIVE_TESTS=yes exit fi