Code coverage can help find under-tested problem spots. Devel::Cover is a nice solution for Perl code. In fact, it shows that the Autoconf test suite can be improved quite a bit (but see the respective message for Automake, it exercises some of the shared files better, <http://thread.gmane.org/gmane.comp.sysutils.automake.general/11088>).
----------------------------------- ------ ------ ------ ------------ ------ ------ File stmt bran cond sub pod time total ----------------------------------- ------ ------ ------ ------------ ------ ------ ...f/autoconf/lib/Autom4te/C4che.pm 100.0 75.0 66.7 100.0 100.0 0.0 95.2 ...conf/lib/Autom4te/ChannelDefs.pm 57.8 54.5 n/a 73.3 100.0 0.3 59.4 ...utoconf/lib/Autom4te/Channels.pm 66.3 53.2 57.1 71.1 100.0 0.1 63.6 ...onf/lib/Autom4te/Configure_ac.pm 96.4 87.5 100.0 100.0 0.0 0.0 95.6 ...toconf/lib/Autom4te/FileUtils.pm 84.6 59.4 90.9 84.0 100.0 54.3 77.8 ...autoconf/lib/Autom4te/General.pm 82.5 45.8 50.0 90.0 100.0 1.2 77.3 ...autoconf/lib/Autom4te/Request.pm 78.3 0.0 n/a 83.3 0.0 0.0 74.2 .../autoconf/lib/Autom4te/Struct.pm 74.4 43.9 33.3 69.6 12.5 0.1 63.1 ...f/autoconf/lib/Autom4te/XFile.pm 82.6 45.8 14.3 100.0 55.6 24.2 70.4 ...vs/autoconf/build/bin/autoheader 92.9 78.9 53.8 100.0 n/a 0.1 87.4 .../cvs/autoconf/build/bin/autom4te 96.2 74.2 69.0 95.7 n/a 10.9 88.3 ...vs/autoconf/build/bin/autoreconf 78.8 55.5 47.5 100.0 n/a 5.6 69.3 .../cvs/autoconf/build/bin/autoscan 78.0 66.2 54.9 95.5 n/a 0.0 73.6 ...vs/autoconf/build/bin/autoupdate 95.7 65.8 50.0 100.0 n/a 0.2 87.8 ...d/cvs/autoconf/build/bin/ifnames 100.0 83.3 80.0 100.0 n/a 0.0 96.1 [... automake-related files elided ...] ----------------------------------- ------ ------ ------ ------ ------ ------ ------ Obvious followup question: how can we examine Autoconf macro coverage? Bash/Zsh have some facilities that allow tracing shell scripts, but that's more useful for mostly static scripts like 'libtool', and even there, it is not simple to use effectively. Add the fact that configure and libtool have lots of system-dependent parts, so only a merge of the coverage over various systems will provide complete data. Can M4 tracing be instrumented to show coverage of expanded macros? (I guess this ought to be fairly easy). Can M4/M4sugar be instrumented to map shell coverage information back from some random configure script to our input macro files? That would be cool, but I'd have no idea how to realize it. You might need this additional diff for Devel::Cover to work right: | --- a/lib/Autom4te/General.pm | +++ b/lib/Autom4te/General.pm | @@ -197,7 +197,7 @@ sub END | | # This is required if the code might send any output to stdout | # E.g., even --version or --help. So it's best to do it unconditionally. | - if (! close STDOUT) | + if (! STDOUT->flush) | { | print STDERR "$me: closing standard output: $!\n"; | $? = 1; WDYT? Please note that IMHO you shouldn't delay 2.65 for any of this. I was going to hook cleanup to maintainer-clean, but Devel::Cover invalidates traces anyway when the files are newer, so that wouldn't help. Thanks, Ralf Perl coverage convenience targets. * Makefile.am (PERL_COVERAGE_DB, PERL_COVERAGE_FLAGS, PERL_COVER): New variables. (check-coverage, check-coverage-run, check-coverage-report) (clean-coverage): New phony targets. (clean-local): Depend on clean-coverage. diff --git a/Makefile.am b/Makefile.am index 3c90657..165eaf2 100644 --- a/Makefile.am +++ b/Makefile.am @@ -74,3 +74,32 @@ dist-hook: # TODO remove this once automake vs. AC_CONFIG_LINKS issue is fixed. distclean-local: if test x"$(VPATH)" != x ; then rm -f GNUmakefile ; fi + +# Perl coverage statistics. +PERL_COVERAGE_DB = $(abs_top_builddir)/cover_db +PERL_COVERAGE_FLAGS = -MDevel::Cover=-db,$(PERL_COVERAGE_DB),-silent,on,-summary,off +PERL_COVER = cover + +check-coverage-run: all + $(mkinstalldirs) "$(PERL_COVERAGE_DB)" + PERL5OPT="$(PERL_COVERAGE_FLAGS)"; export PERL5OPT; \ + $(MAKE) $(AM_MAKEFLAGS) check + +check-coverage-report: + @if test ! -d "$(PERL_COVERAGE_DB)"; then \ + echo "No coverage database found in \`$(PERL_COVERAGE_DB)'." >&2; \ + echo "Please run \`make check-coverage' first" >&2; \ + exit 1; \ + fi + $(PERL_COVER) $(PERL_COVER_FLAGS) "$(PERL_COVERAGE_DB)" + +# We don't use direct dependencies here because we'd like to be able +# to invoke the report even after interrupted check-coverage. +check-coverage: check-coverage-run + $(MAKE) $(AM_MAKEFLAGS) check-coverage-report + +clean-local: clean-coverage +clean-coverage: + rm -rf "$(PERL_COVERAGE_DB)" + +.PHONY: check-coverage check-coverage-run check-coverage-report clean-coverage