Hello,
both the old and the new Libtool testsuites contain some tests that
cause programs to fail to start, intentionally. As we add more test
coverage, the number of such tests invariably increases, too.
Now, a failed program execution on MSYS will cause a popup window
notifying about the situation, requiring you to acknowledge it, and
maybe even to avoid sending an online report to some software vendor
we are not affiliated with.
Efficient testing is hard if it requires human attention, so let's
make it easy to run only the unattended parts, or only the complement.
The patch below introduces a new Autotest keyword 'interactive' for
those tests that may require user intervention (on w32). It also adds
check-(non)interactive rules that run only the respective set of tests
in the new testsuite.
Some documentation about all used Autotest keywords is added, too, but
I'm not so sure whether that is a wise thing to do as it requires us to
be strict in test group naming (-k aka. --keyword also matches words in
the test group names).
I'm still pondering how to best treat the old testsuite here. IIRC then
at least the demo-relink and depdemo-relink tests failed (any others?),
but they require a handful of other demo-*/depdemo-* tests to run first.
What do you think of this approach?
Note that, since Autotest also uses the test group names for matching
keywords, that means you cannot use the documented keywords freely in
new test group names except where appropriate. I wonder if that should
be documented as well.
Thanks,
Ralf
Testsuite keyword 'interactive' and check-* rules.
* Makefile.am (testsuite_deps, testsuite_deps_uninstalled):
New variables.
(check-local, installcheck-local): Use them.
(check-interactive, check-noninteractive): New rules.
* tests/link-order2.at (Link order of deplibs),
tests/static.at (static linking flags for programs): Add keyword
`interactive'.
* doc/libtool.texi (Test descriptions): Document all keywords
used in the Libtool test suite.
* NEWS: Update.
diff --git a/Makefile.am b/Makefile.am
index 71fd91b..7f2f5c2 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -546,19 +546,38 @@ DISTCLEANFILES += tests/atconfig
CD_TESTDIR = abs_srcdir=`$(lt__cd) $(srcdir) && pwd`; cd tests
+testsuite_deps = tests/atconfig $(srcdir)/$(TESTSUITE)
+testsuite_deps_uninstalled = $(testsuite_deps) libltdl/libltdlc.la
$(bin_SCRIPTS)
+
# Hook the test suite into the check rule
-check-local: tests/atconfig $(srcdir)/$(TESTSUITE) libltdl/libltdlc.la
$(bin_SCRIPTS)
+check-local: $(testsuite_deps_uninstalled)
$(CD_TESTDIR); \
CONFIG_SHELL="$(SHELL)" $(SHELL) $$abs_srcdir/$(TESTSUITE) \
$(TESTS_ENVIRONMENT) $(BUILDCHECK_ENVIRONMENT) $(TESTSUITEFLAGS)
# Run the test suite on the *installed* tree.
-installcheck-local: tests/atconfig $(srcdir)/$(TESTSUITE)
+installcheck-local: $(testsuite_deps)
$(CD_TESTDIR); \
CONFIG_SHELL="$(SHELL)" $(SHELL) $$abs_srcdir/$(TESTSUITE) \
$(TESTS_ENVIRONMENT) $(INSTALLCHECK_ENVIRONMENT) $(TESTSUITEFLAGS) \
AUTOTEST_PATH="$(exec_prefix)/bin"
+# Run only noninteractive parts of the new testsuite.
+check-noninteractive: $(testsuite_deps_uninstalled)
+ $(CD_TESTDIR); \
+ CONFIG_SHELL="$(SHELL)" $(SHELL) $$abs_srcdir/$(TESTSUITE) \
+ $(TESTS_ENVIRONMENT) $(BUILDCHECK_ENVIRONMENT) \
+ -k !interactive INNER_TESTSUITEFLAGS=",!interactive" \
+ $(TESTSUITEFLAGS)
+
+# Run only interactive parts of the new testsuite.
+check-interactive: $(testsuite_deps_uninstalled)
+ $(CD_TESTDIR); \
+ CONFIG_SHELL="$(SHELL)" $(SHELL) $$abs_srcdir/$(TESTSUITE) \
+ $(TESTS_ENVIRONMENT) $(BUILDCHECK_ENVIRONMENT) \
+ -k interactive -k recursive INNER_TESTSUITEFLAGS=",interactive" \
+ $(TESTSUITEFLAGS)
+
# We need to remove any file droppings left behind by testsuite
clean-local: clean-local-legacy
-$(CD_TESTDIR); \
diff --git a/NEWS b/NEWS
index 899e085..ac15b55 100644
--- a/NEWS
+++ b/NEWS
@@ -15,6 +15,9 @@ New in 2.2.8 2009-??-??: git version 2.2.7a, Libtool team:
- New libtool command line flag --no-finish, to use as an optimization to
avoid multiple finish mode commands stemming from multiple installs into
the same library directory.
+ - New convenience make targets `check-noninteractive' to avoid long testsuite
+ runs on Windows with popup windows in the middle, and `check-interactive'
+ for the complement set of tests.
* Changes in supported systems or compilers:
diff --git a/doc/libtool.texi b/doc/libtool.texi
index 176f797..b4b9f37 100644
--- a/doc/libtool.texi
+++ b/doc/libtool.texi
@@ -5175,6 +5175,74 @@ directory, except that Fortran 90 is used in combination
with the
@end table
+The new, Autotest-based test suite uses keywords to classify certain
+test groups:
+
+...@table @samp
+...@item CXX
+...@itemx F77
+...@itemx FC
+...@itemx GCJ
+The test group exercises one of these @command{libtool} language tags.
+
+...@item autoconf
+...@itemx automake
+These keywords denote that the respective external program is needed
+by the test group. The tests are typically skipped if the program is
+not installed. The @samp{automake} keyword may also denote use of the
+...@command{aclocal} program.
+
+...@item interactive
+This test group may require user interaction on some systems. Typically,
+this means closing a popup window about a DLL load error on Windows.
+
+...@item libltdl
+Denote that the @file{libltdl} library is exercised by the test group.
+
+...@item libtool
+...@itemx libtoolize
+Denote that the @command{libtool} or @command{libtoolize} scripts are
+exercised by the test group, respectively.
+
+...@item recursive
+Denote that this test group may recursively re-invoke the test suite
+itself, with changed settings and maybe a changed @command{libtool}
+script. You may use the @env{INNER_TESTSUITEFLAGS} variable to pass
+additional settings to this recursive invocation. Typically, recursive
+invocations delimit the set of tests with another keyword, for example
+by passing @code{-k libtool} right before the expansion of the
+...@env{inner_testsuiteflags} variable (without an intervening space, so
+you get the chance for further delimitation).
+
+Test groups with the keyword @samp{recursive} should not be denoted with
+keywords, in order to avoid infinite recursion. As a consequence,
+recursive test groups themselves should never require user interaction,
+while the test groups they invoke may do so.
+...@end table
+
+For example, in order to avoid any interactive test groups, you could
+use this:
+
+...@smallexample
+make check-local \
+ TESTSUITEFLAGS='-k !interactive INNER_TESTSUITEFLAGS=",!interactive"'
+...@end smallexample
+
+...@noindent
+while to run only those test groups, you would use this:
+
+...@smallexample
+make check-local \
+ TESTSUITEFLAGS='-k interactive -k recursive
INNER_TESTSUITEFLAGS=,interactive'
+...@end smallexample
+
+...@noindent
+but the convenience targets @samp{check-interactive} and
+...@samp{check-noninteractive} avoid needing to remember these complex
+commands, in addition to also disabling interactive tests in the old
+test suite.
+
+
@node When tests fail
@subsection When tests fail
@cindex failed tests
diff --git a/tests/link-order2.at b/tests/link-order2.at
index 5f1c936..4bf9716 100644
--- a/tests/link-order2.at
+++ b/tests/link-order2.at
@@ -45,6 +45,7 @@
AT_SETUP([Link order of deplibs])
AT_KEYWORDS([libtool])
+AT_KEYWORDS([interactive])dnl running 'wrong' may cause a popup window.
eval "`$LIBTOOL --config | $EGREP '^(shlibpath_var|allow_undefined_flag)='`"
diff --git a/tests/static.at b/tests/static.at
index 1a09c56..b5fde17 100644
--- a/tests/static.at
+++ b/tests/static.at
@@ -67,6 +67,7 @@
AT_SETUP([static linking flags for programs])
AT_KEYWORDS([libtool])
+AT_KEYWORDS([interactive])dnl Some of the exec_fail test cause popups with
MinGW.
LDFLAGS="$LDFLAGS -no-undefined"
prefix=`pwd`/inst
