On Tue, 01 Aug 2017 11:18:30 +0300, Niko Tyni wrote: > Since 0.37, we're running 'prove --recurse' by default in the smoke > test. This has resulted in ~50 regressions according to ci.d.n: > there are packages ship .t files under t/ subdirectories (or starting > with a dot) that are not run during builds and are either broken > or not intended to be run at all. Fixing these by adding t/*.t in > debian/tests/pkg-perl/smoke-tests gets boring quite quickly, though it's > certainly doable.
Hm, interesting finding ..
What triggered the change was that adding all the tests in
subdirectories also became boring. I had a look: Currently we have 24
debian/tests/pkg-perl/smoke-tests files with something like:
% cat libhtml-formfu-perl/debian/tests/pkg-perl/smoke-tests
t/*.t
t/*/*.t
t/*/*/*.t
t/*/*/*/*.t
> All these packages seem to be relying on the ExtUtils::MakeMaker default
> of TEST_FILES => t/*.t. I'm wondering if we should try to parse that
> out of Makefile.PL during autopkgtest instead of always recursing,
> though that does seem somewhat error prone.
libhtml-formfu-perl's Makefile.PL has
"test" => {
"TESTS" => "t/*.t t/bugs/*.t t/constraints/*.t t/deflators/*.t
t/elements/*.t t/examples/*.t t/filters/*.t t/force_errors/*.t t/form/*.t
t/i18n/*.t t/inflators/*.t …
}
> FWIW, Module::Build seems to have a related 'recursive_test_files'
> option that is off by default (see Module::Build::API(3pm)).
>
> It might help a bit to collect some statistics about the number of
> packages with *.t files in subdirectories, and whether those have
> TEST_FILES specified in Makefile.PL. Currently I have no idea if the
> failing packages are just a small minority of the affected ones, or if
> almost all of them are failing now.
Not sure if TEST_FILES is actually that helpful:
% grep TEST_FILES */Makefile.PL
ack-grep/Makefile.PL:TEST_FILES=t/*.t t/lib/*.t
ack-grep/Makefile.PL: $(FULLPERLRUN) t/runtests.pl 0 $(TEST_VERBOSE)
"$(INST_LIB)" "$(INST_ARCHLIB)" $(TEST_FILES)
ack-grep/Makefile.PL: $(FULLPERLRUN) t/runtests.pl 1 $(TEST_VERBOSE)
"$(INST_LIB)" "$(INST_ARCHLIB)" $(TEST_FILES)
libanyevent-gearman-perl/Makefile.PL: .q{ $(TEST_FILES)} .
"\n";
libapache2-mod-perl2/Makefile.PL:TEST_FILES =
libapache2-mod-perl2/Makefile.PL: t/TEST -bugreport
-verbose=$(TEST_VERBOSE) $(TEST_FILES)
libautovivification-perl/Makefile.PL: PERL_DL_NONLAZY=1
PERLDB_OPTS="NonStop=1" $(FULLPERLRUN) -MTAP::Harness -e
'TAP::Harness->new({verbosity => q{$(VERBOSE)}, lib => [ q{$(INST_LIB)},
q{$(INST_ARCHLIB)} ], switches => [ q{-d} ]})->runtests(@ARGV)' $(TEST_FILES)
libcpanel-json-xs-perl/Makefile.PL: s/TEST_FILES = /TEST_FILES =
-Mcommon::sense /;
libdata-messagepack-perl/Makefile.PL: .q{ $(TEST_FILES)};
libdata-peek-perl/Makefile.PL: $(TEST_FILES) 3>valgrind.log
libdata-peek-perl/Makefile.PL: ' -@env TEST_FILES="xt/*.t" make -e
test_dynamic',
libdata-peek-perl/Makefile.PL: q{ sandbox/leaktest $(FULLPERLRUN)
"test_harness($(TEST_VERBOSE), '$(INST_LIB)', '$(INST_ARCHLIB)')"
$(TEST_FILES)},
libdbd-csv-perl/Makefile.PL: ' -@env DBI_SQL_NANO=1 make -e
test_dynamic TEST_FILES=t/[1-9]*.t',
libdbd-csv-perl/Makefile.PL: ' -@env TEST_FILES="xt/*.t" make -e
test_dynamic',
libgeo-distance-xs-perl/Makefile.PL:\t\$(MAKE) -e \$(TEST_TYPE)
TEST_FILES="xt/*.t"
libmemcached-libmemcached-perl/Makefile.PL:
HARNESS_PERL_SWITCHES='-MDevel::Cover=-coverage,branch,-coverage,condition,-coverage,pod,-coverage,statement,-coverage,subroutine'
PERL_DL_NONLAZY=1 $(FULLPERLRUN) "-MExtUtils::Command::MM" "-e"
"test_harness($(TEST_VERBOSE), '$(INST_LIB)','$(INST_ARCHLIB)')" $(TEST_FILES)
libpoe-perl/Makefile.PL: TEST_FILES
libpoe-perl/Makefile.PL: test => { TESTS => TEST_FILES },
libppix-regexp-perl/Makefile.PL: '$(FULLPERLRUN)', '$(AUTHORTEST_FILES)'
);
libppix-regexp-perl/Makefile.PL:AUTHORTEST_FILES = t/*.t xt/author/*.t
libscope-upper-perl/Makefile.PL: PERL_DL_NONLAZY=1
PERLDB_OPTS="NonStop=1" $(FULLPERLRUN) -MTAP::Harness -e
'TAP::Harness->new({verbosity => q{$(TEST_VERBOSE)}, lib => [ q{$(INST_LIB)},
q{$(INST_ARCHLIB)} ], switches => [ q{-d} ]})->runtests(@ARGV)' $(TEST_FILES)
libsoap-wsdl-perl/Makefile.PL: test => { RECURSIVE_TEST_FILES => 1, },
libspreadsheet-read-perl/Makefile.PL: $(TEST_FILES) 3>valgrind.log
libspreadsheet-read-perl/Makefile.PL: q{ sandbox/leaktest $(FULLPERLRUN)
"test_harness($(TEST_VERBOSE), '$(INST_LIB)', '$(INST_ARCHLIB)')"
$(TEST_FILES)},
libsystem-info-perl/Makefile.PL: $(TEST_FILES) 3>valgrind.log
libsystem-info-perl/Makefile.PL: ' -@env TEST_FILES="xt/*.t" make
-e test_dynamic',
libsystem-info-perl/Makefile.PL: q{ sandbox/leaktest $(FULLPERLRUN)
"test_harness($(TEST_VERBOSE), '$(INST_LIB)', '$(INST_ARCHLIB)')"
$(TEST_FILES)},
libtext-csv-xs-perl/Makefile.PL: $(TEST_FILES) 3>valgrind.log
libtext-csv-xs-perl/Makefile.PL: ' -@env TEST_FILES="xt/*.t" make
-e test_dynamic',
libtext-csv-xs-perl/Makefile.PL: q{ sandbox/leaktest
$(FULLPERLRUN) "test_harness($(TEST_VERBOSE), '$(INST_LIB)',
'$(INST_ARCHLIB)')" $(TEST_FILES)},
libtext-csv-xs-perl/Makefile.PL: q{ sandbox/leaktrace
$(FULLPERLRUN) "test_harness($(TEST_VERBOSE), '$(INST_LIB)',
'$(INST_ARCHLIB)')" $(TEST_FILES)},
Another random grep:
% egrep 't/\*/\*\.t' */{Makefile,Build}.PL
libauthen-simple-ldap-perl/Makefile.PL:tests 't/*.t t/*/*.t';
libauthen-simple-perl/Makefile.PL:tests 't/*.t t/*/*.t';
libb-utils-perl/Makefile.PL: test => { TESTS => 't/*.t t/*/*.t' },
libclass-data-accessor-perl/Makefile.PL:tests "t/*.t t/*/*.t";
libcrypt-rijndael-perl/Makefile.PL: 'TESTS' => 't/*.t t/*/*.t'
libdata-clone-perl/Makefile.PL:tests 't/*.t t/*/*.t';
libdevel-cover-perl/Makefile.PL:push @tests, grep !/e2e/, glob "t/*/*.t";
libdevel-cover-perl/Makefile.PL: :
"t/*/*.t"
libextutils-modulemaker-perl/Makefile.PL: test => { TESTS => join q{
}, map {glob} qw( t/*.t t/*/*.t ) },
libfile-zglob-perl/Makefile.PL:tests 't/*.t t/*/*.t t/*/*/*.t t/*/*/*/*.t';
libimager-perl/Makefile.PL:my $tests = 't/*.t t/*/*.t';
libio-file-withpath-perl/Makefile.PL:tests 't/*.t t/*/*.t t/*/*/*.t
t/*/*/*/*.t';
libmakefile-dom-perl/Makefile.PL:tests('t/*.t t/mdom/*.t t/mdom/*/*.t'); #
t/*/*.t t/*/*/*t t/*/*/*/*.t');
libmango-perl/Makefile.PL: test => {TESTS => 't/*.t t/*/*.t'}
libminion-perl/Makefile.PL: test => {TESTS => 't/*.t t/*/*.t'}
libmodule-cpants-analyse-perl/Makefile.PL: test => {TESTS => 't/*.t t/*/*.t'},
libmojo-pg-perl/Makefile.PL: test => {TESTS => 't/*.t t/*/*.t'}
libmojolicious-perl/Makefile.PL: test => {TESTS => 't/*.t t/*/*.t'}
libmojomojo-perl/Makefile.PL:tests(join ' ', (glob('t/*.t'), glob('t/*/*.t')));
libmousex-types-perl/Makefile.PL:tests 't/*.t t/*/*.t';
libplack-app-proxy-perl/Makefile.PL:tests 't/*.t t/*/*.t';
libregexp-common-perl/Makefile.PL:my @tests = qw [t/*.t t/*/*.t];
libset-object-perl/Makefile.PL: test => { TESTS => join(' ', glob('t/*/*.t'))
},
libtest-without-module-perl/Makefile.PL:my @tests = map {glob $_ } 't/*.t',
't/*/*.t';
libtext-mecab-perl/Makefile.PL:tests 't/*.t t/*/*.t';
libtext-mecab-perl/Makefile.PL:## TESTS => 't/*.t t/*/*.t'
libunicode-utf8-perl/Makefile.PL:tests 't/*.t t/*/*.t';
libwww-mechanize-shell-perl/Makefile.PL:my @tests = map { glob $_ } 't/*.t',
't/*/*.t';
libyaml-syck-perl/Makefile.PL:tests 't/*.t t/*/*.t';
libclass-dbi-sweet-perl/Build.PL: test_files => [ glob('t/*.t'),
glob('t/*/*.t') ]
libgenome-perl/Build.PL: #test_files => [qw|t/*.t t/*/*.t t/*/*/*.t
t/*/*/*/*.t|],
libgetopt-complete-perl/Build.PL: #test_files => [qw|t/*.t t/*/*.t t/*/*/*.t
t/*/*/*/*.t|],
alexm has more idea, we'll add something hopefully more coherent later :)
Cheers,
gregor
--
.''`. https://info.comodo.priv.at/ - Debian Developer https://www.debian.org
: :' : OpenPGP fingerprint D1E1 316E 93A7 60A8 104D 85FA BB3A 6801 8649 AA06
`. `' Member of VIBE!AT & SPI, fellow of the Free Software Foundation Europe
`-
signature.asc
Description: Digital Signature
