Package: autopkgtest
Severity: wishlist
Control: block -1 by 901804
(From #901804 discussion, moving to a new bug as requested.)
On 29/07/18 14:58, Paul Gevers wrote:
[...]> On 27-07-18 22:49, Rebecca N. Palmer wrote:
[...]
Simon McVittie wrote:
At some point I might add a way to mark some tests as trivial, with
the intention that migration is never sped up by trivial tests passing,
only slowed down by them failing - but that's a future feature.
With the neutral state proposed here, a test can effectively mark itself
as trivial by returning 77(skip) if it passes.[...]
However, that's also true of flaky (return 77 on fail), and I agree it
would be more user-friendly to make this option explicit.
Which option, to use flaky versus skippable? Sorry, could you try to
phrase that sentence again, because I don't get what you mean here.
I meant that once you have skippable, you can use it to simulate either
flaky or trivial:
#equivalent to Restrictions: flaky
Restrictions: skippable
Test-Command: do_the_test || exit 77
#equivalent to Restrictions: trivial
Restrictions: skippable
Test-Command: do_the_test && exit 77
but that this doesn't imply they shouldn't exist explicitly. For much
the same reasons, trivial is easy to implement:
--- a/lib/testdesc.py
+++ b/lib/testdesc.py
@@ -43,7 _43,7 @@
known_restrictions = ['rw-build-tree', 'breaks-testbed', 'needs-root',
'build-needed', 'allow-stderr',
'isolation-container',
'isolation-machine', 'needs-recommends',
'needs-reboot',
- 'flaky', 'skippable']
+ 'flaky', 'skippable', 'trivial']
class Unsupported(Exception):
--- a/runner/autopkgtest
+++ b/runner/autopkgtest
@@ -177,7 +177,7 @@
errorcode |= 2
else:
errorcode |= 4
- else:
+ elif 'trivial' not in t.restrictions:
any_passed = True
if 'breaks-testbed' in t.restrictions:
(Not calling that a patch because there should also be documentation and
tests; needs to be applied after
https://salsa.debian.org/ci-team/autopkgtest/merge_requests/24.)
Do you[Simon McVittie] propose to add this[trivial] to autodep8-generated
"check that we can
import the top-level module" tests?
His original proposed use case (in
https://salsa.debian.org/ci-team/autopkgtest/merge_requests/19) was:
I've seriously considered mass-bug-filing patches to add trivial autopkgtests to as many
shared libraries as I can, because even something as trivial as "link an
implementation of hello(1) to the library using pkg-config and -Wl,--no-as-needed"
catches surprisingly many embarrassing packaging errors.
which would effectively be an autopkgtest-pkg-c, so probably yes.
A similar idea has come to my mind
about allow-stderr, so I think this is worth discussing.
Do you mean setting allow-stderr by default in autodep8-generated tests?
There are packages that would pass only with that (e.g. theano prints
a warning to stderr if one of its Recommends is not installed), but I
don't know how many.
