I take this discussion to mean the existence of autopkgtests is
considered a good thing even when they can't be run on ci.debian.net
(should we explicitly say that somewhere?). I hence added some to
beignet (an OpenCL driver, so hardware specific).
Simon McVittie wrote:
At some point I might add a way to mark some tests as trivial, with
the intention that migration is never sped up by trivial tests passing,
only slowed down by them failing - but that's a future feature.
With the neutral state proposed here, a test can effectively mark itself
as trivial by returning 77(skip) if it passes. (beignet's test1 is sort
of this: its skip state means "the package couldn't find a suitable GPU,
but did manage to say so without crashing", which is not a total no-op
(it would have caught #745363 and #779213) but also not the test it does
when it does find one.)
However, that's also true of flaky (return 77 on fail), and I agree it
would be more user-friendly to make this option explicit.
Do you propose to add this to autodep8-generated "check that we can
import the top-level module" tests? (And should this discussion move to
a new wishlist bug?)
Simon McVittie wrote:
On Mon, 18 Jun 2018 at 21:14:13 +0200, Paul Gevers wrote:
On 18-06-18 17:02, Simon McVittie wrote:
[...]
> * exit 4, 6 or 12: at least one test failed and was not ignored
> * same as now for those exit statuses
> * debci currently interprets status 8 like this, but doesn't see
> packages with no tests, so in practice it doesn't happen
Have you checked that there are no packages without a test, but that
claim they have one?
I have not checked that. If such packages exist, debci would currently
say they failed, which is arguably a debci bug (although I think it's
also fair to say that declaring a Testsuite but no tests is a bug in
the package). They seem likely to end up in the "always failed" state
(although I don't know whether it's debci or britney that tracks the
"always" part).
britney -
https://salsa.debian.org/release-team/britney2/commit/624b185ba60709f1686fbaa2a7623a94c5cb23ef#58138127957310211039e96862288ff2b43ba531_766_802
The proposed changes would move them to debci's new
neutral state,
There is at least one circumstance where debci sees a package that
doesn't have (or claim to have) tests: when a package starts having
tests, debci checks both the unstable (with the new tests) and testing
(without tests) versions, e.g.
https://ci.debian.net/packages/b/beignet/testing/amd64/
which britney would effectively treat the same as "always
failed" afaics.
If I read the above code correctly (I haven't tried actually running
it), no. While 'always failed' and 'neutral' give the same action
(default migration time), they aren't the same state, and neutral ->
fail counts as a regression.
Hence, the above would change the outcome if a package that previously
had no tests adds some and they fail, from "always failed" neutral to
"regression" delay. I'm not sure whether that's a good or bad thing.
