On Wed, Feb 13, 2019 at 09:23:00PM +0100, Xavier Guimard wrote: > Package: pkg-perl-autopkgtest > Version: 0.50 > Severity: wishlist
> - tests skipped should return a 77 exit code and all tests marked as > "Restrictions: skippable". It avoids to consider that a test succeeds > if maintainer skipped it, but needs a merge request to autodep8. See > > https://salsa.debian.org/ci-team/autodep8/blob/master/support/nodejs/generate > (changed by MR !11) Makes sense to me. > - runtime-deps* tests should be tagged as "Restrictions: superficial" > since these tests don't really test package features but just Perl > syntax > > Then with this 2 changes, if "build-deps.d" is skipped, success won't > give the benefit of 3-days-reduce. I agree with the intention here. The only non-superficial test of the current ones is indeed the 'smoke' test. However, I have an "architectural" concern. The proposed implementation overloads the 'runtime-deps' vs 'build-deps' separation. Theoretically, we might come up with a 'runtime-deps' test that is not superficial even though all of them currently are, or a 'build-deps' test that is superficial unlike the current ones. The separation precedes autodep8 and was designed to make it possible to update the set of standard pkg-perl autopkgtests without having to update every package separately. This was later solved in a more centralized way by autodep8. I think we should probably get rid of this now unnecessary layer of indirection and just list smoke, use.t and syntax.t in autodep8, with the proper restrictions. This seems buster+1 material. Please don't consider this a blocker or veto for the proposed implementation. It's a clear improvement and does not make it any harder to simplify the architecture later. Thanks for working on this, -- Niko