Bug#994139: lintian: warning about superficial autopkgtests is counterproductive
On Wed, 2021-09-29 at 18:59 -0700, Felix Lechner wrote: > Would you be willing to revert your commit that bumped the visibility > [1] until we can figure out a better way to proceed? Reverted. -- bye, pabs https://wiki.debian.org/PaulWise signature.asc Description: This is a digitally signed message part
Bug#994139: lintian: warning about superficial autopkgtests is counterproductive
Hi Paul, On Wed, Sep 29, 2021 at 5:54 PM Sean Whitton wrote: > > I agree, and would like to see the new tag downgraded below the W: > level. Taking great pride in the fact that Lintian is team-maintained, I am reluctant to act here. Would you be willing to revert your commit that bumped the visibility [1] until we can figure out a better way to proceed? Kind regards Felix Lechner [1] https://salsa.debian.org/lintian/lintian/-/commit/cbf654cce71dd2ac294c82767963cc0507093d42
Bug#994139: lintian: warning about superficial autopkgtests is counterproductive
Hello, On Sun 12 Sep 2021 at 07:07PM +01, Simon McVittie wrote: > I see lintian has recently started emitting warnings for packages that > have autopkgtests, but only superficial autopkgtests. I think this is > counterproductive. > > Obviously, if a package can have reliable autopkgtests that are not > superficial (not always feasible!), then we would prefer to have those. > > However, if non-superficial autopkgtests are not achievable, then it's > *considerably* better to have superficial autopkgtests than no coverage > at all - a superficial test, like running "foo --help" and checking > that it doesn't segfault or linking a trivial program to a library and > checking that it can link, can at least check that the package is not > *completely* broken (perhaps in time to stop a serious regression in the > package or a dependency from migrating to testing). I agree, and would like to see the new tag downgraded below the W: level. It is not always a bug of greater severity than "wishlist" that a package doesn't have non-superficial autopkgtests. Perhaps it would be a bug of a greater severity for some packages, based on certain roles they might have, but not in general. I would also note that Policy doesn't say anything about the degree to which it is valuable to have autopkgtests -- unlike, for example, how it says that all installed programs should have manpages. -- Sean Whitton signature.asc Description: PGP signature
Bug#994139: lintian: warning about superficial autopkgtests is counterproductive
On Sun, 12 Sep 2021 at 16:50:03 -0700, Felix Lechner wrote: > As a side note, the old tag to which you seem attached took a stance > against superficial tests. I think the text of its recommendation to maintainers was written before the special handling of the "superficial" restriction was implemented (it was originally named "trivial" and renamed during code review). https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=904979 Before the "superficial" restriction was understood by autopkgtest, debci and britney, all superficial tests would have necessarily been in the category that is somewhat unhelpful: tests that do not have meaningful coverage, but are not marked as such. smcv
Bug#994139: lintian: warning about superficial autopkgtests is counterproductive
Hi, On Sun, Sep 12, 2021 at 3:27 PM Simon McVittie wrote: > > the first of those is the new missing-tests-control, and I would > agree with making it an error. Done. It is now an error. [1] I'll note that the condition is somewhat artificial. It would probably be better to declare all testing matters—like deployment of a team's test suite—in d/tests/control. The mere presence of that file would declare the test suite. The condition could then never occur, rendering the tag obsolete. [1] https://salsa.debian.org/lintian/lintian/-/commit/79c47559e95213c4318b7e02130aa962332c75ea > I think the second of those is the old testsuite-autopkgtest-missing, > which apparently no longer exists. Saying that testsuite-autopkgtest-missing > is an old name for the new missing-tests-control seems inaccurate? The old tag testsuite-autopkgtest-missing tested somewhat indiscriminately whether 'autopkgtest' was mentioned in the field Testsuite in d/control. [2] Dpkg since version 1.17.11, however, automatically adds that value to the dsc when d/tests/control is present [3]. That part of the condition is no longer meaningful. (It was arguably also the only error detected by the tag.) For the other half of the tag—and the purpose stated in its description—namely "package does not declare a test suite" [3] there is from what I can tell no project consensus. That is why the tag was deleted. As a side note, the old tag to which you seem attached took a stance against superficial tests. [5] [2] https://salsa.debian.org/lintian/lintian/-/commit/c81fb3dbc4425c4322c67f0f3eeb2c208e337736#6f4367b101cfbfb8143c3faa1edd9c67326e2614_76_70 [3] https://sources.debian.org/src/dpkg/1.20.9/debian/changelog/ [4] https://salsa.debian.org/lintian/lintian/-/commit/c81fb3dbc4425c4322c67f0f3eeb2c208e337736#339cb554585174385393ac6779bbc6202a1339b4_4_0 [5] https://salsa.debian.org/lintian/lintian/-/commit/c81fb3dbc4425c4322c67f0f3eeb2c208e337736#339cb554585174385393ac6779bbc6202a1339b4_19_0 > Conversely, if superficial-tests is serious enough to justify a lintian > tag, then testsuite-autopkgtest-missing should be a lintian tag of equal > or higher severity, because it's a strictly lower level of test coverage > than superficial-tests. I agree with you here, but that logic does not hold universally. The level of coverage is only one of several criteria that apply when settling on a tag's appropriate visibility. For example, there may be a stronger project consensus that tests should be meaningful rather than that tests should be required. > to avoid > "noise" for the maintainers of packages where autopkgtest coverage is not > feasible to implement, then perhaps they should be classification tags? Classification tags are used for research in the archive. They are examined via our website's JSON interface and can yield helpful information about relative prevalence ratios. They cannot reduce noise for certain package families. For that, you may be excited about a new feature in Lintian. [6] You can now request screens for your package family although there are some hurdles to being so recognized. [7] In response to Paul's email, which came in the meantime, the screens can be used to mask the tag superficial-tests (or any other autopkgtest tag discussed here) for sources for which meaningful tests do not exist. [6] https://salsa.debian.org/lintian/lintian/-/commit/8c3d587559cfbdf5dd41e9ba1f66c9ab52de577e [7] https://lintian.debian.org/screens Kind regards Felix Lechner
Bug#994139: lintian: warning about superficial autopkgtests is counterproductive
On Sun, 2021-09-12 at 23:27 +0100, Simon McVittie wrote: > I don't think it makes sense for the new superficial-tests to be considered > worse (= higher severity) than the old testsuite-autopkgtest-missing. I was initially thinking of cases were the package is perfectly possible to test properly but the maintainer just added a foo -v superficial test instead of adding a real test. I hadn't considered packages that aren't possible to test, for those I guess I assumed maintainers would just not add any tests. If the amount of packages with superficial tests that aren't possible to properly test is higher than the amount of packages that are possible to properly test, then your reasoning makes sense and the severities should be changed. From the examples you presented, I think that is correct so I agree the severities should be changed indeed. I do feel however that the value of superficial tests is usually quite minimal and so I would suggest to use the same severity for zero tests as for only superficial tests. -- bye, pabs https://wiki.debian.org/PaulWise signature.asc Description: This is a digitally signed message part
Bug#994139: lintian: warning about superficial autopkgtests is counterproductive
On Sun, 12 Sep 2021 at 13:49:35 -0700, Felix Lechner wrote: > Either way, the project relies here on the fact > that having a meaningful testsuite may provide a faster migration from > unstable to testing. I think there might be some misunderstanding here. Tests that are marked with "Restrictions: superficial" cannot make migration faster, even if they pass: they can only block or delay migration, by failing. That's the purpose of the superficial keyword: it tells the autopkgtest infrastructure not to assume that this particular test-case has meaningful coverage. If every test in the test suite has a "neutral" result, then there is no migration bonus. Tests that pass but are marked as superficial are considered to be "neutral" rather than a success, meaning they do not speed up migration for a package that does not "deserve" it. (Other neutral results include tests that fail but are marked as "flaky", and tests that are skipped because they require isolation or other functionality that the testbed cannot provide - for example, trying to run flatpak's test suite inside an lxc container is skipped, because the security restrictions imposed by lxc prevent flatpak from working.) Adding tests that *are* superficial, but are not *marked* as superficial, is a serious problem, because it can cause packages to migrate too soon (and the release team's RC policy since bullseye says this is a release-critical bug). However, this is not really feasible for Lintian to detect, so I think the Lintian maintainers should treat detection of that class of bug as being out-of-scope. > On Sun, Sep 12, 2021 at 1:17 PM Simon McVittie wrote: > > If that's the case, I would have expected it to be emitted for packages > > that have absolutely no autopkgtest coverage > > You are right! The tag is issued when 'Testsuite: autopkgtest' was > declared in d/control but no d/tests/control is present. [1] It should > arguably be an error instead of info. [2] I think there are three separate tags that would potentially be useful in this general space, in decreasing order of severity: 1. (Testsuite: autopkgtest in d/control) && (no d/tests/control) 2. absence of autopkgtest coverage, declared correctly (no Testsuite and no d/tests/control) 3. there is autopkgtest coverage but it is all marked as superficial I think the first of those is the new missing-tests-control, and I would agree with making it an error. It's a contradiction between related source files: d/control says there is a test suite, but (the absence of) d/tests/control says there is not, and only one of those can be true. I think the second of those is the old testsuite-autopkgtest-missing, which apparently no longer exists. Saying that testsuite-autopkgtest-missing is an old name for the new missing-tests-control seems inaccurate? The third of those is the new superficial-tests. > Lintian does not currently complain when no test suite is present. As > you noted, some sources like GNOME shell extensions are essentially > impossible to test. I don't think it makes sense for the new superficial-tests to be considered worse (= higher severity) than the old testsuite-autopkgtest-missing. If the old testsuite-autopkgtest-missing is not serious enough to justify even an info-level lintian tag, then neither is superficial-tests. Conversely, if superficial-tests is serious enough to justify a lintian tag, then testsuite-autopkgtest-missing should be a lintian tag of equal or higher severity, because it's a strictly lower level of test coverage than superficial-tests. If you think these should not emit maintainer-facing tags, to avoid "noise" for the maintainers of packages where autopkgtest coverage is not feasible to implement, then perhaps they should be classification tags? smcv
Bug#994139: lintian: warning about superficial autopkgtests is counterproductive
Hi, On Sun, Sep 12, 2021 at 1:17 PM Simon McVittie wrote: > > If that's the case, I would have expected it to be emitted for packages > that have absolutely no autopkgtest coverage You are right! The tag is issued when 'Testsuite: autopkgtest' was declared in d/control but no d/tests/control is present. [1] It should arguably be an error instead of info. [2] Lintian does not currently complain when no test suite is present. As you noted, some sources like GNOME shell extensions are essentially impossible to test. Also, some long-time contributors reportedly dislike autopkgtest. Either way, the project relies here on the fact that having a meaningful testsuite may provide a faster migration from unstable to testing. Kind regards Felix Lechner [1] https://salsa.debian.org/lintian/lintian/-/blob/master/lib/Lintian/Check/Testsuite.pm#L89-91 [2] https://salsa.debian.org/lintian/lintian/-/blob/master/tags/m/missing-tests-control.tag
Bug#994139: lintian: warning about superficial autopkgtests is counterproductive
On Sun, 12 Sep 2021 at 12:48:36 -0700, Felix Lechner wrote: > On Sun, Sep 12, 2021 at 11:09 AM Simon McVittie wrote: > > > > lintian has recently started emitting warnings for packages that > > have autopkgtests, but only superficial autopkgtests. > > The tag was implemented in response to Bug#932870. [1] It was > originally suggested on OFTC#debci by dkg (who was copied here). I > never heard back whether my implementation met the original intent. > The tag originally had the info visibility, but was later upgraded to > a warning. [2] To be clear, I think this tag would have been fine as originally implemented; it's the subsequent increase of severity from pedantic to warning that I think is counterproductive. I think a package with correctly-marked superficial autopkgtests, like libsdl2-mixer 2.0.4+dfsg1-3, is better than a package with no autopkgtests at all, like libsdl2-mixer 2.0.4+dfsg1-2. As a result, to avoid giving maintainers wrong incentives, I think the Lintian tags emitted for 2.0.4+dfsg1-3 should be of equal or lower severity. > > the meaning of > > testsuite-autopkgtest-missing might have changed at some point so that it > > is only emitted for packages that have debian/tests/control, rather than > > for all packages that lack autopkgtests? > > Maybe a word is missing? The tag 'testsuite-autopkgtest-missing' was > renamed to 'missing-tests-control' and is issued at the info level > when sources do NOT ship debian/tests/control (or declare a predefined > test suite in d/control, usually for teams). [3] If that's the case, I would have expected it to be emitted for packages that have absolutely no autopkgtest coverage, such as gnome-shell-extension-caffeine. In the past, I think I saw testsuite-autopkgtest-missing emitted for that package, but at the moment, the replacement tag missing-tests-control is not emitted. Should it be? The missing-tests-control description now says: The source package declares the generic Testsuite: autopkgtest field but provides no debian/tests/control file. but packages like gnome-shell-extension-caffeine that don't have any tests at all will not usually have a Testsuite field. If the intention was for this tag to detect packages with zero test coverage, then it should also be emitted for packages with no Testsuite. GNOME shell extensions are probably a convenient example to use when evaluating "you have no tests" tags, because they are unlikely to get meaningful test coverage: they're essentially impossible to test without using something like openQA, which would be a lot of effort and inconveniently likely to fail as a result of changes that are not actually bugs, and it isn't obvious how it would fit into the autopkgtest framework even if someone has the time to implement it. I think if we get test coverage for desktop environments, it would be more likely to be a separate, parallel GUI-testing infrastructure alongside autopkgtest. smcv
Bug#994139: lintian: warning about superficial autopkgtests is counterproductive
Hi, On Sun, Sep 12, 2021 at 11:09 AM Simon McVittie wrote: > > lintian has recently started emitting warnings for packages that > have autopkgtests, but only superficial autopkgtests. The tag was implemented in response to Bug#932870. [1] It was originally suggested on OFTC#debci by dkg (who was copied here). I never heard back whether my implementation met the original intent. The tag originally had the info visibility, but was later upgraded to a warning. [2] > I think this is counterproductive. Thank you for that pointer. This reply merely acknowledges your filing. I am trying to reconstruct how users perceive the interaction of these tags. Lintian hopes to offer middle-of-the-road advice. > the meaning of > testsuite-autopkgtest-missing might have changed at some point so that it > is only emitted for packages that have debian/tests/control, rather than > for all packages that lack autopkgtests? Maybe a word is missing? The tag 'testsuite-autopkgtest-missing' was renamed to 'missing-tests-control' and is issued at the info level when sources do NOT ship debian/tests/control (or declare a predefined test suite in d/control, usually for teams). [3] Lintian also issues 'no-tests' for defective control stanzas [4] and the somewhat specialized 'no-op-testsuite' for fakers [5]. The relevant code with a few other tags is located here. [6] I will likely respond with a suggested solution at some point in the future. Meanwhile, comments are welcome! Kind regards Felix Lechner [1] https://bugs.debian.org/932870 [2] https://salsa.debian.org/lintian/lintian/-/commit/cbf654cce71dd2ac294c82767963cc0507093d42 [3] https://salsa.debian.org/lintian/lintian/-/blob/master/tags/m/missing-tests-control.tag [4] https://salsa.debian.org/lintian/lintian/-/blob/master/tags/n/no-tests.tag [5] https://salsa.debian.org/lintian/lintian/-/blob/master/tags/n/no-op-testsuite.tag [6] https://salsa.debian.org/lintian/lintian/-/blob/master/lib/Lintian/Check/Testsuite.pm
Bug#994139: lintian: warning about superficial autopkgtests is counterproductive
Package: lintian Version: 2.105.0 Severity: normal X-Debbugs-Cc: debian...@lists.debian.org, Paul Wise I see lintian has recently started emitting warnings for packages that have autopkgtests, but only superficial autopkgtests. I think this is counterproductive. Obviously, if a package can have reliable autopkgtests that are not superficial (not always feasible!), then we would prefer to have those. However, if non-superficial autopkgtests are not achievable, then it's *considerably* better to have superficial autopkgtests than no coverage at all - a superficial test, like running "foo --help" and checking that it doesn't segfault or linking a trivial program to a library and checking that it can link, can at least check that the package is not *completely* broken (perhaps in time to stop a serious regression in the package or a dependency from migrating to testing). While checking the progress of the GNOME 40 transition I noticed that the maintainer of budgie-desktop has removed its superficial autopkgtest in order to silence the Lintian warning, leaving it with no autopkgtest coverage at all. I think this is the opposite of what we want! If maintainers are unable to add extensive coverage, we should be encouraging them to at least add superficial coverage. The only class of tests that should be discouraged is superficial tests that are not correctly marked with the 'superficial' keyword - but those are probably not something that is feasible to machine-detect, except in extremely simple cases, since the test is an arbitrary Turing-complete script. Some packages in the three possible categories, in order from least to most desirable: * gnome-shell-extension-caffeine is not really feasible to test at all: https://lintian.debian.org/sources/gnome-shell-extension-caffeine?version=38-1 * libsdl2-mixer only has superficial tests, which are better than nothing: https://lintian.debian.org/sources/libsdl2-mixer * dbus has actual unit tests: https://lintian.debian.org/sources/dbus I would expect that the test-related tags in these three packages should reflect that order from least to most desirable, with tag severity decreasing as we progress through the list. Something like this: * gnome-shell-extension-caffeine: a low-severity tag encouraging the maintainer to add tests if feasible, but not creating so much noise that it masks real problems (this used to be "info" severity, and I think that was entirely appropriate) * libsdl2-mixer: a low-severity tag encouraging the maintainer to add non-superficial tests if feasible, but not creating so much noise that it masks real problems (I would say "info" or "pedantic" severity) * dbus: no tags In fact they are: * gnome-shell-extension-caffeine: no tags * libsdl2-mixer: W: superficial-tests * dbus: no tags I think packages like gnome-shell-extension-caffeine used to be flagged with testsuite-autopkgtest-missing, but it looks as though the meaning of testsuite-autopkgtest-missing might have changed at some point so that it is only emitted for packages that have debian/tests/control, rather than for all packages that lack autopkgtests? smcv
