Hi All, This is a new list for the autopkgtest superficial test.
If the test done in the autopkgtest does not provide significant test coverage then it should be marked with "Restrictions: superficial". Ref: https://salsa.debian.org/ci-team/autopkgtest/-/blob/master/doc/README.package-tests.rst Examples of tests which are not significant includes (its not a complete list): 1) Executing the binary to check version Test-Command: foo -v Test-Command: foo -V Test-Command: foo --version 2) Executing the binary to check help (foo -h) Test-Command: foo -h Test-Command: foo --help 3) A Python or Perl library runs import foo or require Foo; but does not attempt to use the library beyond that. Test-Command: python3 -c "import foo" Similar to the last MBF, the intended bug text is: ************************************************************************* Subject: <package>: autopkgtest must be marked superficial Severity: important User: sudipm.mukher...@gmail.com Usertags: superficialtest It has been noticed that the autopkgtest in <package> is running a trivial command that does not provide significant test coverage: - <command being run, e.g. foo --help> Executing that command is considered to be a trivial test, which does not provide significant coverage for a package as a whole. But these tests are a useful way to detect regressions in dependencies and prevent them from breaking your package. However, it is important that we are realistic about the level of test coverage provided by these commands: most regressions cannot be detected in this way. So it is not appropriate for packages with only superficial tests to have the reduced migration time to migrate from unstable to testing as that means less opportunity for testing by users compared to the package with no tests. To support this, the keyword "Restrictions: superficial" has been defined [1]. Packages where all tests are marked with this keyword are not considered for the reduced migration age from unstable to testing, and will not be allowed to migrate automatically in later stages of the freeze [2]. Its always better to have more extensive testing than having superficial testing, which again is better than having no test. Please consider i) Adding a non-trivial test, and/or ii) Mark the trivial test with "Restrictions: superficial", similar to [3] or [4]. The Release Team has listed this issue in the list of Release Critical Issues for bullseye [5] and has mentioned that the test must be marked superficial if it is not testing one of its own installed binary packages in some way. As a result, the severity of this bug report might be increased to serious in future. [1] https://salsa.debian.org/ci-team/autopkgtest/-/blob/master/doc/README.package-tests.rst#defined-restrictions [2] https://release.debian.org/bullseye/freeze_policy.html [3] https://salsa.debian.org/utopia-team/dbus/-/commit/a80908df7d119b181eec5eb0542634a30c2ad468 [4] https://salsa.debian.org/apparmor-team/apparmor/-/commit/580667513a097088ebe579884b38ac8d8666d3b3 [5] https://release.debian.org/bullseye/rc_policy.txt ************************************************************************* Attached is the dd-list. -- Regards Sudip
Aaron Zauner <a...@azet.org> lmod Adrian Vondendriesch <adrian.vondendrie...@credativ.de> wtf-peewee (U) Agustin Henze <t...@debian.org> crrcsim Alastair McKinstry <mckins...@debian.org> cdo eckit libdap pyferret pygrib Albert Huang <alb...@csail.mit.edu> pybluez (U) Alvin Chen <sonoma...@gmail.com> cgreen elementpath Andreas Moog <andreas.m...@warperbbs.de> nzbget Andreas Tille <ti...@debian.org> artemis (U) Barry Warsaw <ba...@debian.org> enum34 (U) flufl.password (U) python-progress (U) python3-openid (U) Ben Finney <bign...@debian.org> dput Carl Chenet <cha...@ohmytux.com> python-memcache (U) Carlos Maddela <e7ap...@gmail.com> dvdisaster (U) Carsten Schoenert <c.schoen...@t-online.de> python-ilorest Chris Lamb <la...@debian.org> xtrlock (U) Christoph Berg <m...@debian.org> libcharon (U) Christopher Hoskin <mans0...@debian.org> case (U) django-memoize (U) django-setuptest (U) pycryptodome (U) Craig Small <csm...@debian.org> procps Debian 3-D Printing Packages <3dprinter-gene...@lists.alioth.debian.org> libarcus libcharon libsavitar uranium Debian Astronomy Maintainers <debian-astro-maintain...@lists.alioth.debian.org> galpy orbit-predictor Debian Bluetooth Maintainers <team+pkg-blueto...@tracker.debian.org> pybluez Debian Electronics Team <pkg-electronics-de...@lists.alioth.debian.org> fpga-icestorm Debian Med Packaging Team <debian-med-packag...@lists.alioth.debian.org> artemis pyode Debian Middleware Maintainers <pkg-middleware-maintain...@lists.alioth.debian.org> qpid-proton Debian Multimedia Maintainers <debian-multime...@lists.debian.org> csound Debian OpenStack <team+openst...@tracker.debian.org> python-croniter python-debtcollector python-memcache python-openstacksdk python-osprofiler python-taskflow python-tosca-parser Debian Python Modules Team <python-modules-t...@lists.alioth.debian.org> case django-memoize django-setuptest enum34 flake8-docstrings flufl.i18n flufl.password pycryptodome pymodbus pyserial-asyncio python-box python-colorama (U) python-iptables python-irodsclient python-progress python-scripttest python3-openid pyvisa-py readlike reparser sortedcollections sphinxcontrib-asyncio wtf-peewee zope.deprecation Debian Python Modules Team <team+python-modu...@tracker.debian.org> gevent-websocket Debian Python Team <team+pyt...@tracker.debian.org> python-exif python-pyramid python-socksipy Debian QA Group <packa...@qa.debian.org> aewm antigrav gramofile iroffer zope.exceptions Debian Salt Team <pkg-salt-t...@lists.alioth.debian.org> salt-pepper Debian Science Maintainers <debian-science-maintain...@lists.alioth.debian.org> arachne-pnr python-fluids python-ulmo qflow Debian Security Tools <team+pkg-secur...@tracker.debian.org> forensics-extra Debian Tryton Maintainers <team+tryton-t...@tracker.debian.org> goocalendar python-zeep tryton-client tryton-meta vatnumber Diane Trout <di...@ghic.org> readlike (U) reparser (U) sortedcollections (U) sphinxcontrib-asyncio (U) Federico Ceratto <feder...@debian.org> django-ranged-response (U) python-socksipychain (U) Felipe Sateler <fsate...@debian.org> csound (U) Filip Pytloun <fi...@pytloun.cz> salt-pepper (U) Forrest Cahoon <forrest.cah...@gmail.com> csound (U) FreedomBox packaging team <freedombox-pkg-t...@lists.alioth.debian.org> django-ranged-response python-socksipychain Gaetano Guerriero <x.guerri...@tin.it> eyed3 Ghislain Antony Vaillant <ghisv...@gmail.com> flake8-docstrings (U) Giovani Augusto Ferreira <giov...@debian.org> forensics-extra (U) Gregor Riepl <onit...@gmail.com> libarcus (U) libcharon (U) libsavitar (U) uranium (U) Gürkan Myczko <gur...@phys.ethz.ch> python-colorama IOhannes m zmölnig (Debian/GNU) <umlae...@debian.org> python-iptables (U) James Valleroy <jvalle...@mailbox.org> django-ranged-response (U) Jamie Wilkinson <j...@debian.org> pyao pymad Joao Eriberto Mota Filho <eribe...@debian.org> forensics-extra (U) gimp-texturize Josue Ortega <jo...@debian.org> consonance galpy (U) orbit-predictor (U) kretcheu <kretc...@gmail.com> hcxtools (U) Kurt Kremitzki <kurt@kwk.systems> python-fluids (U) python-ulmo (U) Marcin Owsiany <porri...@debian.org> apt-forktracer Marcio de Souza Oliveira <marcioso...@debian.org> edb-debugger Martin <deba...@debian.org> python-socksipy (U) Mathias Behrle <mathi...@m9s.biz> goocalendar (U) tryton-client (U) tryton-meta (U) vatnumber (U) Mathias Behrle <mbeh...@debian.org> python-zeep (U) Matthew Vernon <matt...@debian.org> xtrlock Maximiliano Curia <m...@debian.org> pymodbus (U) Michael Hanke <michael.ha...@gmail.com> pyode (U) Michal Arbet <michal.ar...@ultimum.io> python-box (U) python-openstacksdk (U) python-taskflow (U) Mo Zhou <cdlumin...@gmail.com> termbox Nilesh Patra <npatra...@gmail.com> pyode (U) Olivier Sallou <osal...@debian.org> python-irodsclient (U) Ondřej Tůma <mc...@zeropage.cz> formiko Optical Media Tools Team <pkg-opt-media-t...@lists.alioth.debian.org> dvdisaster Otto Kekäläinen <o...@debian.org> rdiff-backup (U) Petter Reinholdtsen <p...@debian.org> python-socksipychain (U) Pierre Gruet <pgtdeb...@free.fr> artemis (U) Pierre-Elliott Bécue <p...@debian.org> flufl.i18n (U) Python Applications Packaging Team <python-apps-t...@lists.alioth.debian.net> rdiff-backup Rogério Brito <rbr...@ime.usp.br> dvdisaster (U) Ruben Undheim <ruben.undh...@gmail.com> arachne-pnr (U) fpga-icestorm (U) pyvisa-py (U) qflow (U) Sunil Mohan Adapa <su...@medhas.org> django-ranged-response (U) python-socksipychain (U) SZ Lin (林上智) <sz...@debian.org> cgreen (U) elementpath (U) TANIGUCHI Takaki <tak...@debian.org> dvdisaster (U) python-exif (U) python-pyramid (U) zope.deprecation (U) Thomas Goirand <z...@debian.org> python-croniter (U) python-debtcollector (U) python-memcache (U) python-openstacksdk (U) python-osprofiler (U) python-taskflow (U) python-tosca-parser (U) qpid-proton (U) Thomi Richards <thomi.richa...@canonical.com> python3-openid (U) Ulises Vitulli <der...@debian.org> cbatticon hcxtools W. Martin Borgert <deba...@debian.org> pymodbus (U) pyserial-asyncio (U) python-exif (U) William Grzybowski <will...@grzy.org> gevent-websocket (U) ytree Yaroslav Halchenko <deb...@onerussian.com> pyode (U)