Hi all,
I've just pushed a bunch of commits to our deployed branch of
autopkgtest¹ that bring us up to date with upstream, where the
developers have done a lot of work. Many thanks to them. The most
interesting thing to subscribers to this list will be the changes -
mainly additions - to the list of known test restrictions. I've pasted
the diff of README.package-tests.rst below.
I don't currently expect any breakage to result from this, but if you
notice anything weird please reply here or come to #ubuntu-release and
we'll take a look.
Cheers,
--
Iain Lane [ i...@orangesquash.org.uk ]
Debian Developer [ la...@debian.org ]
Ubuntu Developer [ la...@ubuntu.com ]
[1] https://code.launchpad.net/~ubuntu-release/autopkgtest/+git/development/
@@ -237,14 +245,84 @@ needs-reboot
The test wants to reboot the machine using
``/tmp/autopkgtest-reboot`` (see below).
-needs-recommends
+needs-recommends (deprecated)
Enable installation of recommended packages in apt for the test
dependencies. This does not affect build dependencies.
+Don't use this for new tests, we want to remove it altogether.
+See https://lists.debian.org/debian-ci/2018/06/msg00016.html
+and bug 896698 for the rationale.
+
+flaky
+The test is expected to fail intermittently, and is not suitable for
+gating continuous integration. This indicates a bug in either the
+package under test, a dependency or the test itself, but such bugs
+can be difficult to fix, and it is often difficult to know when the
+bug has been fixed without running the test for a while. If a
+``flaky`` test succeeds, it will be treated like any other
+successful test, but if it fails it will be treated as though it
+had been skipped.
+
+skippable
+The test might need to be skipped for reasons that cannot be
+described by an existing restriction such as isolation-machine or
+breaks-testbed, but must instead be detected at runtime. If the
+test exits with status 77 (a convention borrowed from Automake), it
+will be treated as though it had been skipped. If it exits with any
+other status, its success or failure will be derived from the exit
+status and stderr as usual. Test authors must be careful to ensure
+that ``skippable`` tests never exit with status 77 for reasons that
+should be treated as a failure.
+
+skip-not-installable
+This test might have test dependencies that can't be fulfilled on all
+architectures. Therefore, when apt-get installs the dependencies, it will
+fail. Don't treat this as a test failure, but instead treat it as if the
+test was skipped.
+
+hint-testsuite-triggers
+This test exists purely as a hint to suggest when rerunning the
+tests is likely to be useful. Specifically, it exists to
+influence the way dpkg-source generates the Testsuite-Triggers
+.dsc header from test metadata: the Depends for this test are
+to be added to Testsuite-Triggers. (Just as they are for any other
+test.)
+
+The test with the hint-testsuite-triggers restriction should not
+actually be run.
+
+The packages listed as Depends for this test are usually indirect
+dependencies, updates to which are considered to pose a risk of
+regressions in other tests defined in this package.
+
+There is currently no way to specify this hint on a per-test
+basis; but in any case the debian.org machinery is not able to
+think about triggering individual tests.
+
+superficial
+The test does not provide significant test coverage, so if it
+passes, that does not necessarily mean that the package under test
+is actually functional. If a ``superficial`` test fails, it will be
+treated like any other failing test, but if it succeeds, this is
+only a weak indication of success. Continuous integration systems
+should treat a package where all non-superficial tests are skipped as
+equivalent to a package where all tests are skipped.
+
+For example, a C library might have a superficial test that simply
+compiles, links and executes a "hello world" program against the
+library under test but does not attempt to make use of the library's
+functionality, while a Python or Perl library might have a
+superficial test that runs ``import foo`` or ``require Foo;`` but
+does not attempt to use the library beyond that.
+
Defined features
-There are no currently defined Features.
+test-name
+Set an explicit test name for the log heading and the ``summary`` file
+for a ``Test-Command:`` inline test. When not given, these are
+enumerated like ``command1``.
signature.asc
Description: PGP signature
--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
