Bug#824935: sbuild: better support for running autopkgtests

[Johannes Schauer]

Hi Sean,

On Tue, 2 Aug 2016 07:23:08 -0700 Sean Whitton  wrote:
> Per the discussion in #828025, I'm pretty sure this will fail because the
> .changes generated by sbuild doesn't contain the .dsc.  So I think you need
> 
> sudo -- autopkgtest foo.dsc foobar.changes -- null

not unconditionally. Otherwise, if somebody builds the source package too, the
.changes file will also contain the .dsc and then autopkgtest will fail with:

autopkgtest: error: You must specify only one source package to test

I now made it such that the .dsc gets included before the .changes if and only
if the source package was not being built (and is thus not included in the
.changes file).

Thanks!

cheers, josch


signature.asc
Description: signature

Bug#824935: sbuild: better support for running autopkgtests

[Sean Whitton]

Dear Johannes,

Thank you for working on my feature request :)

On Tue, Aug 02, 2016 at 01:26:03PM +0200, Johannes Schauer wrote:
> I now have a local patch that adds autopkgtest support in the same way that
> piuparts support is added. But if I understand the changelog entry for version
> 4.0 correctly, then I should now use the autopkgtest binary instead of adt-run
> and use '--' instead of '---'. Is that correct? Somehow the autopkgtest man
> page still talks about '---' in the synopsis.

Yes, that's correct.  The manpage is wrong.

> The default is now to run:
> 
> sudo -- autopkgtest foobar.changes -- null

Per the discussion in #828025, I'm pretty sure this will fail because
the .changes generated by sbuild doesn't contain the .dsc.  So I think
you need

sudo -- autopkgtest foo.dsc foobar.changes -- null

> Should the default be to run autopkgtest with sudo in front?

I'm pretty sure the null test bed requires this to install packages,
yes.  However, it's good that you've made it possible to remove it,
since it isn't needed for schroot test beds.

> The '--autopkgtest-opts' command line option allows one to replace the
> '-- null' part in the example above with custom options. Implementing
> percentage escapes will come at a later point if required. Please open
> a separate bug for that and then I can also implement that for
> piuparts.

Filed #833285 and #833286.

> The 'autopkgtest-root-args' command line option allows one to replace
> the 'sudo --' part in the example above.
> 
> Does this sound sensible as an initial implementation?

Looks good to me, though I haven't done any testing yet.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#824935: sbuild: better support for running autopkgtests

[Johannes Schauer]

Control: tag -1 + pending patch

Hi Sean,

CC-ing Martin Pitt to get this right.

On Wed, 8 Jun 2016 15:13:58 +0900 Sean Whitton  wrote:
> So the problem is to come up with a sensible default set of options to pass
> to adt-run that will Just Work?  This should work without any schroot:
> 
> adt-run --changes foo.changes --- null
> 
> Someone who *does* have a schroot will want to override this default to
> use their schroot: in my experience, the null test is useful but it
> doesn't catch some failures such as missing test dependencies.
> 
> So we also need --adt-run-opts.  The user will want to use %a in this,
> and ideally a percent escape sequence representing the
> suite/distribution -- this would also be useful in $piuparts_opts but I
> don't believe it is currently supported.
> 
> If we assume that sbuild implicitly passes the changes file to adt-run
> as it currently does with lintian and piuparts, I could have
> 
> $adt_run_opts = [ '---', 'schroot', '%D-%a-sbuild'];
> 
> where %D is my imagined escape sequence giving the distribution.  If I
> need to do a build for a distribution that I don't have a schroot
> prepared for, I could use `--no-run-adt-run --no-run-piuparts` to disable
> those steps.

I now have a local patch that adds autopkgtest support in the same way that
piuparts support is added. But if I understand the changelog entry for version
4.0 correctly, then I should now use the autopkgtest binary instead of adt-run
and use '--' instead of '---'. Is that correct? Somehow the autopkgtest man
page still talks about '---' in the synopsis.

The default is now to run:

sudo -- autopkgtest foobar.changes -- null

Should the default be to run autopkgtest with sudo in front?

The '--autopkgtest-opts' command line option allows one to replace the '--
null' part in the example above with custom options. Implementing percentage
escapes will come at a later point if required. Please open a separate bug for
that and then I can also implement that for piuparts.

The 'autopkgtest-root-args' command line option allows one to replace the 'sudo
--' part in the example above.

Does this sound sensible as an initial implementation?

Patch attached.

Thanks!

cheers, josch
From f33ecbef3ad771c41e8421300c2eb0b48c7686f1 Mon Sep 17 00:00:00 2001
From: Johannes 'josch' Schauer 
Date: Tue, 2 Aug 2016 13:13:12 +0200
Subject: [PATCH] Add feature to run autopkgtest after a successful build.

---
 lib/Sbuild/Build.pm   | 45 ++
 lib/Sbuild/Conf.pm| 45 ++
 lib/Sbuild/Options.pm | 31 ++
 man/sbuild.1.in   | 60 +++
 4 files changed, 181 insertions(+)

diff --git a/lib/Sbuild/Build.pm b/lib/Sbuild/Build.pm
index f453fbb..ed34ea0 100644
--- a/lib/Sbuild/Build.pm
+++ b/lib/Sbuild/Build.pm
@@ -821,6 +821,10 @@ sub run_fetch_install_packages {
 	$self->check_abort();
 	$self->run_piuparts();
 
+	# Run autopkgtest.
+	$self->check_abort();
+	$self->run_autopkgtest();
+
 	# Run post build external commands
 	$self->check_abort();
 	if(!$self->run_external_commands("post-build-commands")) {
@@ -1550,6 +1554,45 @@ sub run_piuparts {
 return 1;
 }
 
+sub run_autopkgtest {
+my $self = shift;
+
+return 1 unless ($self->get_conf('RUN_AUTOPKGTEST'));
+
+$self->log_subsubsection("autopkgtest");
+
+my $autopkgtest = $self->get_conf('AUTOPKGTEST');
+my @autopkgtest_command;
+if (scalar(@{$self->get_conf('AUTOPKGTEST_ROOT_ARGS')})) {
+	push @autopkgtest_command, @{$self->get_conf('AUTOPKGTEST_ROOT_ARGS')};
+} else {
+	push @autopkgtest_command, 'sudo', '--';
+}
+push @autopkgtest_command, $autopkgtest, $self->get('Changes File');
+if (scalar(@{$self->get_conf('AUTOPKGTEST_OPTIONS')})) {
+	push @autopkgtest_command, @{$self->get_conf('AUTOPKGTEST_OPTIONS')};
+} else {
+	push @autopkgtest_command, '--', 'null';
+}
+$self->get('Host')->run_command(
+{ COMMAND => \@autopkgtest_command,
+  PRIORITY => 0,
+});
+my $status = $? >> 8;
+$self->set('Autopkgtest Reason', 'pass');
+
+$self->log("\n");
+# fail if neither all tests passed nor was the package without tests
+if ($status != 0 && $status != 8) {
+$self->log_error("Autopkgtest run failed.\n");
+	$self->set('Autopkgtest Reason', 'fail');
+return 0;
+}
+
+$self->log_info("Autopkgtest run was successful.\n");
+return 1;
+}
+
 sub build {
 my $self = shift;
 
@@ -2345,6 +2388,8 @@ sub generate_stats {
 	if $self->get('Lintian Reason');
 $self->add_stat('Piuparts', $self->get('Piuparts Reason'))
 	if $self->get('Piuparts Reason');
+$self->add_stat('Autopkgtest', $self->get('Autopkgtest Reason'))
+	if $self->get('Autopkgtest Reason');
 }
 
 sub log_stats {
diff --git a/lib/Sbuild/Conf.pm b/lib/Sbuild/Conf.pm
index 

Bug#824935: sbuild: better support for running autopkgtests

[Sean Whitton]

Dear Johannes,

On Tue, Jun 07, 2016 at 10:12:54PM +0200, Johannes Schauer wrote:
> To be able to run autopkgtests, one needs to have a chroot configured,
> needs to know which backend to use and needs to know which
> distribution to use. The sbuild wiki just plainly assumes that the
> user is using schroot, has a chroot configured and is always building
> for unstable. But people using sbuild are neither only building for
> unstable, nor do they necessarily use the schroot backend and might
> thus not have a schroot-style chroot configured.
> 
> Now one could say that one could add a similar command line argument
> like --piuparts-opts for piuparts or --lintian-opts for lintian. But
> this then would mean that users that do not use the defaults (schroot
> and unstable) would have to pass these options all the time which gets
> us back to the original situation where users had to manually add a
> --post-build-command.
> 
> So is there an elegant way to solve this dilemma and execute adt-run
> with the right options with the least amount of user intervention per
> build?

So the problem is to come up with a sensible default set of options to
pass to adt-run that will Just Work?  This should work without any
schroot:

adt-run --changes foo.changes --- null

Someone who *does* have a schroot will want to override this default to
use their schroot: in my experience, the null test is useful but it
doesn't catch some failures such as missing test dependencies.

So we also need --adt-run-opts.  The user will want to use %a in this,
and ideally a percent escape sequence representing the
suite/distribution -- this would also be useful in $piuparts_opts but I
don't believe it is currently supported.

If we assume that sbuild implicitly passes the changes file to adt-run
as it currently does with lintian and piuparts, I could have

$adt_run_opts = [ '---', 'schroot', '%D-%a-sbuild'];

where %D is my imagined escape sequence giving the distribution.  If I
need to do a build for a distribution that I don't have a schroot
prepared for, I could use `--no-run-adt-run --no-run-piuparts` to
disable those steps.

Does this address your concerns?

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#824935: sbuild: better support for running autopkgtests

[Johannes Schauer]

Control: tag -1 + moreinfo

Hi,

On Sat, 21 May 2016 23:11:03 +0900 Sean Whitton  
wrote:
> It is possible to run autopkgtests by invoking adt-run as an sbuild
> post-build-command, as described on the wiki.[1]  However, it would be
> better if sbuild could be configured to invoke adt-run as an independent
> stage of the build, like the stages for lintian and piuparts.

I see your point. Unfortunately, there is a big practical consideration for
which I don't see an easy solution. To be able to run autopkgtests, one needs
to have a chroot configured, needs to know which backend to use and needs to
know which distribution to use. The sbuild wiki just plainly assumes that the
user is using schroot, has a chroot configured and is always building for
unstable. But people using sbuild are neither only building for unstable, nor
do they necessarily use the schroot backend and might thus not have a
schroot-style chroot configured.

Now one could say that one could add a similar command line argument like
--piuparts-opts for piuparts or --lintian-opts for lintian. But this then would
mean that users that do not use the defaults (schroot and unstable) would have
to pass these options all the time which gets us back to the original situation
where users had to manually add a --post-build-command.

So is there an elegant way to solve this dilemma and execute adt-run with the
right options with the least amount of user intervention per build?

Thanks!

cheers, josch


signature.asc
Description: signature

Bug#824935: sbuild: better support for running autopkgtests

[Sean Whitton]

Package: sbuild
Version: 0.69.0-2
Severity: wishlist

Dear maintainer,

It is possible to run autopkgtests by invoking adt-run as an sbuild
post-build-command, as described on the wiki.[1]  However, it would be
better if sbuild could be configured to invoke adt-run as an independent
stage of the build, like the stages for lintian and piuparts.  This
would have the following advantages:

1. If the test suite fails, this would be reported in the final output
   as "autopkgtest: fail".  At present a failure in the test suite
   counts as a failure for the whole build since a failure in a
   post-build-command is considered to be a failure of the build.

2. adt-run dies with exit code 8 if a package does not have any DEP-8
   tests defined.  In a post-build command this counts as a failure, but
   for sbuild's purposes it should not count as a failure of the build
   that a package doesn't have any tests.  This can be worked around
   using wrapper scripts etc. in the post-build-command, but the user
   shouldn't have to worry about this.

3. There could be built-in support for autodep8, if needed (maybe
   adt-run handles this; I'm not sure).

Thanks.

[1] https://wiki.debian.org/sbuild#Using_autopkgtest

-- System Information:
Debian Release: stretch/sid
  APT prefers testing
  APT policy: (900, 'testing')
Architecture: i386 (i686)

Kernel: Linux 4.5.0-2-686-pae (SMP w/2 CPU cores)
Locale: LANG=en_GB.utf8, LC_CTYPE=en_GB.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages sbuild depends on:
ii  adduser 3.114
ii  apt-utils   1.2.11
ii  libsbuild-perl  0.69.0-2
ii  perl5.22.2-1

Versions of packages sbuild recommends:
ii  debootstrap  1.0.81
ii  fakeroot 1.20.2-1

Versions of packages sbuild suggests:
pn  deborphan  
ii  wget   1.17.1-1+b1

-- no debconf information

-- 
Sean Whitton


signature.asc
Description: PGP signature